World Library  
Flag as Inappropriate
Email this Article

Byte-order mark

Article Id: WHEBN0024859106
Reproduction Date:

Title: Byte-order mark  
Author: World Heritage Encyclopedia
Language: English
Subject: Mapping of Unicode characters, Binary Ordered Compression for Unicode, WinMerge, UTF-1
Collection:
Publisher: World Heritage Encyclopedia
Publication
Date:
 

Byte-order mark

The byte order mark (BOM) is a Unicode character used to signal the endianness (byte order) of a text file or stream. It is encoded at U+FEFF byte order mark (BOM). BOM use is optional, and, if used, should appear at the start of the text stream. Beyond its specific use as a byte-order indicator, the BOM character may also indicate which of the several Unicode representations the text is encoded in.[1]

Because Unicode can be encoded as 16-bit or 32-bit integers, a computer receiving these encodings from arbitrary sources needs to know which byte order the integers are encoded in. The BOM gives the producer of the text a way to describe the text stream's endianness to the consumer of the text without requiring some contract or metadata outside of the text stream itself. Once the receiving computer has consumed the text stream, it presumably processes the characters in its own native byte order and no longer needs the BOM. Hence the need for a BOM arises in the context of text interchange, rather than in normal text processing within a closed environment.

Usage

If the BOM character appears in the middle of a data stream, Unicode says it should be interpreted as a "zero-width non-breaking space" (essentially a null character). In Unicode 3.2, this usage is deprecated in favour of the "Word Joiner" character, U+2060.[1] This allows U+FEFF to be only used as a BOM.

UTF-8

The UTF-8 representation of the BOM is the byte sequence 0xEF,0xBB,0xBF. A text editor or web browser interpreting the text as ISO-8859-1 or CP1252 will display the characters  for this.

The Unicode Standard permits the BOM in UTF-8,[2] but does not require or recommend its use.[3] Byte order has no meaning in UTF-8,[4] so its only use in UTF-8 is to signal at the start that the text stream is encoded in UTF-8. The BOM may also appear when UTF-8 data is converted from other encodings that use a BOM. The standard also does not recommend removing a BOM when it is there, so that round-tripping between encodings does not lose information, and so that code that relies on it continues to work.[5] [6]

Reasons the standard does not advocate the UTF-8 BOM include:

  • To encourage conversion to Unicode. Usually ASCII-based legacy encodings can be detected because sequences of bytes with the high bit sets are unlikely to be UTF-8 sequences. Therefore the BOM is not needed to determine whether the text stream is UTF-8 or not. Using the BOM by convention adds to the work of programming, which would discourage programmers from using it and instead continue to use legacy encodings.
  • A plain ASCII file is in UTF-8 encoding. Requiring a BOM makes an artificial distinction between ASCII and UTF-8.
  • A language parser that transparently handles bytes with the high bit set in certain free-text contexts (such as string literals or comments) but otherwise uses a syntax defined only by ASCII characters, is already able to read and process UTF-8 correctly, even if it is not designed for Unicode. However the BOM at the start would violate its syntax and cause a parsing error. This is true of almost all languages written for personal computers and designed to handle legacy encodings such as CP1252.
  • It defeats software that uses pattern matching on the start of a text file, since it inserts 3 bytes before the pattern. Though commonly associated with the Unix shebang at the start of an interpreted script,[7] the problem is more widespread.

Despite this, Microsoft compilers[8] and interpreters, and many pieces of software on Microsoft Windows such as Notepad will not correctly read UTF-8 text unless it has only ASCII characters or it starts with the BOM, and will add a BOM to the start when saving text as UTF-8. Google Docs will add a BOM when a Microsoft Word document is downloaded as a plain text file.

UTF-16

In UTF-16, a BOM (U+FEFF) may be placed as the first character of a file or character stream to indicate the endianness (byte order) of all the 16-bit code units of the file or stream.

  • If the 16-bit units are represented in big-endian byte order, this BOM character will appear in the sequence of bytes as 0xFE followed by 0xFF. This sequence appears as the ISO-8859-1 characters þÿ in a text display that expects the text to be ISO-8859-1.
  • if the 16-bit units use little-endian order, the sequence of bytes will have 0xFF followed by 0xFE. This sequence appears as the ISO-8859-1 characters ÿþ in a text display that expects the text to be ISO-8859-1.

Programs expecting UTF-8 may show these or error indicators, depending on how they handle UTF-8 encoding errors. In all cases they will probably display the rest of the file as garbage (a UTF-16 text containing ASCII only will be fairly readable).

For the IANA registered charsets UTF-16BE and UTF-16LE, a byte order mark should not be used because the names of these character sets already determine the byte order. If encountered anywhere in such a text stream, U+FEFF is to be interpreted as a "zero width no-break space".

Clause D98 of conformance (section 3.10) of the Unicode standard states, "The UTF-16 encoding scheme may or may not begin with a BOM. However, when there is no BOM, and in the absence of a higher-level protocol, the byte order of the UTF-16 encoding scheme is big-endian." Whether or not a higher-level protocol is in force is open to interpretation. Files local to a computer for which the native byte ordering is little-endian, for example, might be argued to be encoded as UTF-16LE implicitly. Therefore the presumption of big-endian is widely ignored. When those same files are accessible on the Internet, on the other hand, no such presumption can be made. Searching for ASCII characters or just the space character (U+0020) is a method of determining the UTF-16 byte order.

UTF-32

Although a BOM could be used with UTF-32, this encoding is rarely used for transmission. Otherwise the same rules as for UTF-16 are applicable.

Representations of byte order marks by encoding

This table illustrates how BOMs are represented as byte sequences and how they might appear in a text editor that is interpreting each byte as a legacy encoding (CP1252 and symbols for the C0 controls):

Encoding Representation (hexadecimal) Representation (decimal) Bytes as CP1252 characters
UTF-8[t 1] EF BB BF 239 187 191 
UTF-16 (BE) FE FF 254 255 þÿ
UTF-16 (LE) FF FE 255 254 ÿþ
UTF-32 (BE) 00 00 FE FF 0 0 254 255 ␀␀þÿ (␀ refers to the ASCII null character)
UTF-32 (LE) FF FE 00 00 255 254 0 0 ÿþ␀␀ (␀ refers to the ASCII null character)
UTF-7[t 1] 2B 2F 76 38
2B 2F 76 39
2B 2F 76 2B
2B 2F 76 2F
[t 2]
2B 2F 76 38 2D[t 3]
43 47 118 56
43 47 118 57
43 47 118 43
43 47 118 47
43 47 118 56 45
+/v8
+/v9
+/v+
+/v/
+/v8-
UTF-1[t 1] F7 64 4C 247 100 76 ÷dL
UTF-EBCDIC[t 1] DD 73 66 73 221 115 102 115 Ýsfs
SCSU[t 1] 0E FE FF[t 4] 14 254 255 ␎þÿ (␎ represents the ASCII "shift out" character)
BOCU-1[t 1] FB EE 28 251 238 40 ûî(
GB-18030[t 1] 84 31 95 33 132 49 149 51 „1•3

See also

References

External links

This article was sourced from Creative Commons Attribution-ShareAlike License; additional terms may apply. World Heritage Encyclopedia content is assembled from numerous content providers, Open Access Publishing, and in compliance with The Fair Access to Science and Technology Research Act (FASTR), Wikimedia Foundation, Inc., Public Library of Science, The Encyclopedia of Life, Open Book Publishers (OBP), PubMed, U.S. National Library of Medicine, National Center for Biotechnology Information, U.S. National Library of Medicine, National Institutes of Health (NIH), U.S. Department of Health & Human Services, and USA.gov, which sources content from all federal, state, local, tribal, and territorial government publication portals (.gov, .mil, .edu). Funding for USA.gov and content contributors is made possible from the U.S. Congress, E-Government Act of 2002.
 
Crowd sourced content that is contributed to World Heritage Encyclopedia is peer reviewed and edited by our editorial staff to ensure quality scholarly research articles.
 
By using this site, you agree to the Terms of Use and Privacy Policy. World Heritage Encyclopedia™ is a registered trademark of the World Public Library Association, a non-profit organization.
 


Copyright © World Library Foundation. All rights reserved. eBooks from Project Gutenberg are sponsored by the World Library Foundation,
a 501c(4) Member's Support Non-Profit Organization, and is NOT affiliated with any governmental agency or department.