|Copyright||(c) The University of Glasgow 2008-2009|
Text codecs for I/O
TextEncoding is a specification of a conversion scheme between sequences of bytes and sequences of Unicode characters.
For example, UTF-8 is an encoding of Unicode characters into a sequence of bytes. The
TextEncoding for UTF-8 is
Stopped because the input contains insufficient available elements, or all of the input sequence has been successfully translated.
Stopped because the output contains insufficient free elements
Stopped because there are sufficient free elements in the output to output at least one encoded ASCII character, but the input contains an invalid or unrepresentable sequence
The Latin1 (ISO8859-1) encoding. This encoding maps bytes directly to the first 256 Unicode code points, and is thus not a complete Unicode encoding. An attempt to write a character greater than '\255' to a
Handle using the
latin1 encoding will result in an error.
The UTF-8 Unicode encoding
The UTF-8 Unicode encoding, with a byte-order-mark (BOM; the byte sequence 0xEF 0xBB 0xBF). This encoding behaves like
utf8, except that on input, the BOM sequence is ignored at the beginning of the stream, and on output, the BOM sequence is prepended.
The byte-order-mark is strictly unnecessary in UTF-8, but is sometimes used to identify the encoding of a file.
The UTF-16 Unicode encoding (a byte-order-mark should be used to indicate endianness).
The UTF-16 Unicode encoding (litte-endian)
The UTF-16 Unicode encoding (big-endian)
The UTF-32 Unicode encoding (a byte-order-mark should be used to indicate endianness).
The UTF-32 Unicode encoding (litte-endian)
The UTF-32 Unicode encoding (big-endian)
The Unicode encoding of the current locale
The Unicode encoding of the current locale, but allowing arbitrary undecodable bytes to be round-tripped through it.
TextEncoding is used to decode and encode command line arguments and environment variables on non-Windows platforms.
On Windows, this encoding *should not* be used if possible because the use of code pages is deprecated: Strings should be retrieved via the "wide" W-family of UTF-16 APIs instead
The Unicode encoding of the current locale, but where undecodable bytes are replaced with their closest visual match. Used for the
CString marshalling functions in Foreign.C.String
An encoding in which Unicode code points are translated to bytes by taking the code point modulo 256. When decoding, bytes are translated directly into the equivalent code point.
This encoding never fails in either direction. However, encoding discards information, so encode followed by decode is not the identity.
Look up the named Unicode encoding. May fail with
isDoesNotExistErrorif the encoding is unknown
The set of known encodings is system-dependent, but includes at least:
There is additional notation (borrowed from GNU iconv) for specifying how illegal characters are handled:
UTF-8//IGNORE, will cause all illegal sequences on input to be ignored, and on output will drop all code points that have no representation in the target encoding.
//TRANSLITwill choose a replacement character for illegal sequences or code points.
//ROUNDTRIPwill use a PEP383-style escape mechanism to represent any invalid bytes in the input as Unicode codepoints (specifically, as lone surrogates, which are normally invalid in UTF-32). Upon output, these special codepoints are detected and turned back into the corresponding original byte.
In theory, this mechanism allows arbitrary data to be roundtripped via a
String with no loss of data. In practice, there are two limitations to be aware of:
On Windows, you can access supported code pages with the prefix
CP; for example,
Internal encoding of argv
© The University of Glasgow and others
Licensed under a BSD-style license (see top of the page).