Text encoding guides

Convert a legacy text file and reopen its encoded bytes

·

A receipt containing a euro sign can be preserved in Windows-1252 or UTF-8 and may lose that sign in a different encoding. Compare the exported bytes and the decoded result before uploading it to an importer.

Encoding Converter

Check the euro sign in a Windows-1252 receipt

Choose Western European under Example text, select Load example, and convert. The target is Windows-1252. The download has 22 bytes. Open Show output bytes to find E9 for é, 80 for the euro sign and 0D 0A for the line break.

Select Use output as input. Source encoding becomes Windows-1252 and the target becomes UTF-8. Convert again. With the byte-order mark option off, the file has 26 bytes and the euro sign is now E2 82 AC. The same text appears in both decoded fields.

Check the euro sign in a Windows-1252 receipt
CharacterWindows-1252UTF-8
éE9C3 A9
€80E2 82 AC
CRLF0D 0A0D 0A

iconv-lite 0.6.3 codec documentationWHATWG Windows-1252 byte indexRFC 3629, UTF-8 byte sequence syntax

Review the difference when a target changes the text

With the example loaded, choose literal ISO-8859-1 as the target. The converter stops when it cannot preserve the euro sign. Choose UTF-8 if you need every character. When the importer requires Latin-1, enable Allow target substitutions and review the question mark in the decoded output.

To confirm the difference between source code pages, paste 80 as hexadecimal bytes and decode it with Windows-1252, then literal ISO-8859-1. Windows-1252 yields a euro sign. The literal Latin-1 byte represents the control character U+0080.

WHATWG Windows-1252 byte indexWHATWG Encoding Standard, names and labels

Select a byte order or a region-specific encoding

For a marked UTF-16 file, select Local file and leave the source on Auto. Choose UTF-16BE and include a byte-order mark if the receiving program requires big-endian UTF-16. Its first two bytes should be FE FF. Use a known source selection when a file has no marker or supports more than one interpretation.

The Japanese, Korean and Chinese options identify the mapping family in the selector. The pinned codec includes CP932 mappings for Shift_JIS, HKSCS for Big5 and CP949 for EUC-KR. Compare a small file in the application that will receive it when a system uses a particular code page.

iconv-lite 0.6.3 codec documentationWHATWG Encoding Standard, names and labels

Common questions

Do encoding guesses change the file automatically?

Unicode signatures and a valid UTF-8 reading can select a source. Other evidence is shown for review and you make the source selection.

Why can two outputs contain different bytes for the same text?

Each target maps characters into its own byte sequences. Compare decoded text to verify preservation and check raw bytes to verify the format.

Sources

Put it into practice.

Related guides