Text encoding guides

Find the damaged bytes in a UTF-8 file

·

A text editor can show a replacement diamond when it reads malformed bytes. Inspect the original data to locate the damage and see which characters the UTF-8 decoder can still read.

UTF-8 Validator

Locate the three invalid spans in the sample

Load example and select Validate UTF-8. The result has 20 source bytes, 15 valid bytes and three invalid spans. The first failure starts at byte offset 9. Its prefix E2 82 expects one more continuation, and 21 at offset 11 is a standalone exclamation point.

Open Errors and select byte offset 9. The byte view shows E2 and 82 as invalid bytes and 21 as ASCII. The leading EF BB BF at offsets 0 through 2 stays in the original offset count.

Hexadecimal bytes for the worked validation
EF BB BF 43 61 66 C3 A9 20 E2 82 21 20 F0 9F A7 AD 80 E2 82
Locate the three invalid spans in the sample
OffsetRejected spanWhat happened
9E2 82The following 21 belongs to a separate character.
1780A continuation arrived without a lead byte.
18E2 82The file ended before the last continuation.

RFC 3629, syntax of UTF-8 byte sequencesWHATWG Encoding Standard, UTF-8 decoder

Compare the decoded copy with the original bytes

Switch to Decoded text. Café, the exclamation point and the compass emoji remain readable. There is one U+FFFD for the rejected prefix before the exclamation point and two after the compass.

Download JSON to save the offsets and evidence. Download decoded TXT to get a valid UTF-8 copy. Each new replacement character encodes as EF BF BD. Keep the original file when you need to recover the intended bytes.

WHATWG Encoding Standard, UTF-8 decoder

Check what the bytes can tell you about their encoding

Use Local file to test the original export or Hexadecimal bytes for an exact recorded sample. Pasting into a text field uses characters already decoded by the browser. It can show what those characters encode to in UTF-8.

A clear byte-order signature can suggest a different encoding. An unmarked file can produce tentative model scores or ambiguous evidence. Check the producing application’s settings and compare decodings before choosing an encoding.

RFC 3629, syntax of UTF-8 byte sequencesWHATWG Encoding Standard, UTF-8 decoder

Common questions

Do offsets count printed characters?

They count original bytes from zero. A UTF-8 emoji can occupy four bytes, so later offsets may exceed the number of visible characters.

Does a short error list mean every byte was checked?

All bytes are checked and classified. The detailed error list and JSON display at most 2,500 spans. The byte view remains available for the full input.

Sources

Put it into practice.

Related guides