Unicode and text encoding guides

Trace an accent and a joined emoji through Unicode offsets

·

A single displayed character can span several Unicode code points. Inspect the character groups and bytes when an editor, search system or program reports a position that differs from the number of visible characters.

Unicode Code Point Viewer

Follow the accent and emoji in the original text

Choose Load example and Inspect code points with Source form selected. The text begins with Café followed by a space and a joined mechanic emoji, then a line break and Arabic text. The result has 16 code points, 12 graphemes, 33 UTF-8 bytes and 19 UTF-16 units.

Find U+0301. Its UTF-8 bytes are CC 81. It shares grapheme 4 with the preceding e. The joined mechanic emoji starts at UTF-8 byte offset 7 and UTF-16 unit offset 6. Its components all belong to grapheme 6.

Follow the accent and emoji in the original text
Code pointGraphemeUTF-8 byte offsetUTF-16 unit offset
U+0065, e433
U+0301, combining acute accent444
U+1F469, woman676
U+200D, zero width joiner61510

The Unicode Standard 18.0, Unicode Encoding FormsUnicode Standard Annex #29, Text Segmentation

Compare the offsets after NFC

Select NFC and inspect again. The e and combining acute accent become U+00E9, whose UTF-8 bytes are C3 A9. The same visible text now has 15 code points, 12 graphemes, 32 UTF-8 bytes and 18 UTF-16 units.

Filter by U+1F469 to see the emoji start move to offset 6 in UTF-8 and offset 5 in UTF-16. Run the inspection in Source form when you need positions in the decoded source text.

Unicode Standard Annex #15, Normalization FormsThe Unicode Standard 18.0, Unicode Encoding Forms

Record the offset units and the normalization form

Download the JSON report with your chosen normalization. It lists every scalar value in logical order. The names, blocks and categories come from the pinned Unicode 18.0.0 database. Your browser provides the grapheme boundaries.

If you need the original line-ending bytes, open the source file. A text field or clipboard can normalize pasted line endings. The report measures the decoded Unicode text from offset zero and omits any leading encoding signature.

Unicode 18.0.0 Character Database filesThe Unicode Standard 18.0, Unicode Encoding FormsWHATWG HTML Standard, textarea

Common questions

Are UTF-16 units a sequence of bytes?

Each unit is a 16-bit number. The two bytes used to store it depend on the little-endian or big-endian encoding.

How can I find a zero width joiner?

Filter by U+200D or ZERO WIDTH JOINER. The row shows its name and encoding even though the joiner takes no visible space.

Can I compare without changing the original text?

Choose Source form to report decoded input as supplied. Other forms affect the inspection and export only. The source editor keeps the original text.

Sources

Put it into practice.

Related guides