How Unicode and UTF-8 work

A computer stores only numbers. To store text, it needs an agreement about which number stands for which character, and how those numbers are written as bytes. When two programs disagree, text turns into nonsense: the word “café” saved by one program can appear as “café” in another. This article explains the agreement almost all software uses today, Unicode, and the way of writing it as bytes that dominates the web, UTF-8, and shows exactly where “é” comes from.

From ASCII to Unicode

For decades, the basis was ASCII, a 7-bit code with 128 values for the English alphabet, digits, punctuation and control characters. In 1969, RFC 20 proposed it for the early ARPANET: “standard 7-bit ASCII embedded in an 8 bit byte whose high order bit is always 0.” The capital letter A is 65, or 41 in hexadecimal.

The eighth bit left room for 128 more characters, and many incompatible encodings filled it in different ways, one for Western European languages, others for Cyrillic, Greek or Hebrew, and multi-byte systems for Chinese, Japanese and Korean. The same byte meant different characters depending on the encoding, and a document could not easily mix scripts. The Encoding Standard, which defines how browsers handle text, still has to describe dozens of these legacy encodings.

Unicode replaced them with a single list. It gives every character a number, its code point, written as U+ followed by at least four hexadecimal digits: A is U+0041, é is U+00E9, the euro sign € is U+20AC. Unicode keeps ASCII’s numbers for the first 128 characters.

Code points and planes

The Unicode codespace runs from U+0000 to U+10FFFF: 1,114,112 code points. It is divided into 17 planes of 65,536 code points each:

• Plane 0, the Basic Multilingual Plane (BMP), holds the common characters of all modern scripts. Most text uses nothing else.
• Plane 1, the Supplementary Multilingual Plane, holds historic scripts, notations and large symbol sets, including emoji such as 😀 (U+1F600, GRINNING FACE).
• Planes 2 and 3 hold additional, mostly rare, CJK ideographs.
• Plane 14 holds special-purpose format characters, and planes 15 and 16 are for private use. All other planes are reserved.

Unicode 18.0, released in September 2026, encodes 172,808 characters. New versions add characters, but once a character is assigned, its code point cannot be changed and the character cannot be removed.

From code points to bytes

A code point is still just a number. To store or send it, it must be written as code units of a fixed size. Unicode defines three encoding forms, all able to represent every character:

UTF-32 is the simplest, one unit per code point, but uses four bytes even for A. UTF-16 uses one 16-bit unit for every character in the BMP and two for the rest. UTF-8 uses one to four bytes.

How UTF-8 works

UTF-8 was designed in September 1992 by Ken Thompson, following design criteria from Rob Pike, for the Plan 9 operating system, according to RFC 3629, which specifies it for the internet. Its rules fit in one table:

The x positions are filled with the bits of the code point, from the right. For the euro sign, U+20AC needs three bytes, which have room for 16 bits:

This simple scheme has valuable properties, listed in RFC 3629:

• ASCII stays ASCII. Every ASCII character is the same single byte, and the bytes 00 to 7F never appear inside another character, so ASCII-based software keeps working.
• The first byte gives the length. Its leading 1 bits count the bytes of the character, and every following byte starts with 10.
• Boundaries are easy to find from anywhere in the text: a reader that lands in the middle of a character skips bytes starting with 10 until the next character begins.
• Some bytes never appear: C0, C1 and F5 to FF. Text in another encoding is therefore unlikely to be valid UTF-8 by accident.
• Sorting is preserved. Sorting UTF-8 strings byte by byte gives the same order as sorting by code point.

UTF-16 and surrogate pairs

UTF-16 descends from the earliest form of Unicode, which was designed as a fixed-width 16-bit encoding. To reach the other planes, Unicode set aside 2,048 code points, U+D800 to U+DFFF, called surrogates, which are never assigned to characters. A character outside the BMP is written as a surrogate pair: a high surrogate followed by a low surrogate, which together carry 20 bits.

Because UTF-16 and UTF-32 use multi-byte units, their bytes can be stored in two orders. A byte order mark (BOM), the character U+FEFF at the start of a text, shows which order was used. UTF-8 has no byte order question.

One character, several code points

What a reader sees as one character is not always one code point. Many letters can be written in two ways: é is the single code point U+00E9, but also the letter e followed by U+0301 COMBINING ACUTE ACCENT. Both look identical, yet their bytes differ:

é as one code point    U+00E9          → UTF-8: C3 A9
é as e + accent        U+0065 U+0301   → UTF-8: 65 CC 81

Unicode treats such sequences as equivalent and defines normalization forms to make them identical: NFC composes characters where possible, and NFD decomposes them. Programs that compare text can normalize it first, so that such invisible differences disappear.

Unicode calls what a user perceives as a single character a grapheme cluster: a base character together with the marks that belong to it. Counting code points therefore does not tell you how many characters a reader sees, and cutting a string between code points can split a letter from its accent.

Invalid UTF-8 is a security problem

Each code point has exactly one valid UTF-8 encoding. A sloppy decoder, however, might also accept an overlong form, such as C0 AF for the slash /, which really is 2F. RFC 3629 describes why this matters: a server that blocks the byte sequence for /../ in file paths, but whose decoder accepts the overlong C0 AE for the dot, can be tricked into reading files outside its folder. RFC 3629 notes that this exact exploit “has actually been used in a widespread virus attacking Web servers in 2001”. Decoders must therefore reject every invalid sequence instead of guessing.

UTF-8 on the web

The web has settled on UTF-8. The Encoding Standard, which browsers follow, states that “the UTF-8 encoding is the most appropriate encoding for interchange of Unicode”, requires it for new protocols and formats, and says that authors must use it.

It also explains the “café” from the beginning. Browsers treat the labels latin1, iso-8859-1, ascii and us-ascii all as the encoding windows-1252, which maps every byte to one character. If the UTF-8 bytes of é, C3 A9, are read as windows-1252, C3 becomes à and A9 becomes ©. Garbled text like this is a sign that it was decoded with a different encoding than it was written in.

Key points

• Unicode gives every character a code point, from U+0000 to U+10FFFF, in 17 planes. Version 18.0 (2026) contains 172,808 characters.
• An encoding form turns code points into code units: UTF-8 (1 to 4 bytes), UTF-16 (one or two 16-bit units, with surrogate pairs) or UTF-32 (one 32-bit unit).
• UTF-8 keeps ASCII unchanged, marks every byte as a start or continuation byte, and is the required encoding for new formats on the web.
• One visible character can be several code points; normalization (NFC, NFD) makes equal text compare equal, and grapheme clusters describe what readers see as one character.
• Decoders must reject invalid and overlong UTF-8, which has been used to bypass security checks.