There was a period when sending a text file to another country was genuinely unreliable. Not slow, not expensive — unreliable, in the sense that the recipient might open it and find gibberish, with no way to tell what had gone wrong. This is the story of why that happened, and of the fix that is now so complete most people have never noticed it exists.
The problem in one sentence
Every region had its own character set, the same byte meant a different letter in each, and a file carried no record of which one it used. Software had to guess, and a wrong guess produced unreadable text with no error and no clue.
Where it started
ASCII, standardised in 1963, used 7 bits — 128 characters. That covered the unaccented Latin alphabet, digits, punctuation and some control codes. It was designed for American teleprinters and it did that job well.
Computers store 8-bit bytes, so ASCII left 128 values unused. Everyone helped themselves.
| Character set | Covered | Used by |
|---|---|---|
| ISO-8859-1 (Latin-1) | Western European | Early web default |
| ISO-8859-2 | Central European | Poland, Czechia, Hungary |
| ISO-8859-5 | Cyrillic | Some Russian systems |
| KOI8-R | Cyrillic, differently | Most Russian systems |
| Windows-1252 | Latin-1 plus extras | Windows, Western Europe |
| Windows-1251 | Cyrillic, differently again | Windows, Russia |
| Shift-JIS | Japanese | Japan |
| EUC-JP | Japanese, differently | Japanese Unix |
| Big5 | Traditional Chinese | Taiwan, Hong Kong |
| GB2312 | Simplified Chinese | Mainland China |
| EUC-KR | Korean | Korea |
Note that Cyrillic and Japanese each got several incompatible encodings. Not because anyone wanted that, but because different vendors and standards bodies solved the same problem independently and nobody had authority to pick a winner.
🚨 The fatal flaw: no self-description
A text file is a sequence of bytes with no header, no metadata and no marker. The byte 0xE9 is é in Latin-1, Й in Windows-1251, and part of a multi-byte sequence in Shift-JIS. The file cannot tell you which was meant.
Every program had to guess, usually from the system's locale — which is why a document was readable on the machine that made it and nowhere else. This is where the Japanese word mojibake, "character transformation", entered technical English.
The consequences were not academic. Databases silently corrupted names. Email systems mangled subject lines. Websites rendered as symbol soup unless the reader manually tried encodings from a menu — a menu browsers had to ship precisely because guessing failed so often. Building software for more than one language region meant maintaining separate builds.
The proposal
Work began in the late 1980s at Xerox and Apple, and the Unicode Consortium formed in 1991. The idea was simple to state and enormous to execute: give every character in every writing system one unique number, independent of any encoding, any font and any platform.
The separation is the important design choice. Unicode assigns code points — U+0041 is Latin capital A, permanently and everywhere. How that number becomes bytes is a separate question, answered by encodings such as UTF-8 and UTF-16. Splitting identity from representation is what allows the same text to be stored several ways without ambiguity about what it says.
The 16-bit mistake
The original design used 16 bits — 65,536 code points. The founders had calculated that this covered every character in modern use, and 16 bits was a convenient machine word.
It was not enough. Historic scripts, rare CJK characters used in names and classical texts, mathematical notation, and eventually emoji all needed space. In 1996 Unicode 2.0 expanded the range to 1,114,112 code points.
By then several major systems had committed to the 16-bit assumption. Windows NT, Java and JavaScript had all built their string types around it, and none could change without breaking everything already written. The compromise was surrogate pairs: characters above U+FFFF are encoded as two 16-bit units drawn from a reserved range.
UTF-8, sketched on a placemat
Unicode defined the numbers. Turning them into bytes still needed an encoding, and the obvious candidate — two bytes per character — had serious problems for existing systems.
In September 1992, Ken Thompson and Rob Pike designed UTF-8. The story, which both have recounted, is that Thompson worked it out over dinner at a New Jersey diner, sketching it on a placemat, and they implemented it in Plan 9 within days.
The design satisfies a remarkable set of constraints simultaneously:
| Property | Why it mattered |
|---|---|
| ASCII is unchanged | Every existing ASCII file is already valid UTF-8. No conversion, no migration. |
| No null bytes | C string functions keep working. Decades of code did not need rewriting. |
| No byte order issue | Byte sequences, not 16-bit words — so no big-endian and little-endian variants. |
| Self-synchronising | From any byte you can find the character boundary, so a corrupt byte costs one character rather than the rest of the file. |
| Sorts correctly | Byte-order comparison matches code point order. |
| Efficient for Latin text | One byte per character, versus two in UTF-16. |
✅ Backward compatibility won
UTF-8's decisive advantage was needing no flag day. A system could adopt it incrementally, and every ASCII file it already held was correct UTF-8 the moment it switched. UTF-16 required converting everything at once.
The adoption curve reflects that. UTF-8 was around 1% of web pages in 2001 and is now well over 98% — one of the most complete technology transitions in computing history, achieved with no coordinated migration because none was needed.
Han unification
The most contested decision in Unicode's design, and worth understanding because it shows the genuine difficulty of the problem.
Chinese, Japanese and Korean share thousands of characters descended from Chinese. Encoding each language's set separately would have consumed a large fraction of the original 65,536 code points on characters that are historically the same.
Unicode unified them: characters with a common origin got one code point, even where regional printed forms differ in stroke details.
The objection is substantial. A unified code point may be drawn differently in Japanese and Chinese typography, so correct rendering requires knowing the language of the text — information the character encoding deliberately does not carry. Plain text becomes ambiguous in a way it is not for Latin scripts. Personal and place names, where the exact form is not merely stylistic, were a particular grievance.
In practice it is handled with language tagging — the lang attribute in HTML — and by variation selectors for cases where the distinction genuinely matters. It works, and it remains an asymmetry: Latin-script text is self-sufficient in a way CJK text is not.
Emoji, and how they arrived
Emoji were not designed for Unicode. They came from Japanese mobile carriers in the late 1990s, each with its own incompatible set — so a picture sent from a DoCoMo phone arrived as something else, or nothing, on a Softbank phone. Exactly the code page problem again, one generation later.
When smartphones needed to interoperate globally, the carriers' sets had to be reconciled. Unicode 6.0 in 2010 encoded them, and emoji entered the standard not because anyone planned it but because the alternative was another decade of mojibake.
The consequence was a large increase in public attention. Unicode had operated for two decades as invisible infrastructure; now its annual release schedule is news, and the Consortium fields proposals from the public. The work of encoding endangered historic scripts continues alongside, funded partly by the attention emoji brought.
What we still live with
| Legacy | Cause |
|---|---|
"👍".length === 2 | UTF-16 surrogate pairs from the 16-bit era |
MySQL utf8 is not UTF-8 | Defined as 3 bytes max before 4-byte characters were common |
| Byte order marks in files | UTF-16's endianness problem, applied where it is unnecessary |
| Windows APIs in UTF-16 | Windows NT adopted Unicode before UTF-8 existed |
Two ways to write é | Compatibility with older standards that had both |
| Latin-1 still assumed | Decades of software defaults |
Each of these is a decision that was reasonable when made and became permanent because too much depended on it. That is the recurring pattern in this history — and it is worth noting that UTF-8, the piece designed with backward compatibility as the first requirement, is the piece that caused no lasting problems.
What it achieved
It is easy to miss the scale of this because it worked. Consider what is now unremarkable:
- A web page can contain Arabic, Chinese, Cyrillic, Devanagari and Latin text in the same paragraph.
- A filename in Thai transfers to a machine in Brazil and stays a filename in Thai.
- A database column holds names in any script without configuration.
- A search engine indexes every language with one pipeline.
- Software ships in one build for the entire world.
None of that was possible in 1990. Every item required either a separate build per region or simply could not be done. The internet as a genuinely global medium — rather than a set of regional networks that exchanged text badly — depends on a decision to give every character one number.
It is infrastructure of the best kind: enormously difficult, universally relied upon, and almost entirely invisible.
Working with text encoding?
Encode and decode URLs and Base64, or convert between text formats — all in your browser, with no upload.
Open the URL Encoder →Summary
- Before Unicode, the same byte meant different letters in different regions, and files could not declare which.
- Unicode separates identity from encoding — code points are permanent, encodings are a storage choice.
- The original 16-bit design was too small, and surrogate pairs are the visible scar.
- UTF-8 was designed in 1992 by Thompson and Pike, reportedly on a diner placemat.
- It won on backward compatibility — ASCII files are already valid UTF-8.
- Han unification saved code points at the cost of making rendering language-dependent.
- Emoji entered the standard to end a second code page war between Japanese carriers.
- UTF-8 went from 1% to over 98% of the web with no coordinated migration.
Frequently Asked Questions
What problem did Unicode actually solve?
Before it, every language region used its own character set mapping the same byte values to different letters. A document carried no indication of which set it used, so software had to guess — and a wrong guess turned readable text into nonsense. Unicode gave every character in every script one unambiguous number.
Why was Unicode originally 16-bit?
The founders calculated that 65,536 code points would cover every character in modern use, and 16 bits was a natural size for the hardware of 1991. The estimate proved too low once historic scripts, rare CJK characters and eventually emoji were added, so Unicode expanded to over a million code points in 1996 — after several systems had already committed to 16 bits.
Why did UTF-8 win over UTF-16?
Because existing ASCII files are already valid UTF-8 with no conversion, it has no byte-order ambiguity, it contains no null bytes so C string handling keeps working, and it is self-synchronising — you can find character boundaries from any position. UTF-16 offers none of those properties.
What is Han unification and why is it controversial?
Unicode assigned a single code point to characters that share a historical origin across Chinese, Japanese and Korean, even where the regional printed forms differ. It saved tens of thousands of code points. Critics argue it makes correct rendering dependent on knowing the text's language, since the same code point should be drawn differently in a Japanese and a Chinese context.
Is Unicode finished?
No. New versions are released roughly annually, adding historic scripts, technical symbols and emoji. Over 150,000 characters are assigned out of a possible 1,114,112, so there is substantial room left. The encoding is stable — assigned characters never change meaning — but the catalogue keeps growing.