Character Encoding: Unicode and UTF-8 Explained

You've seen it: a name that should read José shows up as José turning into José, or an emoji becomes a box of question marks. That garble has a name — mojibake — and it's almost always a character-encoding mismatch. This guide explains what characters, Unicode, and UTF-8 really are, so those bugs stop being mysterious.

Computers store numbers, not letters

At the lowest level a computer only stores numbers (bytes). To store text, we need an agreed-upon table that maps each character to a number, and back again. That table is a character encoding. If one program writes text using one table and another reads it using a different table, the numbers get interpreted as the wrong characters — and you get mojibake.

ASCII: the small beginning

The original widespread encoding, ASCII, defined numbers for 128 characters: the English alphabet, digits, punctuation, and some control codes. That's enough for English but has no room for é, ü, ß, 你, 😀, or the tens of thousands of characters used by the world's languages. For decades everyone invented their own incompatible extensions, which is precisely why text broke when it crossed between systems.

Unicode: one number for every character

Unicode is the universal solution: a single standard that assigns a unique number — called a code point — to every character in every writing system, plus symbols and emoji. Code points are written as U+ followed by a hex number:

A   →  U+0041
é   →  U+00E9
你  →  U+4F60
😀  →  U+1F600

Crucially, Unicode defines the numbers but not how to store them as bytes. That's a separate decision — and it's where UTF-8 comes in.

UTF-8: how Unicode becomes bytes

UTF-8 is a way of encoding Unicode code points into bytes, and it's the one the web overwhelmingly uses. Its clever design is that it's variable-length:

• The 128 ASCII characters take 1 byte each — so UTF-8 is backward-compatible with ASCII.
• Most Latin-with-accents and Greek/Cyrillic characters take 2 bytes.
• Most other scripts, including Chinese, Japanese, and Korean, take 3 bytes.
• Emoji and rarer characters take 4 bytes.

This is why a “100-character” string isn't always 100 bytes: one character can be several bytes. It also explains the classic José bug — that's UTF-8 bytes for é being mistakenly read one byte at a time as an older Western encoding.

Characters, code points, and bytes

Three related-but-different concepts trip people up constantly:

• Character — what a human sees (the letter é).
• Code point — the Unicode number for it (U+00E9).
• Byte — how it's stored (in UTF-8, the two bytes 0xC3 0xA9).

Seeing a code point spelled out is often the fastest way to diagnose a text problem — for example, a “space” that turns out to be a non-breaking space (U+00A0). Our Unicode converter shows the code points behind any text and converts between characters and escape sequences.

How to avoid mojibake

• Use UTF-8 everywhere. Save files as UTF-8 and declare it — on web pages, <meta charset="utf-8">; in HTTP, the Content-Type charset; in databases, a UTF-8 collation.
• Keep it consistent end to end. Most corruption happens at a boundary where one side assumes a different encoding.
• Escape for the right context. To put special characters safely into HTML, use HTML entities; to put them into a URL, use URL encoding.

The short version: Unicode gives every character a number, UTF-8 turns those numbers into bytes, and if everyone agrees to use UTF-8, text stops breaking.