Understanding Unicode and UTF-8 Encoding

Unicode is a universal character encoding standard that assigns a unique code point to every character in every language, enabling programs to handle international text.

Programming

Interactive lab

Try it: UTF-8: code points to bytes and back

How str.encode('utf-8') turns each character's Unicode code point into 1 to 4 bytes using fixed bit templates, and how bytes.decode('utf-8') reads them back — including the invalid bytes that raise UnicodeDecodeError.

How it works

Every character is a Unicode code point, written U+XXXX (ord() gives its number).
The code point's size picks a template: up to U+007F → 0xxxxxxx; up to U+07FF → 110xxxxx 10xxxxxx; up to U+FFFF → 3 bytes; above → 4 bytes.
The code point's bits fill the x's from left to right, giving the bytes.
Decoding reads a lead byte, which says how many 10xxxxxx continuation bytes follow, and joins their payload bits.
A byte that cannot start a character, a wrong continuation byte, an overlong or surrogate form, or missing bytes at the end raise UnicodeDecodeError (errors='strict').

Default run (14 steps): 'Aé€😀'.encode("utf-8"): 4 characters, each becomes 1–4 bytes. … 4 characters → 10 bytes: b'A\xc3\xa9\xe2\x82\xac\xf0\x9f\x98\x80'. (bytes repr shows ASCII as text and every other byte as \xNN.)

Simplified: Up to 12 characters or 16 bytes; control characters and lone surrogates are removed from the text; only the strict error handler is shown.

Educational simulation

Loading the simulation…

Beyond English Text

A program that handles only English text can work within the ASCII character set, which covers letters, numbers, and basic punctuation. Real applications often encounter accented letters, Chinese characters, Arabic script, emoji, and other characters that ASCII cannot represent. Supporting this wider range requires a way to identify characters consistently and a way to store or transmit them as bytes.

Key point

Unicode answers the question, “Which character is this?” UTF-8 answers the question, “Which bytes represent that character for storage or transmission?”

Character Identity and Byte Representation

Unicode is a universal character encoding standard that assigns a unique numerical identifier, called a code point, to every character in every language. It is the abstract character set: it describes the characters a program needs to work with and gives each one an identity. Unicode can represent over a million characters, rather than being limited to the much smaller range covered by ASCII.

UTF-8 is a variable-width encoding scheme. It converts Unicode code points into bytes so text can be stored or transmitted. The number of bytes depends on the character: ASCII characters use 1 byte, European characters use 2 bytes, and Asian characters use 3 to 4 bytes. Unicode and UTF-8 therefore describe different layers of the same text-handling process.

What is the difference between a character's Unicode identity and the UTF-8 bytes used to store or transmit it?

Tracing a Character into UTF-8

Worked example

Representing an Accented Character

Trace what happens when the character é must be stored or transmitted using UTF-8.

1
Identify the character
The text contains the character é, which is a European character outside the basic ASCII character set.
2
Use its Unicode identity
Unicode assigns the character a unique code point. At this stage, the program is working with the identity of the character rather than a storage byte sequence.
3
Apply UTF-8
UTF-8 converts the Unicode code point into bytes. According to the variable-width scheme, a European character uses 2 bytes.
4
Store or transmit
Those bytes can be used for storage or transmission. When the text is needed as characters again, the bytes must be interpreted using the appropriate text encoding.

Result

The character remains one character at the Unicode level but is represented by 2 UTF-8 bytes for storage or transmission.

How does a character such as é change as it moves from a Unicode character to stored or transmitted bytes?

Variable-Width Storage

UTF-8 is called variable-width because characters do not all occupy the same number of bytes. Basic ASCII characters use 1 byte. European characters use 2 bytes. Asian characters use 3 to 4 bytes. This allows common ASCII text to use a compact representation while still supporting characters from many languages.

Why do different characters require different numbers of UTF-8 bytes?

Unicode Strings in Python

In the Python model described here, the u prefix creates a Unicode string. For example, u"hello world" tells Python to use the Unicode type instead of the regular string type. A regular string without the prefix has type str, while a string with the prefix has type unicode.

python

regular_text = "hello world"
unicode_text = u"hello world"

print(type(regular_text))
print(type(unicode_text))

regular_text = "hello world"
unicode_text = u"hello world"

print(type(regular_text))
print(type(unicode_text))

Output

str
unicode

Key point

The u prefix identifies the text as Unicode in the Python model covered by this material. UTF-8 is still the representation to apply when that text crosses a file or network boundary as bytes.

Crossing File and Network Boundaries

Where does text become encoded into UTF-8 bytes for a file or network, and where does it become text again?

Unicode strings are used internally in Python, while UTF-8 encoding is used when reading from or writing to files and networks. The practical boundary is important: inside the program, work with text and its Unicode characters; when data must be stored or transmitted, represent that text as UTF-8 bytes. When data comes back from storage or communication, it must be interpreted as text again.

Best practice

Apply UTF-8 when your program writes text to a file, reads text from a file, sends text across a network, or receives text from a network. Pay particular attention when the data may contain languages or symbols outside ASCII.

Mistakes Beginners Make

Common mistake
Treating Unicode and UTF-8 as interchangeable terms.
What it looks like
Saying that Unicode is the byte format used to transmit text.
Why it fails
Unicode is the abstract character set with code points, while UTF-8 is the encoding scheme that converts those code points into bytes.
Do this instead
Use Unicode to discuss character identity and UTF-8 to discuss the byte representation used for storage or transmission.
Common mistake
Assuming ASCII can represent all text.
What it looks like
Designing a program around English letters and then expecting it to handle Chinese, Arabic, accented letters, or emoji automatically.
Why it fails
ASCII covers letters, numbers, and basic punctuation, but it cannot represent the full range of characters used across the world's languages.
Do this instead
Use Unicode text when the program must handle international characters.
Common mistake
Assuming every character uses one UTF-8 byte.
What it looks like
Estimating the storage size of text by counting characters only.
Why it fails
UTF-8 is variable-width: ASCII characters use 1 byte, European characters use 2 bytes, and Asian characters use 3 to 4 bytes.
Do this instead
Remember that the byte count depends on the characters in the text.
Common mistake
Using a Unicode string internally but ignoring encoding at an external boundary.
What it looks like
Sending Unicode text over a network without considering its byte representation.
Why it fails
Files and networks use stored or transmitted bytes, so text must be encoded for those boundaries.
Do this instead
Use UTF-8 encoding when writing, reading, sending, or receiving text through files and networks.

Apply the Boundary Test

MEDIUM

For each situation, decide whether you are dealing mainly with Unicode character identity, UTF-8 byte representation, or both: a Python variable containing international text; writing that variable to a file; receiving text from a network; and deciding how many bytes a European character uses.

Hints

Ask whether the situation is about a character inside the program or bytes crossing a storage or communication boundary.
Remember that UTF-8 is variable-width.

Your attempt (not saved)

Worked example

Checking the Four Situations

Classify the four situations using the Unicode and UTF-8 distinction.

1
Python variable containing international text
This is primarily Unicode text inside the program.
2
Writing the variable to a file
This crosses a storage boundary, so UTF-8 is needed to represent the text as bytes.
3
Receiving text from a network
The program receives transmitted bytes and must interpret them as text.
4
Counting bytes for a European character
This concerns UTF-8 representation, where a European character uses 2 bytes.

Result

Unicode describes the text the program works with; UTF-8 describes the bytes used when that text is stored, transmitted, or interpreted across a boundary.

Key Takeaways

1
Unicode is a universal character set that assigns a unique code point to each character.
2
UTF-8 is a variable-width encoding scheme that converts Unicode code points into bytes.
3
In the Python model covered here, the u prefix creates a Unicode string.
4
ASCII characters use 1 UTF-8 byte, European characters use 2 bytes, and Asian characters use 3 to 4 bytes.
5
Use UTF-8 when text crosses file or network boundaries.

At a glance

Unicode identifies characters across languages through code points.
UTF-8 converts those Unicode characters into variable-length byte sequences.
The u prefix creates a Unicode string in the Python model presented here.
UTF-8 becomes important when text is read from or written to files and when it is sent across networks.
Do not confuse the character-level representation inside a program with the byte-level representation used outside it.

Related concepts

More concepts starting with U