Network Communication and APIs
The bytes can then cross a storage or communication boundary.
Try it: UTF-8: code points to bytes and back
How str.encode('utf-8') turns each character's Unicode code point into 1 to 4 bytes using fixed bit templates, and how bytes.decode('utf-8') reads them back — including the invalid bytes that raise UnicodeDecodeError.
How it works
- Every character is a Unicode code point, written U+XXXX (ord() gives its number).
- The code point's size picks a template: up to U+007F → 0xxxxxxx; up to U+07FF → 110xxxxx 10xxxxxx; up to U+FFFF → 3 bytes; above → 4 bytes.
- The code point's bits fill the x's from left to right, giving the bytes.
- Decoding reads a lead byte, which says how many 10xxxxxx continuation bytes follow, and joins their payload bits.
- A byte that cannot start a character, a wrong continuation byte, an overlong or surrogate form, or missing bytes at the end raise UnicodeDecodeError (errors='strict').
Default run (14 steps): 'Aé€😀'.encode("utf-8"): 4 characters, each becomes 1–4 bytes. … 4 characters → 10 bytes: b'A\xc3\xa9\xe2\x82\xac\xf0\x9f\x98\x80'. (bytes repr shows ASCII as text and every other byte as \xNN.)
Simplified: Up to 12 characters or 16 bytes; control characters and lone surrogates are removed from the text; only the strict error handler is shown.
Loading the simulation…
Why English-Only Text Breaks Down
A program that handles only English text can work within ASCII, a limited character set covering letters, numbers, and basic punctuation. Real applications also encounter accented letters, Chinese characters, Arabic script, emoji, and text from many other languages. To handle that text correctly, a program needs Unicode and a reliable way to turn Unicode text into bytes for storage or transmission.
Unicode identifies characters. UTF-8 represents those characters as bytes when the text must be stored or transmitted.
Following a Character from Meaning to Bytes
Inside a Python program, Unicode text is handled as characters represented by Unicode code points. A code point is an abstract numerical identifier assigned to a character. That abstract identifier is not yet a sequence of bytes that a file or network connection can store or transmit. UTF-8 performs the conversion from code points to bytes. The bytes can then cross a storage or communication boundary.
Unicode and UTF-8 Compared
Unicode is the abstract character set and standard. It assigns a unique code point to every character in every language and can represent over a million characters. UTF-8 is a variable-width encoding scheme that converts those code points into bytes. The two ideas work together, but they answer different questions: Unicode asks which character is meant, while UTF-8 asks how that character is represented as bytes.
| Question | Unicode | UTF-8 |
|---|---|---|
| What does it define? | Characters and their code points | A byte representation for code points |
| What is its role? | Abstract character set and standard | Concrete encoding method |
| Where is it used? | Inside a program to represent and manipulate text | At file, network, and other storage or transmission boundaries |
Unicode and UTF-8 describe different stages of handling text.
Code Points and Variable-Width Bytes
Unicode code points are commonly written with a U+ prefix followed by hexadecimal digits. The source examples include A at U+0041, é at U+00E9, and 😀 at U+1F600. UTF-8 uses a variable number of bytes: ASCII characters use one byte, European characters use two bytes, and Asian characters use three or four bytes. This lets common English characters remain compact while still supporting international text.
Tracing the Character é
Explain the stages involved when a program handles the character é and later writes it to a file.
- 1Identify the character
The program is working with the character é, not merely with an arbitrary byte.
- 2Use its Unicode identity
Unicode assigns é the code point U+00E9. This code point identifies the character independently of the file or network representation.
- 3Encode at the boundary
When the text is written to a file, UTF-8 converts the Unicode code point into bytes. The source describes European characters such as é as using two bytes in UTF-8.
- 4Store the bytes
The resulting UTF-8 bytes are the concrete data written to storage. The same principle applies when text is sent over the internet.
Unicode supplies the character identity; UTF-8 supplies the bytes needed by the file or communication boundary.
Unicode Strings in Python
In the source material, the u prefix creates a Unicode string. It tells Python to use the Unicode type rather than the regular string type. A Unicode string is handled internally using Unicode code points, allowing the program to represent and manipulate characters from many languages.
What do you think happens?
A program creates u"hello world". Has the program necessarily produced UTF-8 bytes for a file or network connection?
Reveal answer
No, the u prefix creates Unicode text; UTF-8 is needed when the text crosses a storage or transmission boundary
The source distinguishes Unicode strings used internally in Python from UTF-8 encoding used when reading from or writing to files and networks.
Encoding at Program Boundaries
A useful boundary rule is to keep text as Unicode while the program is working with it, then use UTF-8 when the text leaves the program for a file or network. When text comes back from a file or network, it must be converted back into Unicode so the program can work with the characters. The source describes this conversion as encoding when text leaves the program and decoding when text is read back into it.
When reading or writing files containing non-English text, specify encoding='utf-8' with the open function. This tells Python which conversion to use so Unicode strings are converted to UTF-8 bytes when written and converted back to Unicode when read. The exact syntax can depend on the Python version and file operation, so consult the current Python documentation for the operation you are using.
Mistakes with Text Representations
Common mistake Treating Unicode and UTF-8 as interchangeable names
What it looks likeCalling Unicode the byte format used to send text over a network
Why it failsUnicode is the abstract character set and assigns code points. UTF-8 is the encoding method that turns those code points into bytes.
Do this insteadDescribe the text internally as Unicode and identify UTF-8 as the representation used at storage and communication boundaries.
Common mistake Assuming the u prefix creates UTF-8 bytes
What it looks likeAssuming u"hello world" is already a network payload
Why it failsThe u prefix creates a Unicode string for internal handling. It does not by itself describe the byte representation needed outside the program.
Do this insteadApply UTF-8 encoding when the text must be written to a file or sent over a network.
Common mistake Ignoring encoding for international files
What it looks likeOpening a file containing accented or non-Latin text without specifying an encoding
Why it failsPython needs an encoding choice to convert between Unicode strings and file bytes correctly.
Do this insteadSpecify encoding='utf-8' for the relevant file operation and check the current documentation for the exact syntax.
Common mistake Expecting every character to use one byte in UTF-8
What it looks likePlanning storage as if every international character had the same byte length as an ASCII character
Why it failsUTF-8 is variable-width. ASCII characters use one byte, European characters use two bytes, and Asian characters use three or four bytes.
Do this insteadRemember that UTF-8 uses different numbers of bytes depending on the character.
Boundary Decisions
For each situation, decide whether the program should primarily work with Unicode text or with UTF-8 bytes: manipulating a message inside Python, writing that message to a file, reading the file back, and sending the message over the internet. Explain what conversion occurs at each boundary.
Hints
- Unicode is used internally for representing and manipulating text.
- UTF-8 converts Unicode code points into bytes for storage and transmission.
- Reading reverses the boundary conversion by turning bytes back into Unicode text.
Choosing the Representation
A program receives international text, changes part of the message, and saves the result.
- 1Receive
The incoming file or network data is represented as bytes, so it must be decoded into Unicode text for the program to handle the characters.
- 2Manipulate
Keep the message as Unicode while changing or examining its characters inside the program.
- 3Save
Encode the Unicode text as UTF-8 when writing it to the file or sending it onward.
Use Unicode for internal text processing and UTF-8 at the input and output boundaries.
Key Takeaways
- 1Unicode is a universal character set and standard that assigns a code point to each character.
- 2UTF-8 is a variable-width encoding method that converts Unicode code points into bytes.
- 3The u prefix creates a Unicode string in the Python model described by the source material.
- 4Keep text as Unicode while processing it inside the program.
- 5Use UTF-8 when reading from or writing to files and when sending text across networks.
At a glance
- Unicode identifies characters with code points; UTF-8 encodes those code points as bytes.
- The Python u prefix creates a Unicode string for internal text handling.
- UTF-8 is necessary when text crosses a file or network boundary.
- UTF-8 is variable-width: ASCII characters use one byte, European characters use two bytes, and Asian characters use three or four bytes.
- Specify UTF-8 for file operations involving international text and apply the same boundary principle to network communication.