Concepts / Reading Files with UTF-8 Encoding

Reading Files with UTF-8 Encoding

The important boundary is between Unicode text in Python and bytes in the file. With UTF-8 selected, Python handles the conversion instead of requiring you to convert each character manually.

  • Programming
Interactive lab

Try it: UTF-8: code points to bytes and back

How str.encode('utf-8') turns each character's Unicode code point into 1 to 4 bytes using fixed bit templates, and how bytes.decode('utf-8') reads them back — including the invalid bytes that raise UnicodeDecodeError.

How it works

  1. Every character is a Unicode code point, written U+XXXX (ord() gives its number).
  2. The code point's size picks a template: up to U+007F → 0xxxxxxx; up to U+07FF → 110xxxxx 10xxxxxx; up to U+FFFF → 3 bytes; above → 4 bytes.
  3. The code point's bits fill the x's from left to right, giving the bytes.
  4. Decoding reads a lead byte, which says how many 10xxxxxx continuation bytes follow, and joins their payload bits.
  5. A byte that cannot start a character, a wrong continuation byte, an overlong or surrogate form, or missing bytes at the end raise UnicodeDecodeError (errors='strict').

Default run (14 steps): 'Aé€😀'.encode("utf-8"): 4 characters, each becomes 1–4 bytes. … 4 characters → 10 bytes: b'A\xc3\xa9\xe2\x82\xac\xf0\x9f\x98\x80'. (bytes repr shows ASCII as text and every other byte as \xNN.)

Simplified: Up to 12 characters or 16 bytes; control characters and lone surrogates are removed from the text; only the strict error handler is shown.

Educational simulation

Loading the simulation…

From Text to Stored Bytes

A Python string can contain characters such as é, €, 中, and こんにちは. A file system stores file data as bytes, so Python must translate the Unicode characters in the string into bytes before storing them. This translation is called encoding. UTF-8 performs that translation automatically when you open the file with encoding='utf-8'.

Unicode stringcaféUTF-8 encodingcharacter conversionByte sequencestored sequentiallyFiledata on diskwrite textconvertstore
How does a Unicode string such as café or 你好 become UTF-8 bytes before those bytes are stored in a file?
Key point

The important boundary is between Unicode text in Python and bytes in the file. With UTF-8 selected, Python handles the conversion instead of requiring you to convert each character manually.

The UTF-8 Conversion Step

When f.write() is called on a file opened with encoding='utf-8', Python converts each Unicode character into one or more bytes according to the UTF-8 encoding scheme. Characters do not all require the same number of bytes. For example, the Euro sign € has Unicode code point U+20AC and is represented in UTF-8 by the three bytes 0xE2, 0x82, and 0xAC. Those bytes are stored sequentially in the file.

Worked example

Encoding the Euro Sign

What happens when a UTF-8 text file receives the character €?

  1. 1
    Start with Unicode text

    The Python string contains the Unicode character €.

  2. 2
    Select UTF-8

    The file was opened with encoding='utf-8', so Python uses UTF-8 for the text-to-byte conversion.

  3. 3
    Convert the character

    The character € is represented in UTF-8 as the three bytes 0xE2, 0x82, and 0xAC.

  4. 4
    Store the bytes

    The three bytes are written sequentially to the file.

Result

The Unicode character is preserved in the file as its UTF-8 byte sequence.

€Unicode character in astring0xE2 0x82 0xACUTF-8 bytes in a fileUTF-8 encoding
What is the difference between the Unicode characters represented by a string literal in memory and the UTF-8 byte sequence written to the file?

Opening a Text File

Code walkthrough
import io

with io.open("messages.txt", mode="wt", encoding="utf-8") as f:
    f.write("Français\n")
    f.write("こんにちは\n")
    f.write("€")
  1. 1
    Python imports the io module.
  2. 2
    io.open opens messages.txt in text-writing mode and selects UTF-8.
  3. 3
    The first write adds a line containing the accented character é.
  4. 4
    The second write adds a line containing Japanese characters.
  5. 5
    The third write adds the Euro sign.
  6. 6
    When the with block ends, the file is closed automatically.
  • The io module provides the open function used here.
  • The mode wt opens the file for writing text.
  • The encoding parameter selects UTF-8 for conversion.
  • Each write call adds text to the file.
  • The with statement handles closing the file after the block.
Output
The file contains:
Français
こんにちは
€
Python programio.openwt, UTF-8f.writeUnicode textFile closedwith block endsopen filereturn file objectfinish with block
What happens step by step when io.open is called with text-writing mode, text is written, and the file is closed?

Encoding Parameter Control

The encoding='utf-8' argument tells io.open how Unicode text should be translated before it is written. Without an explicitly selected encoding, the behavior may not be consistent across systems. Specifying UTF-8 makes the intended conversion clear and helps preserve non-ASCII characters so that other programs can read them correctly when they understand UTF-8.

Unicode textin memoryencoding='utf-8'conversion ruleUTF-8 bytesone or more per characterStored file databyteswrite textselect UTF-8write bytes
How does encoding='utf-8' control the conversion between in-memory Unicode text and stored file bytes?

Unicode String Literals

In Python 3, strings are Unicode by default. A u prefix, as in u"Français", is optional but can clarify that the value is intended as Unicode text. The prefix was more significant in Python 2, where regular strings were byte strings by default. In Python 3, these two forms express the same Unicode-string intent.

String formMeaning in Python 3Use
"Français"A Unicode stringNormal Python 3 spelling
u"Français"A Unicode stringOptional clarity and compatibility notation

The u prefix is optional in Python 3, while both forms represent Unicode text.

Safe File Cleanup

A file should be closed after writing. You can close it explicitly, but a with statement is the safer and cleaner approach. The context manager automatically closes the file when the block ends, including when an error occurs during writing.

Before
import io

f = io.open("messages.txt", mode="wt", encoding="utf-8")
f.write("Français")
f.close()

The program must explicitly close the file.

After
import io

with io.open("messages.txt", mode="wt", encoding="utf-8") as f:
    f.write("Français")

The with statement closes the file automatically when the block ends.

Key point

Use the with form as the default pattern for writing text files. It keeps the opening, writing, and cleanup steps together and guarantees cleanup through the context manager.

When Encodings Disagree

Encoding problems become easier to investigate when you trace the complete path: Unicode text in the Python string, the encoding selected when the file is written, the bytes stored in the file, and the encoding understood by the program or editor that reads it. If the intended UTF-8 encoding is not used consistently, non-ASCII characters may not appear correctly in the output.

Unicode texté or 中Unicode texté or 中UTF-8 writeexplicit encodingUnclear encodingsettings disagreeUTF-8 readercharacters preservedOutput displaymay not appear correctlyconvertread consistentlyconvertread with disagreement
What changes in the data flow when the source text, encoding parameter, and file-reading encoding do not agree, and where can the visible problem appear?
  • Common mistake

    Opening a text file without explicitly specifying UTF-8

    What it looks like

    Using a file-opening pattern that omits encoding='utf-8' when the text contains characters such as é or €.

    Why it fails

    The intended text encoding is not stated explicitly, so behavior may not be consistent across systems.

    Do this instead
    Use io.open with mode='wt' and encoding='utf-8'.
  • Common mistake

    Manually converting every character to bytes before writing

    What it looks like

    Trying to create byte sequences for each non-ASCII character before calling f.write().

    Why it fails

    A text file opened with encoding='utf-8' performs the Unicode-to-byte conversion automatically.

    Do this instead

    Write Unicode strings directly to the UTF-8 text file.

  • Common mistake

    Forgetting to close the file

    What it looks like

    Opening a file, writing text, and leaving the file open.

    Why it fails

    File cleanup is not completed reliably.

    Do this instead

    Use a with statement so the file closes automatically, even if an error occurs during writing.

  • Common mistake

    Assuming the u prefix is required in Python 3

    What it looks like

    Treating "こんにちは" as invalid because it does not have a u prefix.

    Why it fails

    Python 3 strings are Unicode by default.

    Do this instead

    Use the u prefix only when it improves clarity or compatibility.

Practice the Trace

EASY

Write a short Python program that opens notes.txt with io.open in text-writing mode and UTF-8 encoding. Write one line containing € and one line containing 中. Use a with statement. Then identify which step performs the conversion from Unicode characters to bytes.

Hints
  • Import io before calling io.open.
  • Use mode='wt' and encoding='utf-8'.
  • Place the write calls inside the with block.
  • The conversion occurs when text is written through the UTF-8-configured file object.

What do you think happens?

What happens to the characters in this statement when it runs: f.write("Français") on a file opened with encoding='utf-8'?

  • Python requires you to convert é to bytes manually first.

  • Python converts the Unicode characters into UTF-8 bytes automatically.

  • Python removes the non-ASCII character before writing.

Reveal answer
Answer:

Python converts the Unicode characters into UTF-8 bytes automatically.

The UTF-8 encoding selected when the file was opened controls the automatic conversion performed during f.write().

Key Takeaways

  1. 1
    Use io.open with mode='wt' and encoding='utf-8' to write Unicode text to a file.
  2. 2
    UTF-8 converts each Unicode character into one or more bytes, which are stored sequentially.
  3. 3
    The encoding parameter selects the text-to-byte conversion used during f.write().
  4. 4
    Python 3 strings are Unicode by default, so the u prefix is optional.
  5. 5
    Use a with statement to close the file automatically and consistently.

At a glance

  • Open text files with io.open, mode='wt', and encoding='utf-8'.
  • Python automatically encodes Unicode strings as UTF-8 bytes during writing.
  • A single Unicode character can become multiple UTF-8 bytes, such as € becoming 0xE2, 0x82, and 0xAC.
  • Specify UTF-8 explicitly and use a with statement for predictable encoding and safe cleanup.
  • When characters display incorrectly, trace the text, writing encoding, stored bytes, and reading encoding for consistency.

Related Topics

More concepts starting with R