Concepts / Working with External Data Formats

Working with External Data Formats

The reverse operation happens when external data enters the program.

  • Programming
Interactive lab

Try it: UTF-8: code points to bytes and back

How str.encode('utf-8') turns each character's Unicode code point into 1 to 4 bytes using fixed bit templates, and how bytes.decode('utf-8') reads them back — including the invalid bytes that raise UnicodeDecodeError.

How it works

  1. Every character is a Unicode code point, written U+XXXX (ord() gives its number).
  2. The code point's size picks a template: up to U+007F → 0xxxxxxx; up to U+07FF → 110xxxxx 10xxxxxx; up to U+FFFF → 3 bytes; above → 4 bytes.
  3. The code point's bits fill the x's from left to right, giving the bytes.
  4. Decoding reads a lead byte, which says how many 10xxxxxx continuation bytes follow, and joins their payload bits.
  5. A byte that cannot start a character, a wrong continuation byte, an overlong or surrogate form, or missing bytes at the end raise UnicodeDecodeError (errors='strict').

Default run (14 steps): 'Aé€😀'.encode("utf-8"): 4 characters, each becomes 1–4 bytes. … 4 characters → 10 bytes: b'A\xc3\xa9\xe2\x82\xac\xf0\x9f\x98\x80'. (bytes repr shows ASCII as text and every other byte as \xNN.)

Simplified: Up to 12 characters or 16 bytes; control characters and lone surrogates are removed from the text; only the strict error handler is shown.

Educational simulation

Loading the simulation…

Why English Text Is Not Enough

A program that handles only English text can work within ASCII, a character standard covering letters, numbers, and basic punctuation. However, ASCII cannot represent many characters used around the world, including accented French letters, Chinese characters, Arabic script, and emoji. To process international text correctly, a program needs Unicode.

Key point

Unicode gives characters a universal identity. UTF-8 provides a way to turn those identities into bytes that can be stored or transmitted.

Unicodecharacter set and standardCode pointunique numerical identifierUTF-8encoding methodBytesstorage or transmissionformassignsconverts torepresented by
What is the difference between the abstract character or code point and the byte representation used by UTF-8?

From Character to Code Point

Unicode is a universal character encoding standard that assigns a unique code point to every character in every language. A code point is a numerical identifier for a character. Code points are commonly written in hexadecimal notation with a U+ prefix. For example, the letter A is U+0041, the letter é is U+00E9, and the emoji 😀 is U+1F600.

Worked example

Identifying a character

Relate the character A to its Unicode representation.

  1. 1
    Character

    Begin with the visible character A.

  2. 2
    Code point

    Unicode assigns A the unique code point U+0041.

  3. 3
    Meaning

    The code point identifies the character independently of the language or platform using it.

Result

A is represented by the Unicode code point U+0041.

AcharacterU+0041Unicode code pointUnicode textprogram representationassignedrepresented in
How does a character receive an abstract Unicode identity before it is prepared as bytes?

Writing Unicode Strings in Python

In Python, the source material creates a Unicode string with the u prefix, as in u"hello world". The prefix tells Python to use the Unicode type instead of the regular string type. A Unicode string is the form your program uses internally when it needs to represent and manipulate international text.

uUnicode prefix"hello world"string charactersUnicode stringtext used inside PythonUTF-8 bytesexternal data formindicatesformsencoded when needed
What does the u prefix indicate, and how is that value different from encoded byte data?

Encoding at the I/O Boundary

Unicode code points are abstract numbers. A file or internet connection needs concrete bytes, so a Unicode string must be encoded when it leaves the program. UTF-8 is a variable-width encoding scheme that converts Unicode code points into bytes. ASCII characters use 1 byte, European characters use 2 bytes, and Asian characters use 3 or 4 bytes.

Unicode textcharacters and code pointsUTF-8variable-width encodingByte sequenceexternal representationencodeproduces
How does a Unicode character or code point become a sequence of UTF-8 bytes when data is prepared for storage or transmission?

The reverse operation happens when external data enters the program. Bytes read from a file or received through a network connection must be decoded back into Unicode text so the program can display, compare, or manipulate the characters. This is why UTF-8 matters at both sides of an I/O boundary: encoding prepares text to leave the program, and decoding restores text after it arrives.

Python Unicode textinside the programUTF-8 bytesentering the programUTF-8 bytesleaving the programUnicode textusable inside the programFile or networkexternal destinationencodewrite or sendread or receivedecode
How does text move between a Python program, a file, or an internet connection, and where does encoding or decoding occur?

Choosing Encode or Decode

Unicode textinside the programFile or network inputbytes arriveFile or networkoutputtext leavesUnicode textdecodeUTF-8 bytesencodereaches boundaryencodedecode
When should a program encode text to UTF-8 bytes and when should it decode bytes back into Unicode text?

Use UTF-8 encoding when text is written to a file, sent over a network, or otherwise leaves the program. Use decoding when bytes from a file or network connection must become usable Unicode text inside the program. For file operations involving non-English text, the source recommends specifying encoding='utf-8' with the open function so Python can convert correctly while reading or writing.

Worked example

Saving international text

A Python program needs to save text containing characters that ASCII cannot represent.

  1. 1
    Represent internally

    Create and manipulate the text as a Unicode string inside the program.

  2. 2
    Cross the boundary

    When writing to a file, the text must be converted into bytes.

  3. 3
    Choose UTF-8

    Specify UTF-8 for the file operation so the Unicode text is converted to and from the intended byte representation.

  4. 4
    Read it later

    When the file is read, the bytes are converted back into Unicode text for the program.

Result

Unicode is used inside the program, while UTF-8 handles the file boundary.

Mistakes with Text Boundaries

  • Common mistake

    Treating Unicode and UTF-8 as interchangeable names.

    What it looks like

    Calling Unicode the byte format used to send data.

    Why it fails

    Unicode is the abstract character set and standard, while UTF-8 is a concrete method for encoding Unicode code points as bytes.

    Do this instead

    Use Unicode to describe the characters and code points your program handles; use UTF-8 to describe the encoding used at an external boundary.

  • Common mistake

    Assuming ASCII can represent every language.

    What it looks like

    Using an ASCII-only approach for Chinese characters, Arabic script, accented letters, or emoji.

    Why it fails

    ASCII covers English letters, numbers, and basic punctuation but cannot represent many international characters.

    Do this instead

    Use Unicode for text that may contain characters beyond ASCII.

  • Common mistake

    Forgetting the conversion when text crosses a boundary.

    What it looks like

    Treating a Unicode string as if it were already the byte representation required by a file or network connection.

    Why it fails

    External storage and transmission require bytes, so Unicode text must be encoded before leaving the program.

    Do this instead

    Encode to UTF-8 for output and decode incoming bytes back into Unicode text.

  • Common mistake

    Relying on an unspecified file encoding for international text.

    What it looks like

    Reading or writing non-English text without identifying UTF-8.

    Why it fails

    The program may not perform the intended conversion between Unicode text and bytes.

    Do this instead

    Specify encoding='utf-8' for the relevant file operation, following the current Python documentation.

QuestionUnicodeUTF-8
What is it?A universal character set and standardA variable-width encoding scheme
What does it identify or produce?Unique code points for charactersBytes for storage and transmission
Where is it used?Inside the program for text handlingAt file and network boundaries
What is the key action?Assign and represent charactersEncode or decode bytes

Unicode describes character identity; UTF-8 describes byte representation.

Practice the Direction

EASY

A program has Unicode text that it needs to send across an internet connection. Which operation should happen at the boundary, and what should happen when bytes arrive back in the program?

Hints
  • Ask whether the text is leaving or entering the program.
  • Leaving text must become bytes; arriving bytes must become usable text.

What do you think happens?

A Unicode string is about to be written to a file using UTF-8. Which direction is correct?

  • Unicode text to UTF-8 bytes

  • UTF-8 bytes to Unicode text

  • Unicode text remains unchanged and no boundary operation is needed

Reveal answer
Answer:

Unicode text to UTF-8 bytes

Writing sends data out of the program, so the Unicode text must be encoded into bytes. Reading reverses the direction by decoding bytes into Unicode text.

Key Takeaways

  1. 1
    Unicode assigns a unique code point to characters from languages around the world. The u prefix is used in the source material to create a Unicode string in Python. UTF-8 is a variable-width encoding that converts Unicode code points into bytes. Use Unicode while handling text inside the program, then encode to UTF-8 when writing or sending data and decode when reading or receiving it. For file operations involving international text, explicitly specify UTF-8 according to the current Python documentation.

At a glance

  • Unicode is the abstract character set and standard; UTF-8 is an encoding method.
  • Unicode code points give characters unique numerical identities.
  • Python uses Unicode strings internally for international text, with the source material showing the u prefix.
  • UTF-8 converts Unicode text into bytes for files and networks.
  • Encode when text leaves the program and decode when bytes enter it.

Related Topics

Related concepts

More concepts starting with W