Working with String Types in Python

In this sequence, the Unicode string is passed to io.open, which uses UTF-8 to encode the characters into bytes for the file.

Programming

From Stored Bytes to Python Text

A string that looks like one character to you may be represented by several bytes on disk. Correct file handling connects those two representations: encoded bytes in storage and abstract Unicode characters in Python memory. The central question is not only what text your program wants to use, but also how that text is encoded while it travels into and out of a file.

How do UTF-8 bytes on disk become Unicode string characters in Python, and how are strings converted back into bytes when written?

Bytes are concrete and depend on an encoding. For example, the character é is represented by the byte sequence C3 A9 in UTF-8, while another encoding can use a different sequence. The Unicode character itself is abstract: é is U+00E9 regardless of its storage representation. When io.open is given encoding="utf-8", it decodes UTF-8 bytes into Unicode characters while reading and encodes Unicode characters into UTF-8 bytes while writing.

Declaring and Opening Unicode Text

When a Python source file contains Unicode characters in a string literal, place the encoding declaration at the top of the file. The comment # encoding=utf-8 tells the interpreter that the source file itself uses UTF-8, so it can parse the characters correctly. The u prefix on a string literal, such as u"...", makes the intention to use a Unicode string explicit.

python

# encoding=utf-8
import io

with io.open("notes.txt", "rt", encoding="utf-8") as f:
    text = f.read()

message = u"Imagine non-English language here"

# encoding=utf-8
import io

with io.open("notes.txt", "rt", encoding="utf-8") as f:
    text = f.read()

message = u"Imagine non-English language here"

The source encoding declaration and the file encoding parameter solve different problems. The declaration tells Python how to read characters written directly in the program's source file. The encoding parameter tells io.open how to translate the file's bytes when the program reads or writes external text. Both matter when a program contains Unicode literals and works with UTF-8 files.

Choosing a File Mode

The open() function takes a filename and a mode. The mode determines whether the program reads, writes, or appends, and it affects what happens to existing data. Opening a file returns a file object, which provides methods such as read(), readline(), and write().

Mode	Main operation	Effect on existing content
r	Read	Preserves existing content
w	Write	Erases existing file content
a	Append	Preserves existing content and adds data

The three file modes described in the source material

What can happen to existing file content when a file is opened in read, write, or append mode?

Reading Lines and Finding EOF

A file object keeps track of where the next read will begin. readline() returns one complete line, including that line's newline character. When there is no line left, readline() returns an empty string. That empty string is the end-of-file signal.

Code walkthrough

f = open("log.txt", "r")

while True:
    line = f.readline()
    if len(line) == 0:
        break
    print(line, end='')

f.close()

1
open() creates a file object in read mode.
2
readline() obtains the next complete line and advances the file pointer.
3
len(line) == 0 checks whether readline() returned the empty string at end of file.
4
break stops the loop when no more content remains.
5
print(line, end='') avoids adding a second newline because line already includes its newline character.
6
close() closes the file after reading is complete.

Where is the file pointer before and after each readline() call, and how does it eventually reach the end of the file?

Suppose the first readline() reads line 1. The pointer then stands at the beginning of line 2, so the next call reads line 2. This continues until the pointer reaches the end. The next call returns an empty string, len(line) becomes zero, and the loop stops. The pointer's movement explains why repeated readline() calls do not repeatedly return the first line.

Writing, Saving, and Closing

Writing follows the same three-stage pattern as reading: open the file, perform the operation, and close the file. write() accepts a string and places it in the file. Unlike print(), write() does not add a newline automatically, so include one in the string when a line break is needed.

python

# encoding=utf-8
import io

f = io.open("greeting.txt", "wt", encoding="utf-8")
f.write(u"Bonjour, café\n")
f.close()

# encoding=utf-8
import io

f = io.open("greeting.txt", "wt", encoding="utf-8")
f.write(u"Bonjour, café\n")
f.close()

In this sequence, the Unicode string is passed to io.open, which uses UTF-8 to encode the characters into bytes for the file. The close() call finalizes the changes and releases the file. Skipping close() can mean that data is not saved or that the file remains unavailable to other processes.

python

import io

with io.open("greeting.txt", "rt", encoding="utf-8") as f:
    text = f.read()

import io

with io.open("greeting.txt", "rt", encoding="utf-8") as f:
    text = f.read()

What happens to data between write() and close(), and how does closing complete the file operation?

Mistakes That Corrupt Text Handling

Common mistake
Leaving out the source encoding declaration when the source contains Unicode literals.
What it looks like
A source file contains a non-ASCII character in u"..." but does not begin with # encoding=utf-8.
Why it fails
The interpreter may fail to parse the source or misinterpret its characters.
Do this instead
Place # encoding=utf-8 at the top of the source file.
Common mistake
Reading or writing UTF-8 text without specifying the encoding.
What it looks like
Opening a Unicode text file without io.open(..., encoding="utf-8").
Why it fails
Python is not explicitly told how to translate the file's bytes and the program's Unicode characters.
Do this instead
Use io.open with encoding="utf-8" for the Unicode file operation.
Common mistake
Assuming write() adds a newline.
What it looks like
f.write("first") followed by f.write("second") when two lines were intended.
Why it fails
write() places the supplied string in the file but does not automatically add a newline.
Do this instead
Include the newline character in the string when a line break is required.
Common mistake
Printing a line returned by readline() with the default print ending.
What it looks like
```
print(line) when line already includes its newline character.
```
Why it fails
The file's newline and print()'s newline create double spacing.
Do this instead
Use print(line, end='') when displaying lines returned by readline().
Common mistake
Treating an empty line as the same as end of file.
What it looks like
Stopping whenever a line appears visually blank.
Why it fails
The source's EOF test is based on readline() returning an empty string, checked with len(line) == 0.
Do this instead
Use the empty-string result from readline() as the EOF signal.
Common mistake
Opening an existing file in write mode when its content should be preserved.
What it looks like
open("log.txt", "w") for adding another log entry.
Why it fails
Write mode erases existing file content.
Do this instead
Use append mode when existing content must be preserved and new data added.
Common mistake
Forgetting to close a file.
What it looks like
Calling write() and allowing the program to continue without close() or a with statement.
Why it fails
Changes may not be saved and the file may remain unavailable to other processes.
Do this instead
```
Call close() or use a with statement.
```

Practice the Full Pipeline

Worked example

Append a Unicode line and read the file

Add one Unicode greeting to a UTF-8 text file, then read the resulting file one line at a time.

1
Declare source encoding
Begin the source file with # encoding=utf-8 so the Unicode literal can be parsed.
2
Append the greeting
Open the file with io.open in append mode and encoding="utf-8", then write a u-prefixed string containing a newline.
3
Close the writer
Close the file after writing so the change is finalized and the file is released.
4
Read each line
Open the file in read mode, call readline(), and stop when the returned string has length zero.
5
Preserve line spacing
Display each returned line with print(line, end='') because readline() includes the line's newline.

Result

The program converts the Unicode greeting to UTF-8 bytes when appending, converts file bytes back to Unicode strings when reading, and stops at the empty-string EOF signal.

MEDIUM

Write a short Python program that reads a UTF-8 file line by line. For every line, display it without adding an extra blank line. Stop only when readline() returns the empty string.

Hints

Open the file in read mode and specify encoding="utf-8".
Call readline() inside a loop.
Check len(line) == 0 before printing.
Use print(line, end='') because readline() includes the newline.

Your attempt (not saved)

The Reliable File-Handling Pattern

1
Use # encoding=utf-8 when the source file contains Unicode literals, and use u"" to make Unicode string intent explicit.
2
Use io.open with encoding="utf-8" so Python decodes file bytes into Unicode characters when reading and encodes characters into bytes when writing.
3
Choose r to read, w to write and erase existing content, or a to append while preserving existing content.
4
readline() advances the file pointer one line at a time and returns an empty string at end of file.
5
Close files explicitly with close() or use a with statement so changes are finalized and the file is released.

Key Takeaways

UTF-8 bytes on disk and Unicode characters in Python memory are different representations connected by an encoding layer.
The source encoding comment describes the Python file itself; encoding="utf-8" describes external text files.
File modes control whether content is read, replaced, or extended.
readline() moves the file pointer forward and returns an empty string at EOF.
Closing a file, either manually or through with, helps ensure data is saved and the file is released.

Related concepts

More concepts starting with W

From Stored Bytes to Python Text

Declaring and Opening Unicode Text

Choosing a File Mode

Reading Lines and Finding EOF

Writing, Saving, and Closing

Mistakes That Corrupt Text Handling

Practice the Full Pipeline

The Reliable File-Handling Pattern

Key Takeaways

Related Topics

Related concepts