Reading Files and Iterating Over Lines

Parsing is the process of finding interesting lines in a file and extracting specific pieces of information from them.

Programming

Interactive lab

Try it: Reading and Writing Text Files

How open() returns a file object with a position pointer, how read(), read(n), readline(), readlines() and for line in fh advance it (and return '' at the end), how 'w' and 'a' modes change the file, and why a closed file (after close() or a with block) can no longer be read.

How it works

open(name, 'r') starts the pointer at 0; 'w' empties the file first; 'a' starts at the end.
read() returns everything from the pointer to the end; read(n) at most n characters; readline() up to and including the next \n; each moves the pointer past what it returned.
At the end of the file every read returns '' (an empty string), which is how loops know to stop; for line in fh reads line by line the same way.
write() text waits in a buffer and reaches the file when it is closed (or the program ends); reading a 'w'/'a' file or writing an 'r' file raises io.UnsupportedOperation.
close(), or leaving a with block, closes the file; any further read raises ValueError: I/O operation on closed file.

Default run (10 steps): Ready: notes.txt holds 4 lines (51 characters). Nothing is open yet. … Finished. notes.txt now holds "From: stephen\nSubject: meeting\nFrom: louis\nThanks!\n".

Simplified: The lab's own small Python interpreter runs a fixed, hand-written script (you choose the variant and the values, never the code); nothing is executed as real Python. Every script was run in CPython 3.12 with the same inputs and the lab reproduces its output and error messages. One virtual text file with \n line endings (as on Linux/macOS), positions counted in characters (ASCII only), and a write buffer that is only flushed by close() or the end of the program.

Educational simulation

Loading the simulation…

Finding Useful Lines

A file often contains more information than a program needs. An email log, for example, may contain thousands of lines, but a program might need only the day of the week from lines that begin with From. Parsing is the process of finding those interesting lines and extracting specific pieces of information from them.

Parsing follows a filter-then-parse pattern. First, check whether a line has the structure you expect. Then, split that line into fields and select the field you need. This order matters because not every line in a file has the same structure.

How does the program apply the same check to each line as it moves through the file?

Filtering with startswith

The startswith() method checks whether a line begins with a particular string. For the email log pattern, checking line.startswith('From ') selects lines that begin with From followed by a space. The space is significant because it helps distinguish the expected line structure from a line that merely contains the word From somewhere else.

python

for line in file:
    if line.startswith('From '):
        print(line)

for line in file:
    if line.startswith('From '):
        print(line)

The loop reads one line at a time. For each line, the conditional check has two possible results. If the line starts with From followed by a space, the indented code runs. If it does not, the line is skipped and the loop moves to the next line.

Which lines pass the startswith() condition, which lines are skipped, and where does control flow go?

Turning Lines into Fields

After a line passes the filter, split() breaks the line into a list of words or fields. Each resulting list element has a position called an index. Indexing the list lets you select one exact field instead of using the entire line.

Worked example

Locating Fields in a From Line

A matching line has the structure From sender day month time year. Which index contains the sender, day, and month?

1
Split the line
The line is transformed into a list whose elements are the separate words or fields.
2
Count from index zero
The first field is at index 0, so From is words[0] and the sender is words[1].
3
Select the needed field
The day is at words[2], while the month is at words[3].

Result

Use words[1] for the sender, words[2] for the day, and words[3] for the month.

Code walkthrough

line = 'From sender@example.com Tue Jan 14:20 2025'
words = line.split()
day = words[2]
print(day)

1
The string represents a line with the expected From structure.
2
split() creates a list of fields from the line.
3
words[2] selects the field at index 2, which is the day.
4
The selected day is printed.

Output

Tue

What changes when one structured text line is split into separate fields?

After split() creates a list, which index contains each field from the original structured line?

Combining the Two Steps

The complete pattern reads each line, tests its beginning, splits only matching lines, and extracts the desired field. In this example, the desired field is the day of the week at index 2.

python

file = open('mail_log.txt')

for line in file:
    if line.startswith('From '):
        words = line.split()
        print(words[2])

file = open('mail_log.txt')

for line in file:
    if line.startswith('From '):
        words = line.split()
        print(words[2])

Output

Tue
Fri
Mon

How does a line move from the file to the desired field?

Mistakes to Avoid

Common mistake
Splitting every line before checking its structure
What it looks like
```
words = line.split() is executed before testing whether the line starts with From.
```
Why it fails
Files can contain headers, comments, blank lines, or other formats that do not contain the expected fields.
Do this instead
Use startswith() first, then split only the lines that match.
Common mistake
Filtering for From without the trailing space
What it looks like
```
line.startswith('From')
```
Why it fails
This can match lines that begin with the word From but do not have the expected From line structure.
Do this instead
Use the expected prefix From followed by a space when that is the structure being parsed.
Common mistake
Using the wrong list index
What it looks like
```
print(words[1]) when the intended field is the day.
```
Why it fails
In the described From structure, words[1] is the sender and words[2] is the day.
Do this instead
Write down the fields and count their positions starting at index 0.
Common mistake
Ignoring trailing whitespace
What it looks like
Using a line exactly as read without considering trailing whitespace.
Why it fails
Trailing whitespace can affect how a line is handled during parsing.
Do this instead
Strip trailing whitespace when appropriate before applying the rest of the parsing logic.

Practice the Pattern

EASY

Suppose a file contains lines with the structure From sender day month time year, mixed with unrelated lines. Write the loop logic that prints only the day from matching lines.

Hints

Iterate over the file one line at a time.
Use startswith('From ') as the conditional check.
Call split() on a matching line.
The day is at index 2 in this structure.

Your attempt (not saved)

EASY

Change the extraction so that the program prints the sender instead of the day. Which index should you use, and why?