Concepts / Reading Files Line-by-Line with a for Loop

Reading Files Line-by-Line with a for Loop

read() loads an entire file into a single string, including all newline characters, and exhausts the file handle resource after one call.

  • Programming
Interactive lab

Try it: Reading and Writing Text Files

How open() returns a file object with a position pointer, how read(), read(n), readline(), readlines() and for line in fh advance it (and return '' at the end), how 'w' and 'a' modes change the file, and why a closed file (after close() or a with block) can no longer be read.

How it works

  1. open(name, 'r') starts the pointer at 0; 'w' empties the file first; 'a' starts at the end.
  2. read() returns everything from the pointer to the end; read(n) at most n characters; readline() up to and including the next \n; each moves the pointer past what it returned.
  3. At the end of the file every read returns '' (an empty string), which is how loops know to stop; for line in fh reads line by line the same way.
  4. write() text waits in a buffer and reaches the file when it is closed (or the program ends); reading a 'w'/'a' file or writing an 'r' file raises io.UnsupportedOperation.
  5. close(), or leaving a with block, closes the file; any further read raises ValueError: I/O operation on closed file.

Default run (10 steps): Ready: notes.txt holds 4 lines (51 characters). Nothing is open yet. … Finished. notes.txt now holds "From: stephen\nSubject: meeting\nFrom: louis\nThanks!\n".

Simplified: The lab's own small Python interpreter runs a fixed, hand-written script (you choose the variant and the values, never the code); nothing is executed as real Python. Every script was run in CPython 3.12 with the same inputs and the lab reproduces its output and error messages. One virtual text file with \n line endings (as on Linux/macOS), positions counted in characters (ASCII only), and a write buffer that is only flushed by close() or the end of the program.

Educational simulation

Loading the simulation…

One Read, One Position

A file on disk can contain much more data than your program should load into memory at once. Python offers two important strategies: read the complete file into one string with read(), or iterate through the file one line at a time with a for loop. The key difference is how much data is held in memory and how the file's current position changes while it is being read.

What do you think happens?

A file contains three lines. After read() is called once without a size argument, what will a second read() call on the same file handle return?

  • The complete file again

  • The last line only

  • An empty string

  • The first line only

Reveal answer
Answer:

An empty string

A complete read starts at the current position and continues to the end of the file. Afterward, the file handle's position is already at the end, so another read() has no remaining data to return.

File contentsline 1, line 2, line 3contentsone stringEmpty stringsecond read()Position 0beginningEnd of filenothing remainsread() returnsposition movesread() again
What changes after read() loads the complete file, and why does the next read() return an empty string?

What read() Returns

Calling read() without a size argument reads from the current file position through the end of the file. Python returns the complete contents as one string, including the newline characters that separate lines. Assign that returned string to a variable immediately. The variable preserves the data, while the file handle's position continues to move.

Code walkthrough
file_handle = open("notes.txt")
contents = file_handle.read()
print(contents)
  1. 1
    The file handle begins with its position at the beginning of the file.
  2. 2
    The read() call obtains all remaining file contents and returns them as one string.
  3. 3
    Assigning the result to contents preserves the returned string for later processing.
  4. 4
    After the complete read, the file position is at the end.
  • open() creates a connection to the file.
  • read() loads the complete remaining contents into contents.
  • The string in contents includes the file's newline characters.
  • Common mistake

    Calling read() twice and expecting the complete file both times.

    What it looks like
    first_contents = file_handle.read()
second_contents = file_handle.read()
    Why it fails

    The first complete read moves the position to the end of the file, so the second call has no remaining data.

    Do this instead

    Assign the first read() result immediately and reuse that variable when you need the contents again.

  • Common mistake

    Assuming the file handle itself contains all file data.

    What it looks like

    Treating file_handle as though it were the text stored in the file.

    Why it fails

    The file handle is a connection to the file on disk. It maintains a reading position, but it does not contain the file's contents.

    Do this instead

    Use read() to obtain a complete string, or iterate over the handle to fetch lines incrementally.

Following the File Loop

A for loop over a file handle causes Python to fetch the next line during each iteration. The open function creates the connection but does not pull the entire file into memory. The loop body runs once for each line, and the loop variable receives that line, including its newline character when the line has one.

python

In this pattern, line is not the entire file. On the first iteration it refers to the first line, on the next iteration it refers to the next line, and so on. After a line has been processed, it can be discarded before the next line is fetched. The loop therefore supports processing a very large file without loading all of its contents into one string.

File positionbeginninglinefirst line plus newlineLoop endsno lines remainlinesecond line plus newlinelinenext linefetch next lineadvanceadvanceafter final line
How does control flow move through the file, and what does the loop variable contain on each iteration?

Newline Boundaries

A line variable normally includes the newline character at the end of that line. That character is how Python represents the boundary between one line and the next, and the for loop does not remove it when placing the line in the loop variable. The complete string returned by read() also includes newline characters, but it contains all of them in one string rather than one line variable at a time.

python

Using repr() in this demonstration makes the newline character visible in the displayed representation. For a line that ends normally, the representation shows the newline as part of the string. The final line may not have a newline character if the file does not end with one, so the loop variable's ending characters depend on the file's actual contents.

Choosing by File Size

StrategyData held during processingBest fit
read()The complete remaining file in one stringA file small enough to fit in available RAM
for line in file_handleOne line at a timeLarge files or processing where memory efficiency matters

Choose the reading strategy according to file size and available memory.

Complete fileone string in memoryAvailable RAMmust hold complete fileCurrent lineone line in memoryNext linearrives after current lineoccupiesreplace after processing
Where is file data held when read() loads the whole file compared with when a loop processes one line at a time?
Worked example

Selecting a Strategy

Choose a reading strategy for a file that is 1 GB in size when the program has only a smaller amount of available RAM.

  1. 1
    Estimate what read() would require

    read() attempts to place the complete file contents into one string, so the file must fit in available memory.

  2. 2
    Compare the file with available memory

    A 1 GB file requires at least 1 GB of free memory according to the source guidance, which is not available in this situation.

  3. 3
    Select incremental processing

    Iterate over the file handle with a for loop so each line is read, processed, and then discarded before the next line arrives.

Result

Use a for loop over the file handle rather than read().

Mistakes in File Loops

  • Common mistake

    Using read() for a file that may not fit in available RAM.

    Why it fails

    read() loads the entire file into a single string, so memory use grows with the file size.

    Do this instead

    Use a for loop over the file handle to keep only the current line in memory.

  • Common mistake

    Thinking that opening a file has already loaded all its contents.

    Why it fails

    The file handle is a connection to the file on disk, not a container holding all file data.

    Do this instead

    Remember that iteration triggers the fetching of each next line.

  • Common mistake

    Assuming a line variable never contains a newline character.

    Why it fails

    Each iteration normally includes the line's newline character.

    Do this instead

    Account for the newline when processing or displaying a line.

  • Common mistake

    Calling read() again after the complete contents have already been saved.

    Why it fails

    The file position is at the end after the first complete read, so the later call returns an empty string.

    Do this instead

    Reuse the variable holding the first read() result.

Practice the Decision

EASY

A program needs to count the lines in a very large file. Write the file-reading pattern that keeps memory use low, and explain what the loop variable contains during one iteration.

Hints
  • Use a for loop directly with the file handle.
  • Increment a counter inside the loop.
  • The loop variable contains one line, normally including its newline character.
MEDIUM

A small file must be used several times after it is opened. Decide whether read() is suitable, and state what should happen immediately after the call to read().

Hints
  • Consider whether the complete file can fit in available RAM.
  • Assign the returned string to a variable immediately.
  • Do not expect a second complete read from the same position.

Key Takeaways

  1. 1
    read() returns the complete remaining file contents as one string, including newline characters.
  2. 2
    After a complete read(), the file position is at the end, so another read() on that handle returns an empty string.
  3. 3
    A file handle is a connection to the file on disk and maintains a reading position; it does not contain all file data.
  4. 4
    A for loop over a file handle fetches one line per iteration, normally including that line's newline character.
  5. 5
    Use read() for files that fit in available RAM and line-by-line iteration when memory efficiency is important, especially for large files.

At a glance

  • read() loads a complete file into one string and moves the file position to the end.
  • A second read() on the same handle returns an empty string because no data remains at the current position.
  • The file handle is a connection with a position pointer, not a copy of the file's contents.
  • A for loop reads one line at a time and normally keeps only the current line in memory.
  • Choose read() for suitably small files and line-by-line iteration for large files or memory-sensitive processing.

Related Topics

Related concepts

More concepts starting with R