Concepts / Writing Data to Files

Writing Data to Files

The file handle does not contain file data; it is a connection to the file on disk that the for loop uses to read data incrementally.

  • Programming

The Hidden Work Behind a File Loop

A file loop can look simple: open a file, then use a for loop to process it. The important mechanism is easy to miss. Opening the file does not pull all of its data into memory. The file handle is a connection to the file on disk, and the for loop uses that connection to request the next line when each iteration begins.

Handle, Disk, and Loop

Programfor loopFile handleconnectionFile on diskstored datausesconnects to
What does the file handle contain, and how does it connect the program to the file data stored on disk?

A file handle does not contain the file's complete data. It represents a connection to the file on disk. When a for loop iterates over the handle, each iteration causes Python to fetch the next line. The loop body then works with that line before the next line is fetched.

File on diskmany linesLine 1processedProgram memorycurrent lineLine 2processed nextNext lineone at a timefetchesthen fetchescontinuesholds brieflyreplacesreplaces
How does a for loop read one line at a time, and why does this use less memory than loading the entire file?

Only the current line needs to be kept for the loop's immediate work. After that line has been processed, the next line can be fetched. This is more memory efficient than loading an entire file at once and allows files of any size to be processed with only a small, constant amount of memory.

What Each Line Contains

python

In this pattern, the loop variable line receives one line during each iteration. A line read from the file includes its newline character when that character is present at the end of the stored line. The newline is not automatically removed before the loop body receives the line.

First linetext plus newlinelineincludes newlineFinal linetext without newlinelineends at file endread by loopread by loop
When does each line variable include a newline character, and what does the final line look like when the file does not end with one?
Worked example

Tracing Three Iterations

Suppose a file contains three lines. The first two lines end with newline characters, but the final line does not. What does the loop variable receive?

  1. 1
    Iteration 1

    The loop receives the first line, including its newline character.

  2. 2
    Iteration 2

    The loop receives the second line, including its newline character.

  3. 3
    Iteration 3

    The loop receives the final line. Because the file does not end with a newline, this value does not include one at the end.

Result

The loop processes one line per iteration, but the final line can differ: it may have no newline character when the file ends immediately after its text.

Placing Data with write()

Definition

The write() method puts data into a file exactly as provided and returns the number of characters written.

python

The string supplied to write() controls exactly what is inserted. In the example, the string includes a newline character, so the written data includes that line ending. The value returned by write() is the character count for the string that was written, including the newline character when it is part of that string.

Programwrite("A")no newlineFile contentsAB newline Cwrite("B\n")newline includedwrite("C")no newlinefirstnextnextappears sequentially
How will consecutive write() calls appear in a file when the strings do or do not contain newline characters?

Consecutive write() calls place their strings sequentially from the current file position. If the first string does not end with a newline, the next string begins immediately after it. If a string does contain a newline, the following string begins after that line ending.

write() Compared with print()

write()uses string exactlyAno automatic newlineprint()adds a newlineA newlineline ending addedwrite("A")print("A")
What changes in the file when write() receives a string without a newline compared with print(), which adds one automatically?
OperationLine-ending behaviorWhat must the programmer do?
write("A")Writes A exactly as providedAdd a newline explicitly if a line ending is wanted
write("A\n")Writes A followed by a newlinePlace the newline in the string
print("A")Adds a newline automaticallyNo explicit newline is needed for the ordinary print behavior

The important difference is whether the operation adds a line ending automatically or uses the supplied string exactly.

Unlike print(), write() does not automatically append a newline. To create separate lines with write(), include the newline character in the strings you pass to it. Without that character, multiple calls can join their text on the same line.

Mistakes with Reading and Writing

  • Common mistake

    Assuming that opening a file loads all of its data into the file handle

    What it looks like

    Treating the handle as if it were a complete in-memory copy of the file

    Why it fails

    The file handle is a connection to the file on disk. The for loop triggers incremental reading.

    Do this instead

    Think of each loop iteration as fetching the next line through the connection.

  • Common mistake

    Assuming every line variable has no newline character

    What it looks like

    Comparing a read line with text while forgetting that the line may end with a newline

    Why it fails

    A line read by the loop includes its newline character when one is present in the file.

    Do this instead

    When tracing a line, check whether the file stored a newline at that position.

  • Common mistake

    Expecting write() to move to a new line automatically

    What it looks like

    Calling write("A") and then write("B") while expecting two separate lines

    Why it fails

    write() inserts the strings exactly as provided and does not automatically add newlines.

    Do this instead

    Use write("A\n") when the first piece of data should end a line.

  • Common mistake

    Ignoring the return value of write()

    What it looks like

    Treating the returned value as the data that was written

    Why it fails

    write() returns the number of characters written, not the written string.

    Do this instead

    Interpret the return value as a character count.

Best practice

Before predicting a file's final contents, trace every write() call in order and mark whether each supplied string contains a newline. The file result is the direct sequence of those strings; no extra newline is inserted by write().

Predict the File Result

What do you think happens?

A program performs these calls in order: write("red"), write("\n"), write("blue\n"), write("green"). What sequence of text appears in the file?

  • redbluegreen

  • red, blue, and green on separate lines

  • red on the first line, blue on the second line, and green on the third line without an ending newline

  • red on the first line and bluegreen on the second line

Reveal answer
Answer:

red on the first line, blue on the second line, and green on the third line without an ending newline

The first call writes red without ending the line. The second call supplies the newline. The third call writes blue and another newline. The final call writes green without adding a newline.

MEDIUM

Construct a short file-processing loop that counts the lines in a file. Then explain why the loop can process a very large file without loading all of its lines into memory at once.

Hints
  • Iterate directly over the file handle with a for loop.
  • Increment a counter once for each loop iteration.
  • Connect the memory explanation to the fact that each iteration reads one line.

Working Rules to Remember

  1. 1
    A file handle is a connection to file data on disk, not the complete file data stored in memory.
  2. 2
    A for loop over a file handle reads one line per iteration, which keeps memory use small.
  3. 3
    A read line includes its newline character when that character is present in the file.
  4. 4
    write() inserts exactly the supplied string and returns its character count.
  5. 5
    write() does not add newlines automatically, so include newline characters explicitly when separate lines are required.

Key Takeaways

  • Opening a file establishes a connection; iteration is what triggers incremental reading.
  • A file loop keeps one current line in memory rather than loading the entire file at once.
  • Newline characters can be part of the line values read from a file.
  • write() writes the supplied string exactly and returns the number of characters written.
  • Separate output lines require explicit newline characters when using write().

Related Topics

Related concepts

More concepts starting with W