Using Python's os Module for File Operations

The organization has three levels: a main backup directory, a date-based subdirectory, and a time-stamped zip file inside that subdirectory.

Programming

From Clutter to Structure

A backup script can work correctly and still become difficult to use. If every archive is stored in one directory with a long timestamped filename, repeated backups can produce dozens of similarly named files. A hierarchical naming scheme improves this situation by dividing the backups first by date and then by time.

Key point

The organization has three levels: a main backup directory, a date-based subdirectory, and a time-stamped zip file inside that subdirectory.

The Three-Level Backup Path

The main directory is the common home for all backups. Each date-based subdirectory groups backups made on one day. Each zip file inside that date directory represents a backup made at a particular time. This makes it easier to navigate to a day and then distinguish the backups created during that day.

What contains what, and how does a backup path organize files by date and time?

Worked example

Reading a Backup Path

Interpret the path /Users/swa/backup/20240115/143022.zip.

1
Main directory
backup is the main directory that contains the organized backups.
2
Date directory
20240115 is the date-based directory. The source uses this form for January 15, 2024.
3
Time-stamped file
143022.zip is the zip archive named with the time 14:30:22.

Result

The path identifies a backup made on January 15, 2024, at 2:30:22 PM.

Turning Time into Names

The time.strftime function returns a formatted string representing the current date or time. In this backup design, one format produces the date directory name and another produces the zip filename. The format code %Y%m%d produces a year-month-day string such as 20240115, while %H%M%S produces a 24-hour time string such as 143022.

How does the current date and time become the folder name and timestamped file name?

python

import time

date_name = time.strftime('%Y%m%d')
time_name = time.strftime('%H%M%S')

print(date_name)
print(time_name + '.zip')

import time

date_name = time.strftime('%Y%m%d')
time_name = time.strftime('%H%M%S')

print(date_name)
print(time_name + '.zip')

Checking Before Creating

Before saving an archive into the date-based directory, the script must determine whether that directory already exists. os.path.exists performs the check. If the directory exists, the script can use it. If it does not exist, os.mkdir creates it. This check matters when the script runs repeatedly, because a second backup on the same day should use the existing date directory rather than attempt to create it again.

What happens next when the date-based directory already exists versus when it does not?

Code walkthrough

import os
import time

backup_dir = '/Users/swa/backup'
date_name = time.strftime('%Y%m%d')
date_dir = backup_dir + '/' + date_name

if not os.path.exists(date_dir):
    os.mkdir(date_dir)

1
The backup_dir value names the main backup directory.
2
time.strftime('%Y%m%d') creates the date-based directory name.
3
The date directory path combines the main directory and the date name.
4
os.path.exists checks whether that path already exists.
5
os.mkdir runs only when the date directory does not exist.

What do you think happens?

If the script runs again on the same day after the date directory has already been created, which branch should run?

The existing-directory branch
The os.mkdir branch

Reveal answer

Answer:

The existing-directory branch

os.path.exists reports that the date directory is already present, so os.mkdir is not needed for that run.

Building the Archive Command

After the directory and names are ready, the script constructs a system command for creating the compressed archive. The command contains the target zip path and the source paths to be archived. The resulting command is passed to os.system, which executes it as an operating-system command.

How do source files and a destination path become a command that creates a zip archive?

python

target = '/Users/swa/backup/20240115/143022.zip'
source = '/Users/swa/notes'
command = 'zip -r ' + target + ' ' + source
result = os.system(command)

target = '/Users/swa/backup/20240115/143022.zip'
source = '/Users/swa/notes'
command = 'zip -r ' + target + ' ' + source
result = os.system(command)

Definition

os.system executes the supplied shell command and returns 0 when the command succeeds. A non-zero return value indicates failure, so the script can use the result to verify whether the backup completed.

Tracing a Second Backup

Imagine that the script runs on January 15, 2024, at 2:30:22 PM and then runs again on the same day at 6:05:00 PM. The date string remains the same, so both backups belong in the 20240115 directory. The time string changes, so the second backup receives a different time-stamped zip filename.

What changes when the backup runs again on the same date at a different time?

Worked example

Following the Second Run

Determine what happens when the backup runs on January 15, 2024, at 18:05:00 after a backup has already run earlier that day.

1
Format the date
The %Y%m%d format produces 20240115 again.
2
Check the directory
The 20240115 directory already exists because an earlier backup used it.
3
Format the time
The %H%M%S format produces 180500 for 18:05:00.
4
Create the new target
The new archive is placed inside the existing date directory with the name 180500.zip.

Result

The second run reuses 20240115 and adds another time-stamped archive inside it.

What is the sequence from preparing the directory and names to creating the final backup archive?

Mistakes Beginners Make

Common mistake
Creating the date directory every time
What it looks like
Calling os.mkdir without first checking whether the date directory exists.
Why it fails
A repeated run on the same day may encounter a directory that was already created by an earlier run.
Do this instead
Use os.path.exists first and call os.mkdir only when the directory is absent.
Common mistake
Putting the full timestamp into one flat filename
What it looks like
Keeping every backup in the main directory with a long filename containing the complete timestamp.
Why it fails
Repeated backups can make the directory cluttered with similarly named files, making a particular day harder to find.
Do this instead
Use the date portion for a subdirectory and the time portion for the zip filename.
Common mistake
Ignoring the result from os.system
What it looks like
Executing the command and never examining the returned value.
Why it fails
The script loses the supplied success or failure signal for the archive operation.
Do this instead
Check whether the returned value is 0 for success or non-zero for failure.
Common mistake
Mixing up the date and time formats
What it looks like
Using the time format for the directory name or the date format for the archive name.
Why it fails
The intended hierarchy uses the date to group backups and the time to distinguish archives within that group.
Do this instead
Use %Y%m%d for the date directory and %H%M%S for the zip filename.

Practical Design Notes

Best practice

Treat the backup path as a sequence of decisions rather than as one long filename. First produce the date and time strings. Next prepare the date directory by checking and, when necessary, creating it. Then form the target zip path, construct the command with the source paths, execute it with os.system, and inspect the return value.

python

import os
import time

backup_dir = '/Users/swa/backup'
source = '/Users/swa/notes'
date_name = time.strftime('%Y%m%d')
time_name = time.strftime('%H%M%S')
date_dir = backup_dir + '/' + date_name
target = date_dir + '/' + time_name + '.zip'

if not os.path.exists(date_dir):
    os.mkdir(date_dir)

command = 'zip -r ' + target + ' ' + source
status = os.system(command)

if status == 0:
    print('Backup completed')
else:
    print('Backup failed')

import os
import time

backup_dir = '/Users/swa/backup'
source = '/Users/swa/notes'
date_name = time.strftime('%Y%m%d')
time_name = time.strftime('%H%M%S')
date_dir = backup_dir + '/' + date_name
target = date_dir + '/' + time_name + '.zip'

if not os.path.exists(date_dir):
    os.mkdir(date_dir)

command = 'zip -r ' + target + ' ' + source
status = os.system(command)

if status == 0:
    print('Backup completed')
else:
    print('Backup failed')

Practice the Trace

EASY

Suppose the main backup directory is backup, the date string is 20240115, the time string is 180500, and the source is notes. Describe the date directory, the target zip filename, and the order of operations the script should follow.

Hints

The date string names the subdirectory.
The time string names the zip file.
Remember to check for the date directory before creating it.
The final command uses the target path and the source path.

Your attempt (not saved)

Worked example

Practice Solution

Use the supplied names to describe the resulting backup organization.

1
Build the date directory
Combine backup and 20240115 to identify the date-based directory backup/20240115.
2
Build the archive name
Add the time string and the zip extension to obtain 180500.zip.
3
Check the directory
Use os.path.exists on the date directory. Call os.mkdir only if it is absent.
4
Construct the command
Use the target archive path and notes as the inputs to the zip command, then pass the command to os.system.

Result

The target archive is backup/20240115/180500.zip, and the directory check happens before the archive command runs.

Key Takeaways

1
A hierarchical backup path uses a main directory, a date-based subdirectory, and a time-stamped zip file.
2
time.strftime('%Y%m%d') supplies the date directory name, while time.strftime('%H%M%S') supplies the time-based archive name.
3
os.path.exists checks whether the date directory is already present.
4
os.mkdir creates the date directory only when the check shows that it is absent.
5
os.system runs the zip command, and its return value is 0 for success and non-zero for failure.

At a glance

Hierarchical file naming makes repeated backups easier to locate and verify.
Date strings organize backups into subdirectories, while time strings distinguish archives made on the same day.
Checking with os.path.exists before calling os.mkdir allows the script to handle repeated runs.
The final zip command combines a destination path with source paths and runs through os.system.
The return value from os.system gives the script a way to identify success or failure.

More concepts starting with U

From Clutter to Structure

The Three-Level Backup Path

Turning Time into Names

Checking Before Creating

Building the Archive Command

Tracing a Second Backup

Mistakes Beginners Make

Practical Design Notes

Practice the Trace

Key Takeaways

At a glance

Related Topics