Evaluating a Convnet

Evaluating a convolutional neural network begins before any model code runs.

Programming

From Image Collection to Evaluation

Evaluating a convolutional neural network begins before any model code runs. The images must first be arranged so that the training process can find them by both subset and class. In the Dogs vs. Cats task, that means creating separate locations for training, validation, and test images, with separate cat and dog directories inside each location.

Definition

The original Dogs vs. Cats dataset contains 25,000 medium-resolution color JPEG images: 12,500 cat images and 12,500 dog images.

Key point

The working dataset is smaller than the original collection. It is divided into training, validation, and test subsets, and every subset contains separate cats and dogs directories.

The Three-Way Dataset Split

The dataset is organized into three named subsets: training, validation, and test. This division gives the convnet workflow distinct groups of images to work with instead of one undifferentiated collection. The subset name records where an image belongs in the workflow, while the class directory records whether the image is a cat or a dog.

The selected image ranges determine which original files belong to each subset. The important preparation task is therefore not only choosing images, but also placing every selected file in the directory that matches both its subset and its class.

How the original image collection becomes three organized subsets for the convnet workflow.

Required Folder Hierarchy

Each subset needs two class directories. The resulting hierarchy has a training directory containing cats and dogs, a validation directory containing cats and dogs, and a test directory containing cats and dogs. This two-level organization lets the image-loading process identify both the subset and the class from the file location.

What folder structure is required before the convnet can use the split images by subset and class.

Copying Files into Place

Dataset preparation follows a repeated routing operation. First, a selected original image is identified as belonging to the cat or dog class. Next, its selected range determines whether it belongs in training, validation, or test. Finally, the file is copied into the matching class directory inside that subset.

How selected cat and dog files move from the original collection into the matching subset and class directories.

Worked example

Tracing One Class Through the Split

Explain the destination of selected cat images as the working dataset is prepared.

1
Identify the class
The source files are cat images, so they must be routed to cats directories rather than dogs directories.
2
Apply the selected range
The selected image range determines whether a particular cat file belongs to training, validation, or test.
3
Copy to the destination
The file is copied into the cats directory inside the subset selected by its range.

Result

Every selected cat file ends in exactly one class directory that also identifies its subset.

Counting Every Destination

Copying files is not the final step. A preparation failure can happen before any model code runs, so the directories should be checked directly. The source procedure obtains the contents of each directory with os.listdir() and counts those contents with len(). The resulting counts can then be compared with the expected count for that directory.

Directory	Expected image count
Training cats	1,000
Training dogs	1,000
Validation cats	500
Validation dogs	500
Test cats	500
Test dogs	500

Expected counts for the six class directories in the working dataset.

How to compare the count obtained from each class directory with its expected count.

Worked example

Checking the Six Counts

Verify the expected contents of the working dataset.

1
Count training classes
The training cats directory should contain 1,000 images, and the training dogs directory should also contain 1,000 images.
2
Count validation classes
The validation cats directory should contain 500 images, and the validation dogs directory should also contain 500 images.
3
Count test classes
The test cats directory should contain 500 images, and the test dogs directory should also contain 500 images.
4
Compare results
Compare each directory count with its corresponding expected value rather than checking only one directory.

Result

The preparation check covers six directories: two classes in each of three subsets.

Preparation Mistakes

Common mistake
Creating only one class directory inside a subset.
What it looks like
A validation directory exists, but it has no separate cats and dogs directories.
Why it fails
Every split is required to contain separate cats and dogs directories.
Do this instead
Create both class destinations inside training, validation, and test.
Common mistake
Copying a file to a subset without matching its class.
What it looks like
A cat image is placed in a dogs directory.
Why it fails
The destination directory identifies the image class.
Do this instead
Route cat files to cats directories and dog files to dogs directories.
Common mistake
Checking only one directory after copying.
What it looks like
The training cats count is checked, but the other five class directories are not counted.
Why it fails
A preparation failure can affect any subset or class directory.
Do this instead
Use directory contents and their lengths to verify all six expected counts.
Common mistake
Assuming that successful copying proves the dataset is ready.
What it looks like
Files have been copied, but the directory counts do not match the expected values.
Why it fails
The source identifies counting as the most useful first check for a dataset-preparation failure.
Do this instead
Compare every actual directory count with its expected count before training.

Practice the Routing Check

EASY

A working dataset has the correct training cats count and the correct validation counts, but the test dogs directory contains fewer images than expected. Which part of the preparation process should you inspect first, and which expected count should you use for comparison?

Hints

Identify the subset and class named by the directory.
Use the expected-count table rather than another directory's count.

Your attempt (not saved)

What do you think happens?

What should the test dogs directory be compared with?

1,000 images
500 images
The total number of original dog images

Reveal answer

Answer:

500 images

The expected count is determined by both the subset and the class. Test dogs is one of the two test class directories, and each test class directory is expected to contain 500 images.

What to Remember

1
The original Dogs vs. Cats dataset contains 25,000 color JPEG images: 12,500 cats and 12,500 dogs.
2
The working dataset is divided into training, validation, and test subsets.
3
Every subset must contain separate cats and dogs directories.
4
Selected image ranges determine which files are copied into each destination.
5
Use os.listdir() and len() to count directory contents and check all six expected counts: 1,000 for each training class and 500 for each validation and test class.

At a glance

A convnet cannot use the Dogs vs. Cats images reliably until the files are arranged by subset and class.
The required structure has training, validation, and test directories, each containing cats and dogs directories.
Selected image ranges determine where files are copied.
Directory counts provide an early check for dataset-preparation errors.
The expected counts are 1,000 training cats, 1,000 training dogs, 500 validation cats, 500 validation dogs, 500 test cats, and 500 test dogs.

Related concepts

More concepts starting with E