Subsampling layers

Deep convolutional networks organize processing through convolutional, subsampling, and fully connected layers.

Programming

From Image to Recognition

A deep convolutional network does not treat handwritten-character recognition as one indivisible operation. It organizes processing as a sequence. Convolutional layers and subsampling layers alternate, and the resulting representations are then passed to fully connected final layers. This sequence gives the network a route from local information in an input image to recognition of the handwritten character.

What happens to an image and its representation as it moves through convolutional, subsampling, and fully connected layers?

A First-Layer Feature Map

A feature map is a collection of responses produced by units at different locations. Those units apply shared weights at shifted locations, so the map records where the same learned operation responds across the incoming array. The operation is not separately learned for every position. Instead, units in the same feature map use shared weights.

Worked example

Reading the first convolutional layer

Interpret the source network's first-layer feature maps.

1
Identify the maps
The first layer has 6 feature maps.
2
Identify the units
Each map has 28 × 28 units, so each map is an organized collection of responses across locations.
3
Identify each local view
Each unit uses a 5 × 5 receptive field from the incoming array.
4
Identify the learned parameters
Each map has 25 adjustable weights associated with its shared operation.

Result

The first layer produces six spatially organized activity patterns. Within any one map, different units apply the same learned weights at shifted locations.

How does the same learned filter detect a feature at multiple positions in an input image?

Receptive Fields and Overlap

Definition

A receptive field is the local input region seen by one unit. It specifies which part of the incoming array can contribute to that unit's response.

In the source network, first-layer units have 5 × 5 receptive fields. Neighboring units can view regions that overlap. Specifically, the first-layer fields overlap by four columns and five rows. This overlap describes how neighboring units draw their local views from the incoming array. It does not mean that each unit has its own separate set of weights: units in the same feature map still use shared weights.

Which input pixels contribute to each feature-map value, and how can neighboring feature-map values depend on overlapping regions?

Key point

A feature-map value has a unit-level explanation and a map-level explanation. At the unit level, its receptive field identifies the local input region. At the map level, its operation is related to the responses at other locations because the weights are shared.

What Subsampling Changes

Subsampling layers are the stages that follow convolutional stages in the alternating architecture. Their role in the representation is to change the spatial resolution of feature maps while preserving useful detected features. The result is not a single number. It is a collection of activity patterns that continues through later convolutional and subsampling stages.

How does subsampling reduce the spatial size of feature maps while preserving useful detected features?

Building Character Evidence

The architecture can be understood as a gradual organization of information. Convolutional layers create local activity patterns by applying shared operations at different locations. Subsampling layers transform those patterns into later representations. Repeated convolutional and subsampling stages prepare the signal for fully connected final layers, which receive the resulting representations. In this way, the network provides a route from local input information to the recognition of a handwritten character.

How do local features detected in early layers become character-level evidence and a final recognition result?

Worked example

Tracing one recognition pathway

Trace how information can move through the architecture when the input is a handwritten character.

1
Start with local information
Units in an early convolutional layer view local input regions through their receptive fields.
2
Form feature-map activity
Shared weights are applied at shifted locations, creating activity patterns across feature maps.
3
Pass through subsampling
The activity patterns move through subsampling stages as the network organizes the representation.
4
Repeat the preparation
Convolutional and subsampling layers alternate, preparing the signal for later processing.
5
Reach the final layers
Fully connected final layers receive the representations produced by the earlier stages.

Result

The network moves from local views and shared feature responses toward representations that support handwritten-character recognition.

Mistakes to Avoid

Common mistake
Treating a feature map as if every location learned a different operation.
What it looks like
Describing each unit in a map as having its own unrelated weights.
Why it fails
Units in the same feature map apply shared weights at shifted locations.
Do this instead
Describe the map as responses from one shared learned operation applied across locations.
Common mistake
Confusing a receptive field with the whole feature map.
What it looks like
Saying that one unit sees every value in the feature map.
Why it fails
A receptive field specifies the local input region seen by one unit.
Do this instead
Use the unit-level view: one first-layer unit has a 5 × 5 receptive field in the source network.
Common mistake
Assuming overlapping receptive fields require separate weights.
What it looks like
Claiming that overlap means neighboring units must have different learned parameters.
Why it fails
Overlap concerns the local input regions, while weight sharing concerns the learned operation.
Do this instead
Explain both facts together: neighboring fields may overlap, and units in the same map still share weights.
Common mistake
Leaving subsampling out of the network sequence.
What it looks like
Describing the architecture only as convolutional layers followed immediately by fully connected layers.
Why it fails
The described network alternates convolutional and subsampling layers before the fully connected final layers.
Do this instead
State the sequence at the network level before explaining individual units.
Common mistake
Explaining recognition as a single feature-map response.
What it looks like
Treating one local response as the character-level result.
Why it fails
The network produces collections of activity patterns and passes their resulting representations to fully connected final layers.
Do this instead
Trace the representation across stages from local information to the final layers.

Check Your Understanding

MEDIUM

Explain the first layer of the source network in three levels of description: the network-level sequence, the feature-map-level operation, and the unit-level receptive field.

Hints

At the network level, name the layer types that alternate before the fully connected final layers.
At the feature-map level, explain what is shared across locations.
At the unit level, include the first-layer receptive-field size and explain how neighboring fields can overlap.

Your attempt (not saved)

MEDIUM

A learner says, “The six first-layer feature maps each contain unrelated responses because every location has separate weights.” Correct the statement using the source network's 28 × 28 maps, 5 × 5 receptive fields, and shared weights.