Residual Connections
Residual connections are one important use of this structure: an earlier activation keeps a direct route to a later point, where it is added to the result of the intervening layers.
Following More Than One Route
A sequential model gives you one obvious route: one layer feeds the next. A layer graph can be more flexible. One tensor can travel down several routes, each route can process it differently, and the routes can later come together. Residual connections are one important use of this structure: an earlier activation keeps a direct route to a later point, where it is added to the result of the intervening layers.
What do you think happens?
A tensor splits into two forward branches and those branches later merge. Does the split and merge alone create a cycle?
Reveal answer
No, because both branches can still move forward.
A graph may branch and rejoin while remaining acyclic. A cycle would require a later result to travel back and become input to a layer that helped produce that result.
Forward-Only Layer Graphs
A directed acyclic graph is a collection of connected, directed routes that move forward without eventually returning to an earlier point in the computation. In a Keras layer graph, tensors may branch, rejoin, enter through multiple inputs, or leave through multiple outputs, but a tensor cannot pass through layers and then become input to one of the layers that produced that tensor.
The important restriction is not that every graph must be a single chain. The restriction is that computation must keep moving in directed order. A branch can create several forward routes, and a merge can collect those routes, but the resulting tensor cannot be sent back to an earlier layer that helped generate it. This forward-only structure is what keeps the layer graph acyclic.
Branching and Merging
Tracing Parallel Processing
Follow one input tensor through an Inception-style structure with several processing branches.
- 1Start
One input tensor enters the module and becomes the starting point for several small processing paths.
- 2Split
The branches process the same input differently. A branch may use a 1 × 1 convolution, another may use a spatial convolution, and another may use pooling.
- 3Process
Each branch learns a different view of the input. The branches do not need to perform identical work.
- 4Merge
The resulting branch features are concatenated into the module output, collecting the different views into one tensor.
The tensor follows multiple forward routes and becomes one combined output without creating a cycle.
The merge operation determines what compatibility the branches need. In the Inception-style example, the branch outputs are concatenated. The purpose is to collect several learned views of the same input. The branches can perform different kinds of work because their value comes from learning different features before the results are collected.
Pointwise Channel Mixing
A 1 × 1 convolution looks at one spatial tile at a time. Its receptive patch contains a single tile, so it does not combine information across neighboring positions. Instead, it mixes the channels at that position. The source describes this as equivalent to sending each tile vector through a Dense layer.
| Operation | Information combined | Role in a multi-branch module |
|---|---|---|
| 1 × 1 convolution | Channels at one spatial position | Channel-wise feature learning |
| Larger spatial convolution | Information across nearby positions | Spatial feature learning |
| Pooling | A pooled view of the input | An alternative processing path |
The branch operations can provide different views of the same input.
The Residual Route
A residual connection adds a second route to a later layer. The main route passes through the intervening layers. The skip route carries the output of an earlier layer directly to the later point. At that point, the two activations are summed rather than concatenated. The earlier activation therefore remains available alongside the transformation performed by the intervening layers.
Checking a Residual Merge
A graph carries an earlier activation through a skip route while another copy passes through intervening layers. Determine what to inspect before the merge.
- 1Trace the main route
Follow the earlier activation through each intervening layer and note any operation that changes spatial size or channel structure.
- 2Trace the skip route
Confirm that the earlier activation reaches the later merge point directly, or identify a transformation used to make the routes compatible.
- 3Check the merge
Because the routes are added, their activation sizes must match, or a transformation must make them match.
- 4Interpret the result
The later output contains the combined result of the transformed main route and the preserved earlier route.
A residual connection is defined by addition of the earlier route and the later route, not merely by skipping a layer.
Finding the First Divergence
When a graph produces an unexpected result, inspect it in directed order rather than treating the model as one uninterrupted chain. Begin with the tensor entering the split. Check each branch independently. Pay particular attention to operations that change spatial size or channel structure. Then inspect the merge and ask whether its compatibility requirement is satisfied.
- 1Start at the tensor entering the split.
- 2Trace each branch separately instead of assuming that all routes perform the same work.
- 3Inspect operations that change spatial size or channel structure.
- 4At a concatenation merge, check that the branch outputs are compatible for concatenation.
- 5At an addition merge, check that the activation sizes match or that a transformation makes them match.
- 6Treat the first point where the observed structure or value stops matching expectations as the likely divergence point.
The most useful debugging question is precise: which path did the tensor follow, and at which operation did its structure or value stop matching the expectation? This question turns a complex graph into a sequence of inspectable routes.
Graph-Reading Practice
A graph begins with one tensor, sends it through two branches, and later combines the results by addition. The first branch passes through an operation that changes channel structure. The second branch carries the earlier activation directly to the merge. What should you inspect first if the merge produces an unexpected result?
Hints
- Follow the tensor from the split in directed order.
- Compare the activation sizes arriving from both branches.
- Pay special attention to the operation that changes channel structure.
- 1A layer graph can branch and merge while remaining acyclic, as long as tensors continue forward and do not return to an earlier generating layer. Parallel branches provide different views of one input, and their outputs can be concatenated. A 1 × 1 convolution mixes channels at one spatial position without combining neighboring positions. A residual connection keeps an earlier activation on a skip route and adds it to a later activation, so the two routes need matching activation sizes or a transformation that makes them match. For debugging, trace every route in directed order and locate the first branch or merge where the tensor no longer matches expectations.
Key Takeaways
- Directed acyclic layer graphs may branch and merge, but they cannot send a result back to an earlier layer that helped produce it.
- Parallel branches process one input in different ways and can later be combined, such as by concatenation.
- A 1 × 1 convolution mixes channel information at one spatial position without combining neighboring positions.
- A residual connection adds an earlier activation to a later route, so the activation sizes must match or be made compatible.
- To debug an unexpected result, trace each route from the split and identify the first operation where the structure or value diverges.