Chain Rule
The chain rule provides the structure for relating the effect at one stage to the effects of the stages that came before it.
Try it: Backpropagation, Step by Step
A forward pass through a 2-2-1 network, then the backward pass that applies the chain rule node by node to get for every weight and bias, and one gradient-descent update that changes the loss.
How it works
- Forward: compute each hidden pre-activation and activation, then the sigmoid output y and the loss.
- Backward from the loss: for squared error, or for binary cross-entropy.
- Output weights: ; the error sent back to is .
- Hidden units: multiply by the activation's local derivative, then .
- Update every parameter and run the forward pass again to see the new loss.
Default run (12 steps): Inputs x = (1, 0.5), target t = 1. Hidden activation sigmoid, output sigmoid, loss: squared error L = (y - t)2. First the forward pass. … New forward pass with the updated weights: y = 0.6086, loss 0.203 -> 0.1532 (lower: the step moved downhill).
Simplified: One training example, two inputs, two hidden units, one output unit. Binary cross-entropy is evaluated from the output logit (same value, no log(0)). Toy dimensions, not a full real model.
Loading the simulation…
Why the Passes Matter
Backpropagation trains a neural network by helping reduce the error between its predicted output and the actual output. Its central task is to compute gradients of the loss with respect to the model parameters. To organize that work, backpropagation separates computation into two passes: a forward pass that produces the network output and a backward pass that computes the gradients needed to update the parameters.
The forward pass answers: What output does the network produce? The backward pass answers: How should error information move through the network so parameter gradients can be computed?
Forward Pass Through Layers
The forward pass begins with an input and processes it through the network's layers. At each stage, the network computes weighted sums and then activations. The resulting activated outputs become the information passed to the next layer. This continues until the network computes its final output, which is the prediction used to compare against the actual output.
Tracing One Forward Pass
Trace the conceptual path of an input through a network with two processing stages.
- 1Start with the input
The input enters the first layer.
- 2Compute the first weighted sum
The first layer combines the incoming information through its weighted-sum computation.
- 3Apply the first activation
The first layer produces an activated output that is passed to the next layer.
- 4Repeat the layer computation
The next layer computes another weighted sum and activation using the information passed forward.
- 5Produce the final output
The final activated result becomes the network's prediction.
The forward pass must finish computing the final output before the backward pass can begin.
Backward Error Flow
After the final output is available, the network can determine the error between the prediction and the actual output. The backward pass then moves error information from later layers toward earlier layers. The chain rule provides the structure for relating the effect at one stage to the effects of the stages that came before it.
For each step of the backward computation, the relevant information comes from the later layer's error quantity, the connecting weights, and the activation derivative. This lets the procedure propagate error information toward earlier layers and compute gradients for the connections, also called edges, in the network.
Tracing One Backward Pass
Trace what happens after a network has produced a prediction that differs from the actual output.
- 1Locate the final error
The backward pass begins only after the final output is available and its error has been determined.
- 2Start at the later layer
The procedure begins propagating error information from the output side of the network.
- 3Use local information
The computation uses the next layer's error quantity, the connecting weights, and the activation derivative.
- 4Compute an edge gradient
The chain rule connects the error information and local derivative information so the gradient for a connection can be calculated.
- 5Continue toward earlier layers
The same backward pattern continues from later layers toward earlier layers.
The backward pass turns the final error into gradients that describe how the model parameters contributed to that error.
Forward and Backward Compared
| Forward pass | Backward pass |
|---|---|
| Computes weighted sums, activations, and the final output. | Uses the chain rule to propagate error information and compute gradients. |
| Moves from the input through later layers. | Moves from later layers toward earlier layers. |
| Must occur before the final error is available. | Begins after the final output and its error are available. |
The two passes have different directions and purposes, but they form one backpropagation procedure.
Checking the Procedure
A correct trace preserves a specific order: forward computation, final error, backward computation, and edge-gradient calculation. To find a divergence, inspect the phase order rather than looking only at an individual calculation.
Common mistake Starting the backward pass before the forward pass has produced the final output.
What it looks likeA trace begins with an output error even though the weighted sums, activations, and final output have not yet been computed.
Why it failsThe final error depends on having the network output available.
Do this insteadComplete the forward computation first, then determine the final error.
Common mistake Moving error information from earlier layers toward later layers during the backward pass.
What it looks likeThe procedure begins with an early layer instead of starting from the output side.
Why it failsThe expected backward direction is from later layers toward earlier layers.
Do this insteadBegin with the final error and trace the error information backward through the layers.
Common mistake Treating the chain rule as the forward computation.
What it looks likeA trace describes the chain rule as the process that computes the network's prediction.
Why it failsThe forward pass computes the output; the chain rule is used during the backward pass to compute gradients.
Do this insteadSeparate prediction-making from gradient computation.
Common mistake Skipping the edge-gradient calculation.
What it looks likeA trace reaches the backward computation but never accounts for gradients associated with the connecting edges.
Why it failsComputing gradients for model parameters is the central task of backpropagation.
Do this insteadCheck that the backward trace includes the connecting weights, activation derivative, and edge-gradient calculation.
Practice the Trace
A learner writes this sequence: final error, forward weighted sums, backward computation, activations, edge-gradient calculation. Identify the first point where the sequence diverges from the expected backpropagation order, and rewrite the sequence in the correct phase order.
Hints
- Ask whether the final error can be determined before the final output exists.
- Separate all forward-pass actions from all backward-pass actions.
- Remember that edge-gradient calculation belongs after the backward computation begins.
Checking the Practice Sequence
Find the first divergence in the sequence: final error, forward weighted sums, backward computation, activations, edge-gradient calculation.
- 1Check the first step
The sequence begins with the final error, but the forward computation has not yet produced the final output.
- 2Identify the divergence
The first step is the divergence because the forward pass must come before the final error.
- 3Restore the forward phase
The weighted sums and activations must be computed through the layers so the final output becomes available.
- 4Begin the backward phase
After the final error is available, the backward computation can move from later layers toward earlier layers.
- 5Finish with gradients
The procedure then calculates gradients for the connecting edges.
A correct high-level order is forward weighted sums and activations, final output, final error, backward computation, and edge-gradient calculation.
Summary
- 1The forward pass computes weighted sums, activations, and the final output.
- 2The final error must be available before the backward pass begins.
- 3The backward pass uses the chain rule to move error information from later layers toward earlier layers.
- 4Connecting weights and activation derivatives contribute to edge-gradient calculations.
- 5The expected order is forward computation, final error, backward computation, and edge-gradient calculation.
At a glance
- The forward pass transforms an input through weighted sums and activations until it produces a final output.
- The backward pass begins with the error between the prediction and actual output.
- The chain rule organizes how error information moves backward and how gradients are computed.
- A reliable trace preserves the order from forward computation to final error, backward computation, and edge-gradient calculation.
- To diagnose a divergence, check both the phase order and the direction of information flow.