Calculus

The purpose of backpropagation is to determine how the network's parameters contributed to the final loss.

Programming

From Loss to Learning Information

A neural network can be viewed as a sequence of connected operations. One operation produces an output that becomes the input to another operation, and the final result produces a loss value. Backpropagation is the algorithm that starts with this final loss and works backward through the connected operations to compute gradient values.

Key point

The purpose of backpropagation is to determine how the network's parameters contributed to the final loss.

The Forward Route and Its Reverse

Imagine following the network in the order in which its operations are connected. An earlier operation produces a value, a later operation uses that value, and the final operation produces the loss. This is the forward nesting order: later operations depend on results produced earlier in the network.

What is the difference between the forward route that produces the loss and the backward route that propagates gradient information?

Definition

Backpropagation is an algorithm for computing the gradient values of a neural network.

Why the Chain Rule Is Necessary

A network is not one isolated operation. It is built from connected operations, so an earlier value can affect the loss through the later operations that use it. To determine the effect of that earlier value, backpropagation must combine the derivative information from the later part of the network with the local derivative of the operation currently being examined. This is the role of the chain rule.

How are the local derivatives of successive operations combined to determine how an earlier value affects the final loss?

For a nested network, the backward traversal follows the reverse of the forward nesting order. At each operation, the information already obtained from later operations is combined with the local derivative of the current operation. Repeating this process carries gradient information toward earlier operations and their parameters.

Tracing One Parameter's Influence

Worked example

Following a Parameter Backward

Trace how backpropagation determines the contribution of one parameter in an abstract connected network.

1
Start at the loss
Begin with the final loss because it is the endpoint produced by the forward sequence of operations and the starting point for the backward traversal.
2
Inspect the latest operation
Examine the operation immediately before the loss. Use the gradient information from the later part together with this operation's local derivative.
3
Move toward earlier operations
Continue in reverse order through the connected operations. At each point, combine the information arriving from later operations with the local derivative of the current operation.
4
Reach the parameter
When the backward traversal reaches the operation associated with the parameter, the accumulated gradient information describes how that parameter contributed to the final loss.

Result

The parameter's contribution is determined by following the reverse route from the final loss and applying the chain-rule combination at each connected operation.

How does the influence of a specific parameter on the loss get determined along the connected path?

The important idea is not to inspect a parameter in isolation. Its contribution is determined by the entire connected route from that parameter to the final loss. Every later operation on that route contributes local derivative information that must be incorporated during the backward traversal.

Checking a Backpropagation Trace

1
Begin at the final loss.
2
Inspect the operation immediately before the loss.
3
Move toward earlier operations in the reverse of the forward nesting order.
4
At each operation, check that later gradient information is combined with the operation's local derivative.
5
When a result diverges from what is expected, locate the first operation where that backward combination no longer follows the chain-rule structure.

Common Direction and Chain-Rule Errors

Common mistake
Starting the backward calculation at the earliest operation.
What it looks like
Inspecting the first operation before considering the final loss.
Why it fails
Backpropagation starts with the final loss and moves backward toward earlier operations.
Do this instead
Begin at the final loss, then follow the reverse of the forward nesting order.
Common mistake
Treating connected operations as if they were independent.
What it looks like
Using only the local derivative of one operation without considering information from later operations.
Why it fails
The chain rule combines local derivatives of connected operations.
Do this instead
At each operation, combine the derivative information from the later part of the network with the local derivative of the current operation.
Common mistake
Stopping after examining the operation nearest to the loss.
What it looks like
Computing information for the top operation but never tracing it toward earlier operations or parameters.
Why it fails
The purpose of backpropagation is to determine how the network's parameters contributed to the loss.
Do this instead
Continue the reverse traversal until the relevant earlier operations and parameters have been reached.
Common mistake
Looking only at the final result when checking a calculation.
What it looks like
Seeing that a gradient value is incorrect but not checking the intermediate backward combinations.
Why it fails
A divergence can be localized to the first operation where the backward chain-rule structure stops being followed.
Do this instead
Check the traversal one operation at a time, beginning at the final loss.

Practice the Reverse Trace

MEDIUM

A network is described as three connected operations followed by a final loss. Explain the order in which backpropagation examines these operations, and describe what must be combined at each operation to determine the gradient information for an earlier parameter.

Hints

Where does the backward traversal begin?
What is the relationship between the forward nesting order and the backward order?
What two kinds of derivative information are combined at each operation?

Your attempt (not saved)

Key point

A complete answer should say that the traversal begins at the final loss, proceeds toward earlier operations in reverse order, and combines later gradient information with each current operation's local derivative.

What to Remember

1
Backpropagation is an algorithm for computing gradient values in a neural network.
2
A neural network can be understood as connected operations whose final result produces a loss.
3
Backpropagation starts at the final loss and moves backward from later operations toward earlier ones.
4
The chain rule combines local derivatives across connected operations.
5
A parameter's contribution to the loss is determined by tracing the backward route from the loss through the operations connected to that parameter.

At a glance

Backpropagation computes neural-network gradient values.
It begins with the final loss and follows the reverse of the forward nesting order.
The chain rule is required because operations are connected and an earlier value affects the loss through later operations.
At each step, later gradient information is combined with the local derivative of the current operation.
Following this process to a parameter determines how that parameter contributed to the final loss.

More concepts starting with C