Backpropagation

Backpropagation is an algorithm for computing the gradient values of a neural network.

Programming

Interactive lab

Try it: Backpropagation, Step by Step

A forward pass through a 2-2-1 network, then the backward pass that applies the chain rule node by node to get for every weight and bias, and one gradient-descent update that changes the loss.

How it works

Forward: compute each hidden pre-activation and activation, then the sigmoid output y and the loss.
Backward from the loss: for squared error, or for binary cross-entropy.
Output weights: ; the error sent back to is .
Hidden units: multiply by the activation's local derivative, then .
Update every parameter and run the forward pass again to see the new loss.

Default run (12 steps): Inputs x = (1, 0.5), target t = 1. Hidden activation sigmoid, output sigmoid, loss: squared error L = (y - t)^{^2}. First the forward pass. … New forward pass with the updated weights: y = 0.6086, loss 0.203 -> 0.1532 (lower: the step moved downhill).

Simplified: One training example, two inputs, two hidden units, one output unit. Binary cross-entropy is evaluated from the output logit (same value, no log(0)). Toy dimensions, not a full real model.

Educational simulation

Loading the simulation…

From Data to Learning

A neural network performs computation by passing data through connected operations. The data is represented as tensors, and the final computation produces a loss value. Backpropagation begins with that final loss and moves backward through the connected operations to calculate gradient values. Those gradients provide information for optimizing the network's parameters.

Key point

Backpropagation is an algorithm for computing the gradient values of a neural network.

Tensor Representations

Tensors are the multi-dimensional arrays used to represent neural-network data. The word tensor describes the dimensional organization of the data: a scalar has no dimensions, a vector has one dimension, a matrix has two dimensions, and a higher-dimensional tensor has more than two dimensions. Tensor operations then manipulate the values or the organization of these representations.

Representation	Dimensions	Role in the neural-network data model
Scalar	No dimensions	A single value
Vector	One dimension	A one-dimensional collection of values
Matrix	Two dimensions	A two-dimensional organization of values
Higher-dimensional tensor	More than two dimensions	A multi-dimensional organization of values

Tensor representations differ by the number of dimensions used to organize their values.

What do scalars, vectors, matrices, and higher-dimensional tensors contain, and how do their dimensions organize neural-network data?

Tensor operations are the computational machinery of a neural network. Element-wise operations work across corresponding values. Broadcasting supports computation across dimensions when an operation uses differently organized tensors. Tensor dot operations combine tensor values through a dot operation. Reshaping changes the organization of values. These roles should be kept distinct: an operation may change values, organization, or both.

How can tensor operations change values, shape, or arrangement?

Forward Computation

A neural network can be viewed as a sequence of operations. The output of one operation becomes the input to another operation. As this forward computation proceeds, tensor operations manipulate the network's representations and produce a final result. The final result produces a loss value, which provides the starting point for the backward computation.

What data moves forward to produce the prediction, and what gradient information moves backward from the loss?

Worked example

Tracing a Connected Computation

Trace the direction of information through a neural network viewed as connected operations.

1
Represent
The network represents its data as tensors, which may be scalars, vectors, matrices, or higher-dimensional tensors.
2
Compute
Each operation receives input from an earlier operation and produces output for a later operation.
3
Measure
The final result produces a loss value.
4
Reverse
Backpropagation starts with that final loss and follows the connected operations backward.

Result

The forward route produces the loss; the reverse route computes gradient values that describe parameter contributions to that loss.

Reverse Gradient Trace

Backpropagation starts with the final loss and moves backward from the top layers toward the bottom layers. At each connected operation, it uses derivative information from the later part of the network together with the local derivative of the current operation. This backward traversal follows the reverse of the forward nesting order.

How does backpropagation move from the final loss backward through each connected operation to compute gradients?

The Chain Rule

A network is built from connected operations, so a parameter can affect the final loss through more than one operation. The chain rule combines the local derivatives of those connected operations. Backpropagation uses this chained derivative information to determine how each parameter contributed to the final loss.

How are local derivatives combined along a sequence of operations to determine how one parameter affects the final loss?

Worked example

Following One Parameter's Influence

Explain how the contribution of one parameter is determined when it affects a loss through connected operations.

1
Locate the parameter
Start with the parameter inside the forward sequence of tensor operations.
2
Reach the loss
Follow the forward connections from that operation to the final loss.
3
Reverse the route
Start at the final loss and move backward through the same connected operations in reverse order.
4
Combine local information
At every operation, combine the derivative information arriving from later operations with the local derivative of the current operation.

Result

The resulting gradient value represents how that parameter contributed to the final loss.

Gradient-Based Updates

Computation alone does not explain learning. Learning uses gradient-based optimization to optimize the network's parameters and minimize the loss function. The gradient is described as the derivative of a tensor operation, so gradient values provide derivative information about the computation. Stochastic gradient descent is one method used in this optimization process.

How does the loss produce gradients, and how do those gradients change network parameters during one training step?

The relationship between backpropagation and stochastic gradient descent is important: backpropagation chains derivatives so that the effects of tensor operations can contribute to parameter updates. Backpropagation computes the gradient information; gradient-based optimization uses that information to optimize parameters and minimize loss.

Common Tracing Mistakes

Common mistake
Treating backpropagation as another forward computation.
What it looks like
Beginning with the input tensor and moving toward the loss while trying to compute gradients.
Why it fails
Backpropagation begins with the final loss and moves backward through the connected operations.
Do this instead
Start at the loss, inspect the top operation, and continue toward earlier operations.
Common mistake
Ignoring the connected structure of the network.
What it looks like
Assigning a parameter's contribution to the loss from only the operation that directly contains the parameter.
Why it fails
A parameter's effect can pass through later connected operations before reaching the loss.
Do this instead
Use the chain rule to combine local derivatives along the connected route.
Common mistake
Confusing tensor computation with learning.
What it looks like
Assuming that performing tensor operations automatically optimizes the network's parameters.
Why it fails
Tensor operations provide the computation, but learning uses gradient-based optimization to optimize parameters and minimize loss.
Do this instead
Separate the forward computation, gradient calculation, and optimization step conceptually.
Common mistake
Treating all tensor operations as having the same role.
What it looks like
Describing reshaping as if it performed the same task as a tensor dot operation.
Why it fails
Element-wise operations, broadcasting, tensor dot operations, and reshaping manipulate values and organization in different ways.
Do this instead
Identify whether the operation works on corresponding values, supports computation across dimensions, combines tensor values, or changes organization.
Common mistake
Checking a backward calculation from the wrong end.
What it looks like
Searching for an error by starting at the earliest operation.
Why it fails
The backward route is defined by starting at the final loss and moving toward earlier operations.
Do this instead
Find the first operation in the backward route where later derivative information was not combined with the current local derivative according to the chain-rule structure.

Practice the Backward Route

MEDIUM

A network is described as three connected operations followed by a loss. Explain, in order, how you would trace backpropagation to determine the contribution of a parameter located in the first operation.

Hints

Begin with the final loss rather than the parameter.
Move through the operations in the reverse of their forward nesting order.
At each operation, identify the later derivative information and the current local derivative.
End by describing the resulting gradient as information about the parameter's contribution to the loss.

Your attempt (not saved)

What do you think happens?

A parameter appears in an early operation, while the loss is produced by later operations. Which direction should you follow to determine the parameter's gradient contribution?

From the parameter forward to the loss only
From the loss backward through the connected operations
From the tensor representation directly to stochastic gradient descent
From the loss to the input without inspecting intermediate operations

Reveal answer

Answer:

From the loss backward through the connected operations

Backpropagation starts with the final loss, follows the reverse of the forward nesting order, and combines later derivative information with each operation's local derivative.

Key Takeaways

1
Tensors are multi-dimensional arrays used to represent neural-network data; they may be scalars, vectors, matrices, or higher-dimensional tensors.
2
Tensor operations manipulate values, combine values, support computation across dimensions, or change organization.
3
A neural network's connected operations produce a final loss during the forward computation.
4
Backpropagation starts with the final loss and moves backward through the connected operations.
5
The chain rule combines local derivatives so gradient values can describe each parameter's contribution to the loss.
6
Stochastic gradient descent is a gradient-based optimization method that uses derivative information to optimize parameters and minimize loss.

At a glance

Neural networks represent data and computations with tensors.
Tensor operations can manipulate values, combine values, work across dimensions, or change organization.
Backpropagation computes gradient values by starting at the final loss and moving backward through connected operations.
The chain rule combines local derivatives to determine how parameters contributed to the loss.
Gradient-based optimization, including stochastic gradient descent, uses this derivative information to optimize parameters and minimize loss.

More concepts starting with B