Neural Network Loss Functions

A neural network has a loss that depends on its weights. The network first transforms an input into a prediction, and the loss measures the result. To change the weights in a useful direction, we need to know how the loss changes when each individual weight changes. Backpropagation calculates this information: it calculates the gradient of the loss with respect to the weights, but it does not itself apply the weight update.

Consider a network divided into layers V0 through VT. The weights connecting layer Vt-1 to layer Vt are represented by a matrix Wt-1. The incoming values at a layer are collected in at, and the neuron outputs are collected in ot. The layer first computes at = Wt-1 ot-1 and then applies the activation function element by element to obtain ot = σ(at). These forward quantities are calculated from the input side toward the output side.

The gradient collects all of these partial derivatives. If the network has many weights, the gradient has one corresponding component for each weight. The component associated with a particular weight reports the loss sensitivity for that weight; it is not a general score detached from the network's individual connections.

A function can produce several outputs at once. When a function maps Rⁿ to Rᵐ, its derivatives are organized in a Jacobian matrix with m rows and n columns. The entry in row i and column j is the partial derivative of output i with respect to input variable j. Rows therefore correspond to output components, while columns correspond to input components.

A neural network composes many functions. A weight matrix produces pre-activation values, an activation function produces neuron outputs, later layers transform those outputs, and the final output determines the loss. The chain rule connects the derivative of the final loss to the derivatives of each intermediate transformation.

The order of the derivative operations matters because each derivative maps changes from one representation to the next. Applying the chain rule means combining local derivative information in the correct order. Backpropagation organizes this repeated multiplication recursively so that sensitivity arriving at one layer can be reused when calculating the sensitivity for the preceding layer.

Backpropagation has two directions of computation. During the forward pass, the network calculates the values at and ot from the input side toward the output side. During the backward pass, it uses those stored forward quantities while moving from the output side toward the input side. The forward values describe what the network computed; the backward sensitivities describe how strongly the later loss depends on those computed outputs.

At the final layer, the subnetwork loss is the loss function itself. For the squared loss Δ(u, y) = 1/2 ||u − y||², the derivative with respect to u is u − y. Therefore, at the final layer, δT = oT − y. This supplies the starting backward signal.

For an earlier layer, the chain rule expresses δt in terms of the sensitivity at the next layer, the derivative of the activation, and the transformation connecting the layers. The exact matrix arrangement follows from the Jacobians of those transformations. Repeating this step moves the sensitivity from the top layer toward the bottom layer.