SGD and Backpropagation for Neural Networks

Backpropagation calculates the gradient of a loss with respect to a neural network's weights for one example (x, y). It does this by evaluating the network in two directions. The forward pass moves from the input toward the output and records the quantities produced by each layer. The backward pass starts at the output and moves toward earlier layers, combining the information needed to compute derivatives.

Think of the network as a sequence of composed functions. The input is the initial output, written as o_0 = x. At layer t, the network first forms a pre-activation vector by multiplying the layer's weight matrix by the previous output: a_t = W_(t-1)o_(t-1). It then applies the activation function to produce o_t = σ(a_t). The forward pass repeats this process from the input toward the final output.

The backward pass begins at the final layer. For the squared loss used in the source material, the final error vector is δ_T = o_T - y. Earlier error vectors are then computed by combining the next layer's weights, the next layer's error vector, and the derivative of the next layer's activation. This recursion moves the loss information toward the first layer.

The chain rule explains why an early weight cannot be differentiated by looking only at its immediate layer. A change in that weight changes the next layer's output, which changes later outputs, which changes the loss. The derivative of the full network is therefore assembled from the derivatives of these successive transformations, in the appropriate order.

A Jacobian organizes partial derivatives by output component and input variable. Its shape records how many outputs depend on how many inputs. This makes it useful for describing the derivative of a layer whose input and output are vectors rather than single numbers.

Consider the weights connecting layer t - 1 to layer t. The layer computes a_t = W_(t-1)o_(t-1), followed by o_t = σ(a_t). When differentiating with respect to the weights in this particular matrix, the earlier output o_(t-1) is treated as fixed. The derivative for an edge is then assembled from three local pieces: the error at the destination, the activation derivative at the destination, and the output at the source.

A gradient mismatch should be investigated as a sequence of checkpoints. First inspect the forward pre-activations a_t and outputs o_t. Then inspect the final error δ_T and the recursively computed earlier error vectors δ_t. Finally inspect the derivative assembled for each edge. This order follows the dependency structure of backpropagation and helps distinguish a forward error from a backward error or an edge-calculation error.

Backpropagation supplies the gradient of the loss with respect to the network's weights for one example. That gradient is the information an SGD procedure uses when working with the weights. The key boundary to remember is that backpropagation is the gradient-calculation procedure: it traces how the loss depends on the weights and produces the corresponding derivatives.