Neural Network Training
Analytical minimization identifies derivative-zero points, but it is not practical for large neural networks.
From Optimal Weights to Incremental Learning
Training a neural network means finding weight values that produce the smallest possible loss. One tempting approach is to solve for the best weights directly. For a differentiable function, analytical minimization looks for points where the derivative is zero and then checks which candidate has the lowest function value. For a neural network, this would mean solving a gradient-equals-zero condition for all of the network's weights at once.
Why Direct Minimization Does Not Scale
The obstacle is the number of unknowns. The equation has one variable for each coefficient in the network. Analytical methods might be possible for a problem with two or three variables, but real neural networks have at least a few thousand parameters and can have several tens of millions. Solving the complete gradient-equals-zero system would therefore require a global calculation over an enormous number of variables.
| Approach | What it attempts | How parameters change |
|---|---|---|
| Analytical minimization | Find derivative-zero points and compare their function values | Attempts to solve the optimization problem globally |
| Stochastic gradient descent | Use the current loss and gradient from a random batch | Makes many small changes over repeated iterations |
The contrast between a global analytical calculation and repeated incremental updates.
A Batch Becomes an Update Signal
Stochastic gradient descent does not attempt to solve the entire neural-network optimization problem analytically. Instead, it selects a random batch of training data. The network uses that current batch to evaluate the current loss, compute a gradient, and guide a small parameter update. The changed parameters are then used in the next iteration.
Following One Batch
Trace what happens during one stochastic gradient descent iteration after a random batch has been selected.
- 1Select
A random batch of training data is chosen instead of using an analytical calculation over the entire optimization problem.
- 2Evaluate
The current network parameters are used with that batch to determine the current loss value.
- 3Compute
Because the loss-related function is differentiable, a gradient can be computed for the parameters.
- 4Modify
The parameters are changed a small amount in the direction opposite to the gradient.
- 5Continue
The modified parameters become the parameters used in the next iteration.
One batch does not finish training. It supplies the current loss and gradient for one incremental update.
Reading the Gradient Direction
For a collection of weights, the gradient describes how the loss changes with respect to the parameters. This makes the gradient the link between the current loss and the next parameter update. Stochastic gradient descent uses the opposite direction from the gradient because the goal is to reduce loss, not move in the direction associated with increasing it.
Suppose one parameter has a gradient pointing toward increasing loss. The update moves that parameter in the opposite direction. The new value is therefore not an arbitrary replacement: it is the old value after a small, gradient-guided modification. The same directional reasoning applies across the collection of weights.
The Repeating Training Cycle
The training algorithm repeats the same reasoning. It samples a random batch, evaluates the loss for the current parameters on that batch, computes the gradient, changes the parameters in the opposite direction, and then starts the next iteration with those changed parameters. Each cycle uses the current state rather than solving for all optimal weights in one step.
A single iteration should not be confused with a complete solution. Stochastic gradient descent replaces one impractical global calculation with many small parameter changes, so the important object to track is the sequence of updated parameter states.
Mistakes About Gradient Descent
Common mistake Treating the gradient as the new weight value.
What it looks likeReplacing a parameter with its gradient instead of using the gradient to guide a change.
Why it failsThe gradient describes how the loss changes with respect to the parameter. It is the link between the current loss and the update, not an arbitrary replacement value for the parameter.
Do this insteadKeep the current parameter and modify it a small amount in the opposite direction from the gradient.
Common mistake Assuming the update follows the gradient direction.
What it looks likeMoving a weight in the direction associated with increasing loss.
Why it failsThe goal is to reduce loss.
Do this insteadUse the direction opposite to the gradient.
Common mistake Assuming one random batch solves the whole training problem.
What it looks likeTreating the update based on one batch as the final set of optimal weights.
Why it failsA batch guides one incremental update, and training repeats the cycle with changed parameters.
Do this insteadView each batch as one step in a repeated process.
Common mistake Expecting analytical minimization to remain practical as the network grows.
What it looks likeApplying a method designed for a few variables directly to a network with thousands or millions of parameters.
Why it failsThe global equation has one variable for each network coefficient, making the calculation impractical for real neural networks.
Do this insteadUnderstand stochastic gradient descent as a replacement for that global calculation with many small parameter changes.
Check the Update Logic
Explain the complete path from a randomly selected batch to the next parameter state. In your explanation, identify where the current loss is evaluated, what the gradient describes, why the update uses the opposite direction, and why the process must repeat.
Hints
- Start with the batch rather than with the full training set.
- Connect the current loss to the computed gradient.
- Describe the parameter after the update as a modified version of its previous value.
What do you think happens?
A training iteration has computed a gradient for a parameter. Should stochastic gradient descent change the parameter in the same direction as the gradient or the opposite direction?
Reveal answer
The opposite direction
The gradient provides the direction associated with change in the loss, while the training goal is to reduce loss.
Training in One Sentence
- 1Analytical minimization looks for derivative-zero points, but the number of parameters in a real neural network makes the global calculation impractical.
- 2Stochastic gradient descent uses a random batch to obtain the current loss and guide one incremental update.
- 3The gradient describes how the loss changes with respect to the weights.
- 4Weights are modified in the opposite direction from the gradient to reduce loss.
- 5Training repeats batch sampling, loss evaluation, gradient computation, and parameter modification.
Key Takeaways
- Direct analytical minimization is impractical because a neural network can contain thousands to tens of millions of parameters.
- A random batch supplies the current data used to evaluate loss and compute a gradient.
- The gradient guides a small parameter change, and the change uses the opposite gradient direction.
- Training is a repeated cycle in which updated parameters become the starting point for the next iteration.