Machine learning model capacity
Optimization and generalization measure different kinds of model performance.
Try it: Training a Keras-Style Model: History and Early Stopping
The Keras workflow on a tiny dense network: stack Dense layers (the summary counts units * (inputs + 1) parameters per layer), compile with an optimizer, loss and accuracy metric, fit for a number of epochs in mini-batches with validation data, and read the history. Training loss keeps falling while validation loss reaches its lowest point and then rises: the model starts to overfit, and EarlyStopping with patience can stop there and restore the best weights.
How it works
- Build a Sequential stack of Dense layers ending in Dense(1, sigmoid); each layer has units * (inputs + 1) weights and biases, initialised Glorot-uniform with zero biases.
- compile(): mini-batch SGD with the chosen learning rate, binary cross-entropy (or mean squared error) loss, and accuracy (prediction > 0.5) as the metric.
- fit(): each epoch walks the training set in batches of batch_size; every batch runs a forward pass, backpropagates the mean batch loss and updates every weight .
- After each epoch the history records the epoch's training loss and accuracy (batch-weighted average of values seen during the epoch) and the val_loss and val_accuracy on the held-out validation points.
- EarlyStopping(monitor="val_loss", patience) keeps the best val_loss; after patience epochs with no improvement it stops training, and restore_best_weights puts back the weights of the best epoch.
- evaluate() reports loss and accuracy on the validation data with the final weights.
Default run (153 steps): Sequential([Dense(16, "relu"), Dense(16, "relu"), Dense(1, "sigmoid")]) on inputs of shape (2,). model.summary(): dense 16 * (2 + 1) = 48; dense_1 16 * (16 + 1) = 272; dense_2 1 * (16 + 1) = 17; total 337 trainable parameters, initialised with Glorot uniform (biases 0). … model.evaluate(x_val, y_val) with the weights from epoch 150: loss 1.2082, accuracy 0.75. Training loss went 0.6964 -> 0.0194. Stopping at epoch 32 would have given val_loss 0.5272 and val_accuracy 0.775.
Simplified: A toy re-implementation in TypeScript, not Keras or TensorFlow: 2 input features, at most 3 hidden layers of up to 32 units, a seeded 2-D dataset (inside/outside a circle with label noise; 40 validation points), plain SGD without momentum, shuffle=False so batch order is fixed, float64 arithmetic, and binary cross-entropy computed from the output logit (same value as on probabilities, without clipping). The results match an independent NumPy implementation of the same procedure, not a real Keras run.
Loading the simulation…
Two Kinds of Improvement
A model can become better at handling the examples it has already seen while becoming worse at handling new examples. This is possible because optimization and generalization measure different kinds of performance. Optimization concerns improvement on the training data. Generalization concerns how reliably the learned behavior works on data the model has not seen.
The Training Journey
Training commonly begins with agreement between the two measures. As the model starts learning, its performance on training examples improves, and its performance on unseen examples often improves too. During this period, the model may still be underfit because it has not yet captured all the relevant patterns in the training data.
What do you think happens?
After validation performance reaches its strongest point, what can happen if training continues?
Reveal answer
Training performance can improve while validation performance stalls or worsens
The model may continue improving on familiar training examples while becoming increasingly tailored to details that do not transfer reliably to new data.
Underfitting and Overfitting
Underfitting means that the model has not represented all the relevant patterns in its training data. Further useful progress is still available.
Overfitting begins when the model learns details that belong specifically to the training data but do not provide reliable guidance for new data. Those details may be misleading or irrelevant outside the training set.
These terms describe different stages of the relationship between learning and transfer. Underfitting does not simply mean that the model performs poorly for an unexplained reason. It means the model has not yet captured all the useful structure available in the training data. Overfitting is different: the model has started using training-specific information that does not help it perform reliably on new data.
Useful Patterns and Training Details
A useful pattern captures a relationship that remains helpful when the model encounters new data. A training-specific pattern helps with the examples used during training but does not transfer reliably. The important test is not whether the model can improve its results on familiar examples; it is whether the improvement remains visible on unseen examples.
A topic-classification model
A model is trained to classify the topics of documents. Early in training, both its training performance and its validation performance improve. Later, training performance continues to improve, but validation metrics stop improving and then worsen. What is happening?
- 1Early training
The model is learning patterns that help with the training examples and also appear useful on unseen examples. This is consistent with progress away from underfitting.
- 2Validation peak
Validation performance reaches its strongest point. At this point, the model is showing its best observed generalization during the training process.
- 3Continued training
Training performance keeps improving, but validation performance stalls and then declines. The model is increasingly learning details specific to the training data.
- 4Diagnosis
Optimization is continuing, but generalization is no longer improving. The model has moved into overfitting.
The model should be evaluated by its validation behavior, not by training performance alone. Continued training after the validation peak is associated with overfitting in this example.
Model Capacity and Fit
Model capacity concerns how much information a model can retain or represent. The source describes reducing overfitting partly by limiting what the model can retain or by placing constraints on the information it is allowed to store. When unrestricted memorization is made less available, optimization is pushed toward the most prominent patterns, which have a better chance of generalizing.
Reducing Overfitting
The strongest general remedy described for overfitting is obtaining more training data. More examples give the model more information from which to learn and make it more likely to develop patterns that generalize.
When more data is not possible, reduce the amount of information the model can retain or place constraints on what it is allowed to store. These approaches make unrestricted memorization less available and encourage the optimization process to focus on the most prominent patterns.
- 1Prefer obtaining more training data when that is possible.
- 2If more data is unavailable, limit what the model can retain or constrain the information it is allowed to store.
- 3Track validation performance while training continues.
- 4Treat a stall or decline in validation performance, alongside continued training improvement, as evidence that the model is moving into overfitting.
- 5Favor the point in training where validation performance is strongest rather than relying only on the best training performance.
Mistakes to Avoid
Common mistake Assuming that better training performance always means better model performance.
What it looks likeTraining performance continues to improve after validation performance has begun to decline.
Why it failsOptimization and generalization measure different kinds of performance. Improvement on familiar examples can coexist with worsening performance on unseen examples.
Do this insteadCompare training behavior with validation behavior.
Common mistake Calling every poorly performing model overfit.
What it looks likeA model has not yet captured all relevant patterns in its training data.
Why it failsThis describes underfitting, not overfitting. Further useful progress is still available.
Do this insteadUse underfitting when relevant training patterns have not yet been represented.
Common mistake Treating validation performance as unnecessary once training performance is high.
What it looks likeA topic-classification model is judged only by its results on the examples used for training.
Why it failsValidation performance helps reveal when continued training has stopped improving generalization.
Do this insteadContinue checking performance on data the model has not seen.
Common mistake Assuming that a training-specific detail is useful because it improves familiar examples.
What it looks likeThe model learns a detail that helps classify the training documents but does not provide reliable guidance for new documents.
Why it failsOverfitting begins when such details replace patterns that transfer reliably.
Do this insteadAsk whether the pattern continues to help on unseen data.
Check Your Understanding
A model's training performance improves throughout training. Its validation performance improves at first, reaches a strongest point, and then begins to decline. Identify the training phase before the strongest validation point, the phase after it, and the difference between optimization and generalization in this situation.
Hints
- Before the strongest validation point, the model is still learning useful patterns.
- After the strongest validation point, compare the directions of training and validation performance.
- Optimization concerns training examples; generalization concerns unseen examples.
Practice answer
Interpret the changing training and validation performance.
- 1Before the validation peak
The model is in an underfitting phase if it has not yet captured all relevant patterns and both training and unseen-data performance are improving.
- 2After the validation peak
The model is moving into overfitting when training performance continues to improve but validation performance stalls and then worsens.
- 3Meaning of the two measures
Optimization is continuing because performance on training examples improves. Generalization is no longer improving because performance on unseen examples has stopped improving or declined.
The model has changed from underfitting to overfitting as training continued.
Key Takeaways
- 1Optimization measures how performance changes on training examples, while generalization concerns performance on unseen examples.
- 2Underfitting means that relevant training patterns have not yet been captured.
- 3Overfitting begins when training-specific details improve familiar examples but fail to transfer reliably to new data.
- 4A model can move from underfitting to overfitting: both measures may improve at first, then validation performance can stall or worsen while training performance continues improving.
- 5More training data is the strongest general remedy described; when it is unavailable, limiting what the model can retain or constraining stored information can reduce unrestricted memorization.
At a glance
- Optimization and generalization measure different kinds of model performance.
- Underfitting can become overfitting as training continues beyond the point of strongest validation performance.
- Useful patterns transfer to new data, while training-specific details do not transfer reliably.
- More training data, reduced information retention, and constraints on stored information are practical ways to reduce overfitting.