Overfitting and Underfitting
Overfitting is reduced by making memorization less easy and making generalizable representations more useful.
Try it: Dropout at Training vs Inference
How dropout zeroes a random subset of a layer's outputs during training (with a visible seeded mask), how inverted dropout scales the survivors by 1 / (1 - rate) while inference is a no-op (or classic dropout scales at inference instead), and why the average training output matches the inference output.
How it works
- For each unit draw u from the seeded generator; keep it when u >= rate, otherwise output 0.
- Inverted dropout (Keras): multiply kept units by 1 / (1 - rate). Classic dropout: leave them unchanged.
- At inference nothing is dropped: inverted dropout is a no-op, classic dropout multiplies by (1 - rate).
- Average the training output over more and more masks: it converges to the inference output.
Default run (18 steps): A layer outputs 6 activations [0.8, 1.5, 0.2, 2, 1.1, 0.6]. Dropout rate 0.5, training mode, inverted dropout (as Keras does). … Average over 1000 masks (seeds 1..1000): [0.7936, 1.56, 0.198, 1.96, 1.1242, 0.5928]; largest gap to the inference output [0.8, 1.5, 0.2, 2, 1.1, 0.6] is 0.06. The average training output converges to the inference output: dropout preserves the expected activation.
Simplified: One activation vector with up to 10 units and a seeded teaching PRNG (mulberry32) instead of TensorFlow's generator.
Loading the simulation…
Try it: Training a Keras-Style Model: History and Early Stopping
The Keras workflow on a tiny dense network: stack Dense layers (the summary counts units * (inputs + 1) parameters per layer), compile with an optimizer, loss and accuracy metric, fit for a number of epochs in mini-batches with validation data, and read the history. Training loss keeps falling while validation loss reaches its lowest point and then rises: the model starts to overfit, and EarlyStopping with patience can stop there and restore the best weights.
How it works
- Build a Sequential stack of Dense layers ending in Dense(1, sigmoid); each layer has units * (inputs + 1) weights and biases, initialised Glorot-uniform with zero biases.
- compile(): mini-batch SGD with the chosen learning rate, binary cross-entropy (or mean squared error) loss, and accuracy (prediction > 0.5) as the metric.
- fit(): each epoch walks the training set in batches of batch_size; every batch runs a forward pass, backpropagates the mean batch loss and updates every weight .
- After each epoch the history records the epoch's training loss and accuracy (batch-weighted average of values seen during the epoch) and the val_loss and val_accuracy on the held-out validation points.
- EarlyStopping(monitor="val_loss", patience) keeps the best val_loss; after patience epochs with no improvement it stops training, and restore_best_weights puts back the weights of the best epoch.
- evaluate() reports loss and accuracy on the validation data with the final weights.
Default run (153 steps): Sequential([Dense(16, "relu"), Dense(16, "relu"), Dense(1, "sigmoid")]) on inputs of shape (2,). model.summary(): dense 16 * (2 + 1) = 48; dense_1 16 * (16 + 1) = 272; dense_2 1 * (16 + 1) = 17; total 337 trainable parameters, initialised with Glorot uniform (biases 0). … model.evaluate(x_val, y_val) with the weights from epoch 150: loss 1.2082, accuracy 0.75. Training loss went 0.6964 -> 0.0194. Stopping at epoch 32 would have given val_loss 0.5272 and val_accuracy 0.775.
Simplified: A toy re-implementation in TypeScript, not Keras or TensorFlow: 2 input features, at most 3 hidden layers of up to 32 units, a seeded 2-D dataset (inside/outside a circle with label noise; 40 validation points), plain SGD without momentum, shuffle=False so batch order is fixed, float64 arithmetic, and binary cross-entropy computed from the output logit (same value as on probabilities, without clipping). The results match an independent NumPy implementation of the same procedure, not a real Keras run.
Loading the simulation…
The Training-Set Trap
A model can become better at the examples it has already seen while becoming less dependable on new examples. This happens because fitting familiar data is easier than learning patterns that generalize beyond it. The central challenge is not to make the model learn less carefully. It is to prevent the model from using its flexibility to memorize training-specific details.
Overfitting is reduced by making memorization less easy and making generalizable representations more useful.
The same issue can appear in different tasks, including movie-review prediction, topic classification, and house-price regression. In each case, a model may learn details specific to its training examples that do not provide reliable guidance for new data.
Capacity and Memorization
Model capacity is controlled through the number of learnable parameters. A model with too little capacity may fail to capture relevant patterns and is underfit. As capacity increases, the model can represent more useful structure. If it has too much flexibility, it can also use that flexibility to memorize training examples and training-specific details. The target is a balance between underfitting and overfitting.
The visual shows why increasing capacity is not automatically beneficial. Too little capacity leaves relevant patterns unrepresented. Balanced capacity supports useful representations. Excessive capacity can make unrestricted memorization easier.
Three Training Checkpoints
Following a Model Through Training
A model is observed at three points during training. Determine whether it is underfitting, generalizing usefully, or moving into overfitting.
- 1Beginning
The model has not yet captured all the relevant patterns in its training data. This is the underfitting phase, because useful progress is still available.
- 2Middle
Training performance and performance on unseen data both improve. The model is learning patterns that help beyond the examples used for training.
- 3Later
Training performance continues to improve, but validation performance stalls and then worsens. The model is learning details specific to the training data.
The model moves from underfitting toward useful generalization and then into overfitting.
This progression is important because overfitting is not always visible from training performance alone. At first, training and validation behavior often improve together. Later, optimization can continue while generalization stops improving.
Optimization Versus Generalization
| Measure | What it describes | What continued improvement can mean |
|---|---|---|
| Optimization | Performance on the data used for training | The model is fitting the training examples more closely |
| Generalization | Performance on data the model has not seen | The model is becoming more dependable beyond the training set |
Training performance and unseen-data performance can move apart during overfitting.
Optimization and generalization measure different kinds of model performance. Reducing training loss is evidence that optimization is continuing, but it does not prove that the model is learning more useful behavior for unseen data. Validation performance helps reveal whether continued training is still improving generalization.
Pressure Against Memorization
When more data cannot be obtained, overfitting can be addressed by changing how much information the model can store or by placing constraints on the information it is allowed to store. The main controls in this lesson are model size, weight regularization, and dropout. They work differently, but all make accidental memorization less attractive than learning representations that retain predictive value.
| Strategy | What changes | Shared purpose |
|---|---|---|
| Reduce model size | The number of learnable parameters | Limit memorization resources |
| Weight regularization | The loss receives an additional weight-based cost | Make certain weight choices less attractive |
| Dropout | Some output features are removed during training | Discourage dependence on fragile feature combinations |
| Obtain more training data | The model learns from more examples | Increase the chance of learning generalizable patterns |
These strategies reduce overfitting through different mechanisms.
Weight-Based Costs
L1 and L2 regularization add different weight-based costs to the loss function. The regularizer is attached to a layer's kernel weights, so the layer contributes an additional cost based on those weights. This leaves the model architecture in place while making certain weight choices less attractive during learning.
Dropout Changes by Phase
Dropout changes layer outputs during training by randomly zeroing features. Because a different subset of features can be removed for different training examples, the network is discouraged from depending on a fragile combination of features that only works for particular training cases. This breaks up happenstance patterns that the network might otherwise memorize.
Inference uses the full network differently: units are not dropped, and outputs are scaled rather than being produced by randomly removing features. Therefore, training and inference do not present exactly the same output behavior to the model.
Selecting a Remedy
Start by comparing training and validation behavior. If more training data is possible, it is the strongest general remedy because additional examples make generalizable patterns more likely. If more data is not available, reduce memorization resources or constrain what the model can store. Reducing model size changes the number of parameters. Regularization keeps the architecture but adds a weight-based cost. Dropout changes which output features are available during training.
Common Misreadings
Common mistake Treating lower training loss as proof that generalization is improving.
What it looks likeTraining performance continues to improve while validation performance stalls or worsens.
Why it failsOptimization and generalization measure different kinds of performance.
Do this insteadInspect validation performance alongside training performance.
Common mistake Calling every poorly performing model overfit.
What it looks likeA model has not captured all relevant patterns in its training data.
Why it failsThat describes underfitting, not overfitting.
Do this insteadAsk whether useful progress is still missing or whether training-specific details have begun to dominate.
Common mistake Assuming that reducing overfitting means making the model learn less carefully.
What it looks likeA model is constrained so that memorization becomes less easy.
Why it failsThe goal is to prevent memorization and encourage useful representations, not to abandon careful learning.
Do this insteadDescribe the intervention as changing the pressure on learning.
Common mistake Treating dropout as a permanent reduction in model size.
What it looks likeFeatures are randomly zeroed during training, but inference uses the full network with scaled outputs.
Why it failsDropout changes output information during training rather than permanently removing parameters.
Do this insteadDistinguish training-time feature removal from inference-time scaled full-network outputs.
Common mistake Treating L1 and L2 regularization as the same mechanism.
What it looks likeBoth add weight-based costs, but they are different regularization choices.
Why it failsThe source distinguishes them as different costs added to the loss function.
Do this insteadRemember that both constrain weights through the loss, while L1 and L2 are not identical penalties.
Check Your Reasoning
A topic-classification model continues to improve on its training examples. Its validation performance first improves, then stops improving, and finally declines. Explain what is happening and choose one strategy that could make memorization less attractive.
Hints
- Separate optimization from generalization.
- Identify the phase in which validation performance is strongest.
- Choose from more training data, reducing model size, weight regularization, or dropout, and explain the mechanism of your choice.
What do you think happens?
If training performance keeps improving after validation performance has begun to decline, is optimization still continuing?
Reveal answer
Yes, while generalization is no longer improving
Training performance can continue to improve even after validation performance stalls or worsens. That means optimization is continuing, but the model is moving toward overfitting.
Key Takeaways
- 1Underfitting means that the model has not captured all relevant patterns in its training data.
- 2Overfitting begins when the model learns training-specific details that do not transfer reliably to new data.
- 3Optimization concerns fitting the training data, while generalization concerns performance on unseen data.
- 4Model size, weight regularization, and dropout reduce overfitting through different mechanisms.
- 5Validation performance reveals when continued optimization has stopped improving generalization.
At a glance
- A model can improve on training examples while becoming worse on unseen data.
- Underfitting reflects missing useful patterns; overfitting reflects reliance on training-specific patterns.
- Optimization and generalization must be evaluated separately through training and validation behavior.
- Reducing model size, adding L1 or L2 weight costs, and using dropout constrain memorization in different ways.
- More training data is the strongest general remedy when it can be obtained.