Concepts / Training and validation performance

Training and validation performance

Optimization and generalization measure different kinds of model performance.

  • Programming
Interactive lab

Try it: Training a Keras-Style Model: History and Early Stopping

The Keras workflow on a tiny dense network: stack Dense layers (the summary counts units * (inputs + 1) parameters per layer), compile with an optimizer, loss and accuracy metric, fit for a number of epochs in mini-batches with validation data, and read the history. Training loss keeps falling while validation loss reaches its lowest point and then rises: the model starts to overfit, and EarlyStopping with patience can stop there and restore the best weights.

How it works

  1. Build a Sequential stack of Dense layers ending in Dense(1, sigmoid); each layer has units * (inputs + 1) weights and biases, initialised Glorot-uniform with zero biases.
  2. compile(): mini-batch SGD with the chosen learning rate, binary cross-entropy (or mean squared error) loss, and accuracy (prediction > 0.5) as the metric.
  3. fit(): each epoch walks the training set in batches of batch_size; every batch runs a forward pass, backpropagates the mean batch loss and updates every weight .
  4. After each epoch the history records the epoch's training loss and accuracy (batch-weighted average of values seen during the epoch) and the val_loss and val_accuracy on the held-out validation points.
  5. EarlyStopping(monitor="val_loss", patience) keeps the best val_loss; after patience epochs with no improvement it stops training, and restore_best_weights puts back the weights of the best epoch.
  6. evaluate() reports loss and accuracy on the validation data with the final weights.

Default run (153 steps): Sequential([Dense(16, "relu"), Dense(16, "relu"), Dense(1, "sigmoid")]) on inputs of shape (2,). model.summary(): dense 16 * (2 + 1) = 48; dense_1 16 * (16 + 1) = 272; dense_2 1 * (16 + 1) = 17; total 337 trainable parameters, initialised with Glorot uniform (biases 0). … model.evaluate(x_val, y_val) with the weights from epoch 150: loss 1.2082, accuracy 0.75. Training loss went 0.6964 -> 0.0194. Stopping at epoch 32 would have given val_loss 0.5272 and val_accuracy 0.775.

Simplified: A toy re-implementation in TypeScript, not Keras or TensorFlow: 2 input features, at most 3 hidden layers of up to 32 units, a seeded 2-D dataset (inside/outside a circle with label noise; 40 validation points), plain SGD without momentum, shuffle=False so batch order is fixed, float64 arithmetic, and binary cross-entropy computed from the output logit (same value as on probabilities, without clipping). The results match an independent NumPy implementation of the same procedure, not a real Keras run.

Educational simulation

Loading the simulation…

Why Two Performances Matter

A model can appear to improve while becoming less useful. Its performance on the examples used during training may continue to get better, while its performance on data it has not seen stops improving or begins to decline. To understand whether training is producing reliable behavior, compare training performance with validation performance.

Definition

Optimization is the process of improving a model's performance on its training examples. Generalization is the model's ability to perform reliably on new data. These measure different kinds of performance.

Tracing the Training Stages

Beginningtraining: limited;validation: limitedImprovingtraining: better;validation: betterStrongest validationvalidation reaches its bestpointContinued trainingtraining: better;validation: stallsOverfittingtraining: better;validation: worsetraining continuesvalidation improvestraining continuesgeneralization declines
What changes in training and validation performance as the model moves from underfitting to good fit and then to overfitting?

At the beginning, the model has not captured all the relevant patterns in the training data. This is underfitting. As training proceeds, training performance and validation performance often improve together because the model is learning useful patterns. Later, validation performance can reach its strongest point even though training performance still has room to improve.

Optimization Versus Generalization

Optimizationtraining performanceimprovesTraining dataexamples used duringtrainingGeneralizationnew-data performanceValidation datadata the model has not seenmeasures performance onreveals performance on
How can a model continue improving on the training data while becoming worse on unseen validation data?

Optimization and generalization can agree for a while, but they are not the same goal. Optimization focuses on the data used to train the model. Generalization focuses on whether the learned behavior transfers to data the model has not seen. Early in training, reducing the model's loss on training examples is commonly accompanied by better performance on unseen examples. That agreement may later end.

Worked example

A topic-classification model

A model is trained to classify the topics of text examples. Its training performance keeps improving after validation performance has reached its strongest point.

  1. 1
    Early training

    The model has not yet represented all relevant patterns. Improvements on the training examples are useful when validation performance improves as well.

  2. 2
    Best validation point

    Validation performance reaches its strongest point. The model is showing its most dependable performance on data it has not seen.

  3. 3
    Additional training

    Training performance continues to improve, but validation performance stalls and then worsens. Optimization is continuing, while generalization is no longer improving.

  4. 4
    Interpretation

    The model is moving into overfitting because it is learning details specific to the training data rather than behavior that transfers reliably.

Result

Training performance alone does not show whether the model is becoming more dependable. Validation performance reveals when additional training has stopped helping generalization.

Useful and Training-Specific Patterns

Useful patternhelps training and new dataNew datareliable performanceTraining-specificpatternhelps training onlyTraining dataimproved familiar examplestransfers reliablyimproves performance on
Which patterns learned from the training set also help on new data, and which patterns only improve training performance?

Useful learning captures patterns that improve performance beyond the familiar training examples. Overfitting begins when the model starts learning details that belong specifically to the training data but do not provide reliable guidance for new data. Those details may be misleading or irrelevant outside the training set.

In the real world

The same issue can appear in different tasks, including movie-review prediction, topic classification, and house-price regression. In each case, the central question is whether improved performance on training examples is accompanied by improved performance on unseen examples.

The Underfitting-to-Overfitting Transition

Underfittingrelevant patterns remain tolearnStrong validationgeneralization is strongestValidation stallsadditional training becomesquestionableOverfittingtraining improves;validation worsensuseful patterns are learnedtraining continuesvalidation declines
How does the model's behavior change across training stages, and where does additional training start harming generalization?

Underfitting is not simply poor performance with no explanation. It means that the model has not yet represented all the relevant patterns in the training data, so useful progress is still available. Overfitting is a later problem: the model has begun learning training-specific details that do not transfer reliably.

What do you think happens?

Validation performance has stopped improving, but training performance is still improving. What is the most likely interpretation?

  • The model is still improving its generalization

  • The model is moving into overfitting

  • The model has returned to underfitting

Reveal answer
Answer:

The model is moving into overfitting

The training results show continued optimization, but the validation results show that generalization has stalled or begun to decline.

Reducing Overfitting

Training-specificdetailsmore available to retainTraining-validationgapcan grow as overfittingdevelopsMore training datamore examples availableInformationconstraintsless unrestrictedmemorizationProminent patternsbetter chance ofgeneralizingcan contribute tosupports learning frompushes optimization toward
How do practical strategies change the gap between training and validation performance?

The strongest general remedy for overfitting is to obtain more training data. More examples give the model more information from which to learn and make it more likely to develop patterns that generalize.

When obtaining more data is not possible, use approaches that change how much information the model can store or place constraints on the information it is allowed to store. Limiting what the model can retain makes unrestricted memorization less available. The optimization process is then pushed toward prominent patterns that have a better chance of generalizing.

Best practice

Compare training and validation performance throughout training. Treat continued training as questionable when training performance keeps improving but validation performance stalls or worsens. Prefer more training data when it is available; otherwise, consider approaches that limit what the model can retain.

Mistakes in Reading Performance

  • Common mistake

    Treating better training performance as proof that the model is becoming more useful.

    What it looks like

    Training performance keeps improving after validation performance has begun to decline.

    Why it fails

    Optimization on familiar examples can continue while generalization to unseen data gets worse.

    Do this instead

    Check whether validation performance is improving as well.

  • Common mistake

    Calling every poorly performing model overfit.

    What it looks like

    A model has not yet captured all the relevant patterns in its training data.

    Why it fails

    This describes underfitting: useful progress is still available.

    Do this instead

    Use underfitting when the model has not represented all relevant training patterns, and overfitting when it learns training-specific patterns that fail to transfer.

  • Common mistake

    Assuming training and validation performance must move together forever.

    What it looks like

    Early improvements occur on both measurements, so continued training is assumed to help both.

    Why it fails

    The early agreement between the measurements does not continue indefinitely.

    Do this instead

    Continue comparing the measurements after the early stage because validation performance can stall and then worsen.

  • Common mistake

    Focusing only on memorizing more training information as a solution.

    What it looks like

    The model stores increasingly specific details from the training set.

    Why it fails

    Unrestricted memorization can make training performance look better without improving performance on new data.

    Do this instead

    Obtain more training data when possible, or limit the information the model can retain.

Check Your Understanding

MEDIUM

A model's training performance improves during three successive stages. Validation performance improves during the first two stages, reaches its strongest point at the second stage, and worsens during the third stage. Identify the likely training state at each stage and explain why the third stage should not be judged successful from training performance alone.

Hints
  • At the beginning, ask whether the model has captured all relevant patterns.
  • At the strongest validation point, ask whether generalization is at its best.
  • At the final stage, compare the direction of training performance with the direction of validation performance.
Worked example

Interpreting the three stages

Use the described performance changes to identify the model's behavior.

  1. 1
    Stage one

    The model is likely underfitting because useful patterns remain to be learned.

  2. 2
    Stage two

    The model has reached its strongest validation performance, so generalization is currently strongest.

  3. 3
    Stage three

    The model is moving into overfitting because training performance improves while validation performance worsens.

Result

The model's training history moves from underfitting through its strongest validation point and then toward overfitting.

Key Takeaways

  1. 1
    Optimization measures improvement on training examples, while generalization measures reliable performance on new data.
  2. 2
    Underfitting means the model has not captured all relevant training patterns.
  3. 3
    Overfitting begins when training-specific details improve familiar examples without transferring reliably.
  4. 4
    Training and validation performance often improve together at first, but validation performance can later stall and worsen.
  5. 5
    More training data is the strongest general remedy; when it is unavailable, limit the information the model can retain.

At a glance

  • Optimization and generalization describe different kinds of model performance.
  • A model can move from underfitting to useful learning and then into overfitting as training continues.
  • Validation performance helps identify when additional training has stopped improving generalization.
  • Overfitting occurs when training-specific patterns improve familiar examples but fail to help on new data.
  • More training data or constraints on retained information can reduce overfitting.

Related Topics

Related concepts

More concepts starting with T