Preventing Overfitting

One approach is to train long enough for overfitting to begin, determine a suitable number of epochs, and then start another training run from scratch.

Programming

Interactive lab

Try it: Training a Keras-Style Model: History and Early Stopping

The Keras workflow on a tiny dense network: stack Dense layers (the summary counts units * (inputs + 1) parameters per layer), compile with an optimizer, loss and accuracy metric, fit for a number of epochs in mini-batches with validation data, and read the history. Training loss keeps falling while validation loss reaches its lowest point and then rises: the model starts to overfit, and EarlyStopping with patience can stop there and restore the best weights.

How it works

Build a Sequential stack of Dense layers ending in Dense(1, sigmoid); each layer has units * (inputs + 1) weights and biases, initialised Glorot-uniform with zero biases.
compile(): mini-batch SGD with the chosen learning rate, binary cross-entropy (or mean squared error) loss, and accuracy (prediction > 0.5) as the metric.
fit(): each epoch walks the training set in batches of batch_size; every batch runs a forward pass, backpropagates the mean batch loss and updates every weight .
After each epoch the history records the epoch's training loss and accuracy (batch-weighted average of values seen during the epoch) and the val_loss and val_accuracy on the held-out validation points.
EarlyStopping(monitor="val_loss", patience) keeps the best val_loss; after patience epochs with no improvement it stops training, and restore_best_weights puts back the weights of the best epoch.
evaluate() reports loss and accuracy on the validation data with the final weights.

Default run (153 steps): Sequential([Dense(16, "relu"), Dense(16, "relu"), Dense(1, "sigmoid")]) on inputs of shape (2,). model.summary(): dense 16 * (2 + 1) = 48; dense_1 16 * (16 + 1) = 272; dense_2 1 * (16 + 1) = 17; total 337 trainable parameters, initialised with Glorot uniform (biases 0). … model.evaluate(x_val, y_val) with the weights from epoch 150: loss 1.2082, accuracy 0.75. Training loss went 0.6964 -> 0.0194. Stopping at epoch 32 would have given val_loss 0.5272 and val_accuracy 0.775.

Simplified: A toy re-implementation in TypeScript, not Keras or TensorFlow: 2 input features, at most 3 hidden layers of up to 32 units, a seeded 2-D dataset (inside/outside a circle with label noise; 40 validation points), plain SGD without momentum, shuffle=False so batch order is fixed, float64 arithmetic, and binary cross-entropy computed from the output logit (same value as on probabilities, without clipping). The results match an independent NumPy implementation of the same procedure, not a real Keras run.

Educational simulation

Loading the simulation…

Training Without Guesswork

When training begins, you usually do not know in advance how many epochs will be needed to reach the best validation loss. One approach is to train long enough for overfitting to begin, determine a suitable number of epochs, and then start another training run from scratch. That approach wastes time. Keras callbacks let the training process observe what is happening and respond during the same run.

When does a callback receive control as training moves through batches, epochs, validation, and completion?

What a Callback Is

Definition

A Keras callback is an object passed to fit that is called at various points during training.

The important idea is timing. A callback does not replace the training process. Instead, it is given opportunities to respond at selected points while training is in progress. Those opportunities can be associated with batches, epochs, validation results, or the end of training. This lets a callback observe training behavior and take an action without requiring a separate trial run.

Metric Monitoring

A training-control callback needs a signal that describes training behavior. A monitored metric supplies that signal. For example, the callback may watch validation loss. After the relevant training progress is reported, the callback compares the current metric behavior with the behavior it has already observed. Its action depends on what the callback is designed to do and on how long the metric has failed to improve.

How does a callback track a monitored metric, count unimproved epochs, and decide when to act?

Patience is the configured period during which the monitored metric may fail to improve before the callback acts. A shorter patience causes action sooner; a longer patience allows more training to continue before action. The exact action depends on the callback. Patience therefore controls timing, not the purpose of the callback.

Worked example

Reading a Patience Setting

A callback monitors validation loss. The loss improves, then fails to improve for the configured patience period. What should you expect the callback to do?

1
Observe
The callback receives training information at a selected training event and examines the monitored validation loss.
2
Compare
It checks whether the current metric has improved compared with the behavior it has already tracked.
3
Count
When the metric does not improve, the callback increases its count of unimproved training periods. When improvement occurs, that count is reset for the purpose of continuing to monitor the metric.
4
Act
When the configured patience period is reached, the callback performs its own designed action: preserving weights, interrupting training, or changing the learning rate.

Result

Patience determines when a callback acts after insufficient improvement; it does not determine which action the callback performs.

Three Built-In Choices

Callback	Primary response	Use it when
ModelCheckpoint	Saves weights during training and can keep only the best model according to a monitored metric.	Preserving the best model weights matters.
EarlyStopping	Interrupts training when a monitored metric stops improving for the configured patience period.	Continuing to train is no longer useful.
ReduceLROnPlateau	Changes the learning rate when validation loss stops improving.	You want to change the learning rate rather than immediately stop.

Choose according to the change you want to make in response to training behavior.

What is the difference between saving a model, stopping training, and reducing the learning rate?

Worked example

Choosing by Desired Change

Training behavior suggests that the best validation result may occur before the final epoch. You want to preserve that result, stop when further training is no longer useful, or give training a chance to improve after validation loss stalls. Which callback matches each goal?

1
Preserve
Choose ModelCheckpoint when the important response is saving weights and keeping only the best model according to a monitored metric.
2
Stop
Choose EarlyStopping when the important response is interrupting training after the monitored metric has stopped improving for the configured patience period.
3
Adjust
Choose ReduceLROnPlateau when validation loss has stopped improving and the desired response is to change the learning rate instead of immediately stopping.

Result

The correct callback is determined by the action you want to take, not merely by the fact that training behavior has changed.

Custom Callback Design

Built-in callbacks do not cover every possible training action. For a behavior that is not provided by a built-in callback, create a custom callback by subclassing keras.callbacks.Callback and implementing methods that run at selected training events.

How do custom callback methods connect training events to specific actions?

A custom callback connects an event to an action that you define. The event might be an epoch start, a batch end, or training end. The callback receives training information at that point and can use its access to model information and training logs as part of the response. The key design choice is selecting the event and then implementing the callback method associated with that event.

What information flows from the training process into a callback, and how can the callback use it?

Common Selection Mistakes

Common mistake
Choosing a callback only because the metric has stopped improving.
What it looks like
Using EarlyStopping when the intended response is to change the learning rate and continue training.
Why it fails
The same training behavior can call for different responses.
Do this instead
First decide whether you need to preserve weights, interrupt training, or change the learning rate. Then choose ModelCheckpoint, EarlyStopping, or ReduceLROnPlateau.
Common mistake
Treating patience as the callback's purpose.
What it looks like
Assuming that any callback with a patience setting will stop training.
Why it fails
Patience controls how long insufficient improvement is tolerated; it does not determine the callback's action.
Do this instead
Separate the timing rule from the action. Identify the callback first, then understand how its patience setting delays that action.
Common mistake
Assuming that a built-in callback can perform every possible training action.
What it looks like
Trying to express a training response that is not covered by the built-in callbacks.
Why it fails
Built-in callbacks do not cover every possible training action.
Do this instead
Create a custom callback by subclassing keras.callbacks.Callback and implementing methods for selected training events.
Common mistake
Ignoring the monitored metric when interpreting callback behavior.
What it looks like
Expecting a callback to act without considering which metric it is tracking.
Why it fails
Callback behavior is tied to the monitored metric and whether that metric improves.
Do this instead
Name the monitored metric and explain how its improvement or lack of improvement leads to the callback's response.

Practice the Decision

MEDIUM

For each goal, choose ModelCheckpoint, EarlyStopping, ReduceLROnPlateau, or a custom callback. Then explain which monitored training behavior and which response led to your choice: preserve the best weights; stop after a monitored metric fails to improve for the configured patience period; change the learning rate when validation loss stops improving; or perform an action not covered by the built-in callbacks.

Hints

Separate the observed training behavior from the desired response.
Patience tells you when a response should happen, not which response the callback performs.
If no built-in callback covers the required action, consider subclassing keras.callbacks.Callback.

Your attempt (not saved)

Key Takeaways

1
A Keras callback is an object passed to fit that receives control at selected points during training.
2
A monitored metric gives a callback the training signal it uses to decide whether behavior has improved.
3
Patience counts how long insufficient improvement is tolerated before the callback acts.
4
Use ModelCheckpoint to preserve weights, EarlyStopping to interrupt unhelpful training, and ReduceLROnPlateau to change the learning rate when validation loss stalls.
5
Subclass keras.callbacks.Callback when the desired training response is not covered by a built-in callback.

At a glance

Callbacks let Keras observe training and respond without requiring a separate trial run.
Metric monitoring and patience determine when a callback should act.
ModelCheckpoint preserves important weights, EarlyStopping ends unhelpful training, and ReduceLROnPlateau changes the learning rate when validation loss stalls.
Custom callbacks connect selected training events to user-defined actions and can use available model information and training logs.

More concepts starting with P