Optimizers and Learning Rate
ReduceLROnPlateau changes the optimizer's learning rate when validation loss has stopped improving, allowing the training process to respond without immediately stopping.
Why Training Needs Oversight
When training begins, you usually cannot know in advance how many epochs will be needed to reach the best validation loss. Training for a fixed, very long time may allow overfitting; training for too few epochs may stop before the model has improved enough. Keras callbacks provide a way to observe training as it happens and respond without requiring a separate trial run from scratch.
A Keras callback is an object passed to fit that is called at various points during training.
Training Event Handoffs
The training loop remains responsible for carrying out training, but control can move to callback behavior at selected points. These points can occur around training as a whole, around an epoch, or around a batch. At such an event, a callback can observe the current training situation and perform the action it was designed for. After the callback response, training can continue unless the callback's purpose is to interrupt it.
The important idea is not that every callback acts at every event. A callback is designed to respond at selected training events, so its behavior depends on both its purpose and the information it monitors.
Three Built-In Responses
| Callback | Main question | Response |
|---|---|---|
| ModelCheckpoint | Should important model weights be preserved? | Saves weights during training and can keep only the best model according to a monitored metric. |
| EarlyStopping | Is continuing to train no longer useful? | Interrupts training when a monitored metric stops improving for the configured patience period. |
| ReduceLROnPlateau | Has validation loss stopped improving, but should training continue more cautiously? | Changes the learning rate when validation loss stops improving. |
Choose according to the training behavior you want to change.
These callbacks do not solve the same problem. ModelCheckpoint protects a useful result by preserving weights. EarlyStopping changes the duration of training by interrupting it. ReduceLROnPlateau changes the optimizer's learning rate when validation loss has stopped improving, allowing the training process to respond without immediately stopping.
Metric Monitoring and Patience
A monitored metric gives a callback a signal about training behavior. The callback compares the current metric behavior with earlier behavior and waits according to its configured patience. If the monitored metric continues to improve, the callback does not treat the situation as a plateau. If improvement stops for the configured patience period, the callback takes its particular action: preserving weights, interrupting training, or changing the learning rate.
A Plateau During Validation
A training run monitors validation loss. The loss improves for several epochs, then stops improving. Which callback response matches each goal?
- 1Preserve the best result
Use ModelCheckpoint so weights can be saved during training and the best model according to the monitored metric can be kept.
- 2Stop when more training is no longer useful
Use EarlyStopping with a configured patience period. If the monitored metric stops improving for that period, training is interrupted.
- 3Continue with a changed learning rate
Use ReduceLROnPlateau when validation loss stops improving and the desired response is to change the learning rate rather than immediately stop.
The same plateau can support different actions. The correct callback depends on whether the goal is preserving weights, ending training, or changing the learning rate.
Learning-Rate Adjustment
ReduceLROnPlateau is specifically intended for the case where validation loss has stopped improving and changing the learning rate is preferable to immediately stopping training. The callback responds to the observed plateau by changing the learning rate, so subsequent training proceeds under the adjusted learning-rate setting.
ReduceLROnPlateau and EarlyStopping can both react to a lack of improvement, but their outcomes differ. ReduceLROnPlateau changes the learning rate; EarlyStopping interrupts training.
Custom Callback Behavior
Built-in callbacks do not cover every possible training action. A custom callback is created by subclassing keras.callbacks.Callback and implementing methods that run at selected training events. This lets the callback define its own response to the information available at those events instead of using only the predefined save, stop, or learning-rate behaviors.
Choose the callback event and response together. First identify when the needed information becomes available; then implement the custom callback method for that event. This keeps the callback focused on one training-time action.
Common Selection Mistakes
Common mistake Using EarlyStopping when the actual goal is only to preserve the best weights.
What it looks likeTraining should continue, but the best model reached so far must be retained.
Why it failsEarlyStopping interrupts training; it does not primarily provide the weight-preservation behavior described for ModelCheckpoint.
Do this insteadUse ModelCheckpoint when preserving weights according to a monitored metric matters.
Common mistake Using ModelCheckpoint when the goal is to end unproductive training.
What it looks likeThe monitored metric has stopped improving and continuing is no longer useful.
Why it failsSaving weights does not by itself interrupt training.
Do this insteadUse EarlyStopping with an appropriate patience period.
Common mistake Treating ReduceLROnPlateau as an immediate stop mechanism.
What it looks likeValidation loss has plateaued, but the intended response is to keep training with a changed learning rate.
Why it failsReduceLROnPlateau changes the learning rate rather than immediately interrupting training.
Do this insteadUse ReduceLROnPlateau when changing the learning rate is the desired response.
Common mistake Ignoring the monitored metric and patience configuration.
What it looks likeExpecting a callback to act as soon as one observed value fails to improve.
Why it failsCallback behavior depends on the metric being monitored and the configured patience period.
Do this insteadDecide which metric represents useful progress and how long to wait before acting.
Common mistake Assuming built-in callbacks cover every training action.
What it looks likeA required response does not match saving, stopping, or changing the learning rate.
Why it failsBuilt-in callbacks do not cover every possible training action.
Do this insteadCreate a custom callback by subclassing keras.callbacks.Callback and implementing methods for selected training events.
Choose the Response
A model's validation loss has stopped improving. You want to keep a record of the best model, stop training if the lack of improvement continues, and change the learning rate if the plateau should be handled without immediately stopping. Match each goal to the most appropriate callback: preserving the best weights, interrupting training, and changing the learning rate.
Hints
- Ask whether the goal is to preserve weights, interrupt training, or adjust the learning rate.
- Patience controls how long a callback waits after the monitored metric stops improving.
Checking Your Selection
Match each training goal with ModelCheckpoint, EarlyStopping, or ReduceLROnPlateau.
- 1Preserve the best weights
The goal is saving weights during training and keeping the best model according to a monitored metric.
- 2Interrupt unproductive training
The goal is stopping when a monitored metric stops improving for the configured patience period.
- 3Continue after a validation-loss plateau
The goal is changing the learning rate when validation loss stops improving rather than immediately stopping.
The matches are ModelCheckpoint for preserving weights, EarlyStopping for interrupting training, and ReduceLROnPlateau for changing the learning rate.
Key Takeaways
- 1A Keras callback is an object passed to fit and invoked at selected points during training.
- 2ModelCheckpoint preserves important weights, EarlyStopping interrupts training, and ReduceLROnPlateau changes the learning rate after validation loss stops improving.
- 3The monitored metric and patience period determine when a callback recognizes insufficient improvement and acts.
- 4Custom callbacks extend training control by subclassing keras.callbacks.Callback and implementing methods for selected training events.
At a glance
- Callbacks let Keras observe training and respond during the run instead of requiring a separate trial run.
- Choose ModelCheckpoint to preserve weights, EarlyStopping to interrupt unproductive training, and ReduceLROnPlateau to change the learning rate after a validation-loss plateau.
- A monitored metric supplies the progress signal, while patience determines how long the callback waits before acting.
- Custom callbacks provide training-event behavior beyond the built-in save, stop, and learning-rate responses.