Concepts / Model Calibration

Model Calibration

The test set should not be used to guide model calibration.

  • Programming
Interactive lab

Try it: K-Fold Cross-Validation and Model Selection

How hold-out and k-fold cross-validation choose among candidate models: each fold takes a turn as the validation set, the fold errors are averaged, the lowest average picks the model, and only then is the untouched test set used once — while tuning on the test set would report an optimistic number.

How it works

  1. Set 10 test points aside; they take no part in any choice.
  2. Optionally shuffle the 20 development points (seeded), then cut them into k folds (hold-out: one chunk validates, the rest trains).
  3. For every round, train each candidate polynomial on the other folds and measure its validation MSE on the held-out fold.
  4. Average each candidate's fold errors (with repeated shuffling: average the p run scores; p × k fits per candidate).
  5. Select the candidate with the lowest average, retrain it on all development points, and evaluate it once on the test set.
  6. For contrast, show which candidate the test set itself would have picked — that figure is leaked and optimistic.

Default run (12 steps): 20 development points (sorted by x as they arrived) and 10 test points set aside. Candidates: polynomial degree 0, 1, 2, 3, 4, 5, 6. Plan: 5-fold cross-validation, shuffled with seed 1 → 5 fits per candidate (p × k = 1 × 5), 35 in total. … 5-fold cross-validation picks degree 3 (score 0.0768); its honest test MSE is 0.0889. Picking by test error would report degree 4 at 0.0868 — a leaked, optimistic figure.

Simplified: One fixed toy dataset (20 development + 10 test points from sin(2πx) with noise) and polynomial least squares of degree 0–6 as the candidate models. Shuffling uses a seeded mulberry32 Fisher–Yates; folds are contiguous chunks like scikit-learn's KFold. Hold-out uses the first of k chunks as the validation set.

Educational simulation

Loading the simulation…

The Protected Final Check

Model calibration means making choices or adjustments while developing a model. The central rule is that the test set should not guide those choices. If test results influence decisions about the model, the test set is no longer being held aside for an independent final evaluation.

Available dataTraining setBuild modelCalibrated modelValidation setGuide choicesFinal performanceestimateTest setFinal evaluationassignassignhold asidebuildguide choicesevaluate once held asidemeasure
How does using the test set to guide calibration cause information to leak into model decisions and make the final performance estimate unreliable?

Hold-Out Roles

The simplest validation protocol divides the available examples into three roles. Training data is used to build the model. Validation data is used while choosing or adjusting the model. Test data is held aside for the final performance measurement. This three-way separation is needed because a two-way division into training and test data is not enough when decisions are still being made.

Available examplesTraining dataBuild modelModel buildingValidation dataChoose or adjustModel calibrationTest dataFinal measureFinal evaluationdividedividehold asideuseuseuse
How are the available examples divided among training, validation, and test sets, and which set does each stage use?
Worked example

A Three-Way Development Plan

Suppose a learner has a dataset and is still deciding between different model choices. Which role should each portion play?

  1. 1
    Build

    Use the training portion to build the candidate model.

  2. 2
    Choose

    Use the validation portion to compare or adjust candidate choices during development.

  3. 3
    Protect

    Do not use the test portion to guide those choices. Keep it aside for the final evaluation.

Result

The validation portion supports calibration, while the test portion remains an independent final check.

Rotating K-Fold Evaluation

K-fold validation addresses some of the split-dependent variation of a single hold-out split. The data is divided into K equal partitions. One partition takes the validation role while the remaining partitions are used for training. The validation role then rotates so that each partition takes a turn. The resulting scores are averaged.

Partition 1Validation in run 1Remaining partitionsTraining in each runK scoresAveragePartition 2Validation in run 2Partition 3Validation in run 3Partition 4Validation in run 4other partitions trainother partitions trainother partitions trainother partitions trainevaluate each run
How does each partition take a turn as the validation set while the remaining partitions are used for training?
Worked example

Four Partitions, Four Validation Turns

Imagine a K-fold process with four equal partitions.

  1. 1
    Run 1

    Partition 1 is used for validation, and the other three partitions are used for training.

  2. 2
    Run 2

    Partition 2 becomes the validation partition, while the other three are used for training.

  3. 3
    Run 3

    Partition 3 takes the validation role, and the remaining partitions are used for training.

  4. 4
    Run 4

    Partition 4 takes the validation role, and the remaining partitions are used for training.

  5. 5
    Combine

    Average the four resulting validation scores.

Result

Every partition contributes a validation result, rather than relying on only one fixed validation portion.

Repeated Partitioning

When the dataset is especially small and a more precise evaluation is needed, K-fold validation can be repeated. Before each new run, the data is shuffled and divided into K partitions again. Each run therefore produces its own K-fold score. The final score is the average of the scores from all runs.

K-fold run 1First partition arrangementScore 1Average of run 1Shuffle dataRepartitionK-fold run 2New partition arrangementScore 2Average of run 2Final averageAcross runsaverage fold scorescontinuecreate new foldsaverage fold scorescontributecontribute
What changes when the data is reshuffled and repartitioned for another K-fold run, and how do the different splits contribute to the evaluation?

A single K-fold run explores one partition arrangement. Repeating the process exposes the evaluation to additional arrangements. This can help when performance measurements change substantially after different random shuffling and splitting choices.

The Cost of More Evidence

Hold-out splitOne training and validationprocessOne set of runsLower costOne K-fold runK model runsP multiplied by KmodelsHigher costRepeated K-foldP runs, K models eachrunrun K modelstrain and evaluate
How does repeating K-fold validation multiply the number of training and evaluation runs compared with a single hold-out split?
ProtocolPartition behaviorEvaluation resultComputational implication
Simple hold-outOne fixed separation into training, validation, and test rolesOne validation measurement and a final test measurementSimplest protocol
K-fold validationThe validation role rotates across K equal partitionsAverage of the K resulting scoresRequires training and evaluating K models for one K-fold run
Repeated K-fold validationShuffle and repartition before each K-fold runAverage the scores from all runsRepeating the process P times requires training and evaluating P multiplied by K models

The protocols trade simplicity and computational cost against exposure to different partition arrangements.

The benefit of repeated validation has a direct cost. One K-fold run requires training and evaluating models for each of its K partitions. If the process is repeated P times, the total number of trained and evaluated models is P multiplied by K. Repetition can provide more exposure to different splits, but it requires more computation.

Mistakes in Calibration

  • Common mistake

    Using the test score to choose or adjust the model

    What it looks like

    A developer compares several model choices using the test results and keeps the choice with the best test score.

    Why it fails

    The test set has influenced the model decision, so it is no longer an independent final evaluation.

    Do this instead

    Use a separate validation set to guide calibration and reserve the test set for the final measurement.

  • Common mistake

    Assuming a single hold-out split is always reliable

    What it looks like

    A small dataset produces a very different performance measurement after another random shuffle and split.

    Why it fails

    Small validation and test portions may not represent the broader data well.

    Do this instead

    Treat instability across splitting rounds as a warning sign and consider K-fold validation.

  • Common mistake

    Confusing K-fold validation with one fixed validation split

    What it looks like

    Only one partition is ever evaluated while all other partitions are used for training.

    Why it fails

    That does not rotate the evaluation role across K partitions.

    Do this instead

    Let each partition take a turn as validation, then average the resulting scores.

  • Common mistake

    Ignoring the cost of repeated K-fold validation

    What it looks like

    A process is repeated several times without accounting for the additional model training and evaluation.

    Why it fails

    Each repetition performs another K-fold process.

    Do this instead

    Remember that P repetitions with K partitions require training and evaluating P multiplied by K models.

Choose the Validation Protocol

MEDIUM

A dataset is very small, and performance measurements change noticeably after different random shuffling and splitting rounds. The test set has not yet been used. Which validation approach is the strongest candidate, and what trade-off should be explained?

Hints
  • Look for the method designed to respond to split-dependent variation.
  • Consider what happens when the K-fold process is repeated several times.
Worked example

Selecting a More Stable Evaluation Process

A small dataset produces unstable measurements under different hold-out splits. What process should be considered?

  1. 1
    Identify the warning sign

    Large changes between random splitting rounds indicate split-dependent variation.

  2. 2
    Use rotating evaluation

    K-fold validation allows more portions of the data to take part in evaluation by rotating the validation role.

  3. 3
    Consider repetition

    If the dataset is especially small and more precise evaluation is needed, shuffle and repartition before additional K-fold runs.

  4. 4
    State the trade-off

    Additional runs expose evaluation to more partition arrangements but require training and evaluating more models.

Result

K-fold validation, possibly repeated with shuffling, addresses split-dependent variation at increased computational cost.

Key Takeaways

  1. 1
    Do not use test results to guide model calibration; reserve the test set for an independent final evaluation.
  2. 2
    Simple hold-out validation separates training, validation, and test roles.
  3. 3
    K-fold validation rotates the validation role across equal partitions and averages the resulting scores.
  4. 4
    Repeated K-fold validation reshuffles and repartitions the data before each run, then averages the run scores.
  5. 5
    Repeated validation can reduce dependence on one partition arrangement, but it increases computation because P repetitions with K partitions require P multiplied by K model runs.

At a glance

  • The test set must remain separate from calibration decisions.
  • Hold-out validation gives training, validation, and test data distinct roles.
  • K-fold validation rotates evaluation across K partitions and averages the scores.
  • Repeated K-fold validation uses new shuffles and partitions for additional runs.
  • More validation runs provide more split arrangements but increase computational cost.

Related Topics

Related concepts

More concepts starting with M