Concepts / Training, Validation, and Test Sets

Training, Validation, and Test Sets

The test set should not be used to guide model calibration.

  • Programming
Interactive lab

Try it: K-Fold Cross-Validation and Model Selection

How hold-out and k-fold cross-validation choose among candidate models: each fold takes a turn as the validation set, the fold errors are averaged, the lowest average picks the model, and only then is the untouched test set used once — while tuning on the test set would report an optimistic number.

How it works

  1. Set 10 test points aside; they take no part in any choice.
  2. Optionally shuffle the 20 development points (seeded), then cut them into k folds (hold-out: one chunk validates, the rest trains).
  3. For every round, train each candidate polynomial on the other folds and measure its validation MSE on the held-out fold.
  4. Average each candidate's fold errors (with repeated shuffling: average the p run scores; p × k fits per candidate).
  5. Select the candidate with the lowest average, retrain it on all development points, and evaluate it once on the test set.
  6. For contrast, show which candidate the test set itself would have picked — that figure is leaked and optimistic.

Default run (12 steps): 20 development points (sorted by x as they arrived) and 10 test points set aside. Candidates: polynomial degree 0, 1, 2, 3, 4, 5, 6. Plan: 5-fold cross-validation, shuffled with seed 1 → 5 fits per candidate (p × k = 1 × 5), 35 in total. … 5-fold cross-validation picks degree 3 (score 0.0768); its honest test MSE is 0.0889. Picking by test error would report degree 4 at 0.0868 — a leaked, optimistic figure.

Simplified: One fixed toy dataset (20 development + 10 test points from sin(2πx) with noise) and polynomial least squares of degree 0–6 as the candidate models. Shuffling uses a seeded mulberry32 Fisher–Yates; folds are contiguous chunks like scikit-learn's KFold. Hold-out uses the first of k chunks as the validation set.

Educational simulation

Loading the simulation…

Why Three Roles Matter

Model development is not only a train-then-test action. While developing a model, you make configuration choices and need feedback about those choices. Training data helps build the model, validation data helps guide development decisions, and test data provides a final evaluation after the model is ready.

The central rule is simple: use validation performance to guide configuration choices, but keep the test set out of that process. If test results influence model choices, the test set is no longer an independent final evaluation.

Training setBuild the modelValidation setGuide configurationTest setFinal evaluation
What is each dataset allowed to influence during model development?

The Hold-Out Workflow

Simple hold-out validation assigns separate training, validation, and test roles. The training portion is used to build the model. The validation portion supplies feedback while you choose or adjust the model. The test portion remains aside until the model is ready for its final evaluation.

Available dataExamplesTraining setBuild modelValidation setChoose configurationTest setUse once at the endassignedassignedheld aside
How are examples divided, and which set is used at each stage?
Worked example

Choosing Between Two Configurations

A learner develops two model configurations and wants to decide which one to keep without compromising the final evaluation.

  1. 1
    Build

    Use the training set to build each configuration.

  2. 2
    Compare

    Use validation performance to compare the configurations and choose one.

  3. 3
    Protect

    Do not use the test results to decide between the configurations.

  4. 4
    Evaluate

    After the configuration is chosen, use the untouched test set once to estimate performance on new data.

Result

Validation guides the choice; the test set supplies the independent final evaluation.

Rotating K-Fold Evaluation

K-fold validation addresses concern about split-dependent variation. The data is divided into K equal partitions. During each round, one partition takes the evaluation role while the remaining partitions are used for training. The evaluation role rotates until every partition has taken a turn. The resulting scores are then averaged.

Round 1P1 validationAverage scoreK resultsRound 2P2 validationRound 3P3 validationRound KPK validationrotaterotatecontinueaverage scores
How does each partition take a turn as the validation set while the remaining partitions are used for training?
In the real world

Suppose the data is divided into four partitions. In the first round, partition 1 is evaluated and partitions 2, 3, and 4 are used for training. In the next round, partition 2 is evaluated and the other three partitions are used for training. The process continues until all four partitions have taken the evaluation role, after which the four scores are averaged.

Repeated Shuffling

When the dataset is especially small and a more precise evaluation is needed, K-fold validation can be repeated. Before each run, the data is shuffled and divided into K partitions again. Each run produces its own K-fold score, and the final score is the average of the scores from all runs.

Run 1Shuffle, K foldsFinal scoreAverage of P scoresRun 2Shuffle, K foldsRun PShuffle, K foldsK-fold scoreK-fold scoreK-fold score
What changes when the data is reshuffled and repartitioned for multiple K-fold runs?
Rule

Worked example

Counting Repeated K-Fold Work

A validation procedure uses K folds and repeats the entire shuffled process P times.

  1. 1
    Count folds in one run

    One K-fold run trains and evaluates K models because each partition takes the evaluation role once.

  2. 2
    Count repeated runs

    Repeating the procedure P times creates P separate K-fold runs.

  3. 3
    Combine the counts

    The total number of trained and evaluated models is P multiplied by K.

Result

Repeated shuffling gives evaluation exposure to more partition arrangements, but it increases computational cost.

Tuning Without Leaking the Test

Hyperparameters are configuration choices for a model. The source gives the number of layers and the size of layers as examples. They differ from model parameters such as network weights. When validation performance determines which configuration to keep, validation acts as a feedback signal in the search for a good configuration.

Training setBuild modelConfigurationKeep selected modelValidation setGuide choicesFinal evaluationNew-data estimateTest setUntouchedbuildselectevaluatefinal use
How does information flow during model selection while the test set stays isolated until the final evaluation?
Candidate modelsInitial choicesValidation setRepeated feedbackSelectedconfigurationRepeated validation choicescomparechoose best repeatedly
How can repeatedly choosing the best validation score make development fit the validation set indirectly?

Repeated validation-based tuning can make the model fit the validation set indirectly. Each choice uses validation feedback, so the validation result is no longer a completely neutral observation of development. This is why the test set must remain untouched until the model is ready.

Mistakes in Evaluation Design

  • Common mistake

    Using only training data and test data while choosing the model.

    What it looks like

    A developer compares configurations by looking at their test results and keeps the one with the better result.

    Why it fails

    The test results have influenced model development, so the test set is no longer reserved for an independent final evaluation.

    Do this instead

    Use a separate validation set to guide configuration choices and keep the test set untouched.

  • Common mistake

    Treating one hold-out split as unquestionably reliable on a small dataset.

    What it looks like

    A single random split produces one performance number, and the developer assumes it represents the broader data.

    Why it fails

    Small validation and test portions may contain too few samples, and different shuffling and splitting rounds may produce very different measurements.

    Do this instead

    Consider K-fold validation when split-dependent variation is a concern.

  • Common mistake

    Ignoring the cost of repeated K-fold validation.

    What it looks like

    A developer repeats K-fold validation several times without accounting for the additional model training and evaluation.

    Why it fails

    Each repeated run contains K folds, so repeating the process requires training and evaluating P multiplied by K models.

    Do this instead

    Use repeated K-fold validation when the added evaluation exposure justifies its computational cost.

  • Common mistake

    Treating validation performance as the final estimate of performance on new data.

    What it looks like

    The configuration selected through many validation comparisons is reported using its validation result alone.

    Why it fails

    Repeated validation-based tuning can make the model fit the validation set indirectly.

    Do this instead

    After the model is ready, use the untouched test set once for final evaluation.

Apply the Workflow

MEDIUM

A team has a small dataset. It wants to compare several configurations, is concerned that one random split may be unstable, and wants a final performance estimate that did not influence development. Describe a suitable evaluation workflow. State which data guides configuration choices, when K-fold validation could be used, and when the test set is evaluated.

Hints
  • Separate development feedback from the final evaluation.
  • K-fold validation rotates the evaluation role across partitions.
  • If K-fold validation is repeated P times with K folds, count P multiplied by K model evaluations.

What do you think happens?

A developer checks the test score after every configuration change and keeps the configuration with the best test score. Is the final test score still an independent final evaluation?

  • Yes, because the test set was not used to build the model weights

  • No, because test results guided configuration choices

  • Yes, because the best score is the most informative score

Reveal answer
Answer:

No, because test results guided configuration choices.

The test set must remain outside development. Once its results influence calibration or configuration choices, it no longer provides an untouched final evaluation.

Key Takeaways

  1. 1
    Training data builds the model, validation data guides development choices, and test data supplies the final evaluation.
  2. 2
    Hold-out validation keeps training, validation, and test roles separate.
  3. 3
    K-fold validation rotates the evaluation role across K partitions and averages the resulting scores.
  4. 4
    Repeated K-fold validation reshuffles and repartitions the data for multiple runs, requiring P multiplied by K model evaluations when repeated P times.
  5. 5
    Repeated tuning can fit the validation set indirectly, so the untouched test set must be used only after the model is ready.

At a glance

  • Use three dataset roles because building a model, guiding its development, and estimating final performance are different tasks.
  • Use validation performance to choose configurations, not test performance.
  • K-fold validation reduces dependence on one particular split by rotating evaluation across partitions.
  • Repeated shuffled K-fold validation provides more partition arrangements but increases computational cost.
  • Keep the test set untouched until the final evaluation because repeated validation-based tuning can overfit the validation set.

Related Topics

Related concepts

More concepts starting with T