Concepts / Machine learning model evaluation

Machine learning model evaluation

Model evaluation starts with a deliberate division of data into training, validation, and test sets.

  • Programming
Interactive lab

Try it: K-Fold Cross-Validation and Model Selection

How hold-out and k-fold cross-validation choose among candidate models: each fold takes a turn as the validation set, the fold errors are averaged, the lowest average picks the model, and only then is the untouched test set used once — while tuning on the test set would report an optimistic number.

How it works

  1. Set 10 test points aside; they take no part in any choice.
  2. Optionally shuffle the 20 development points (seeded), then cut them into k folds (hold-out: one chunk validates, the rest trains).
  3. For every round, train each candidate polynomial on the other folds and measure its validation MSE on the held-out fold.
  4. Average each candidate's fold errors (with repeated shuffling: average the p run scores; p × k fits per candidate).
  5. Select the candidate with the lowest average, retrain it on all development points, and evaluate it once on the test set.
  6. For contrast, show which candidate the test set itself would have picked — that figure is leaked and optimistic.

Default run (12 steps): 20 development points (sorted by x as they arrived) and 10 test points set aside. Candidates: polynomial degree 0, 1, 2, 3, 4, 5, 6. Plan: 5-fold cross-validation, shuffled with seed 1 → 5 fits per candidate (p × k = 1 × 5), 35 in total. … 5-fold cross-validation picks degree 3 (score 0.0768); its honest test MSE is 0.0889. Picking by test error would report degree 4 at 0.0868 — a leaked, optimistic figure.

Simplified: One fixed toy dataset (20 development + 10 test points from sin(2πx) with noise) and polynomial least squares of degree 0–6 as the candidate models. Shuffling uses a seeded mulberry32 Fisher–Yates; folds are contiguous chunks like scikit-learn's KFold. Hold-out uses the first of k chunks as the validation set.

Educational simulation

Loading the simulation…

Evaluation Starts with Data Roles

Model evaluation begins before you calculate a score. First, decide how the available data will be divided and what role each part will play. The central division is into training, validation, and test sets. Training data supports model training, validation data supports the evaluation process used while working with the model, and test data provides the final evaluation role. Keeping these roles distinct makes the evaluation plan deliberate rather than accidental.

Available dataTraining setmodel trainingValidation setevaluation processTest setfinal evaluationdividedividedivide
Which data records are used for training, model selection, and final unbiased evaluation, and how are these roles kept separate?

A Worked Data Division

Worked example

Assigning roles before scoring

Imagine a dataset that must be organized for a model evaluation process. How can its records be assigned distinct roles?

  1. 1
    Start with the available data

    Treat the complete collection of available records as the source from which the evaluation roles will be created.

  2. 2
    Assign training data

    Place one portion in the training role so it is available for model training.

  3. 3
    Assign validation data

    Place another portion in the validation role for the evaluation process.

  4. 4
    Reserve test data

    Keep a separate portion in the test role for the final evaluation.

Result

The important result is not a particular numeric split. It is a deliberate separation of roles before calculating a score.

The Single Hold-Out Split

Simple hold-out validation is the basic splitting recipe. Instead of creating several fold-based arrangements, it makes one division: one portion is used for training and one portion is held out for validation. This can be enough in some situations. Its defining feature is that the training and validation arrangement is a single split rather than a collection of fold-based arrangements.

Available dataTraining portionHeld-out portionvalidationsingle splitsingle split
How does a single split reserve one portion of the dataset for validation while the rest is used for training?
In the real world

A team might use simple hold-out validation when a single training portion and a single held-out validation portion fit the situation. The method is recognized by its one training-validation split.

K-Fold Rotation

K-fold validation changes the arrangement from one split to multiple folds. The dataset is organized into several folds, and the method forms multiple training and validation arrangements from those folds. In each arrangement, one fold serves as the validation portion while the remaining folds serve as the training portion. The validation role therefore rotates across the folds.

Fold AvalidationFold BvalidationFold CvalidationFold BtrainingFold AtrainingFold CtrainingFold Ctrainingnext arrangementnext arrangement
How does each fold take a turn serving as validation data while the remaining folds are used for training?
Worked example

Reading a three-fold arrangement

Suppose a dataset has three folds named A, B, and C. What does the fold rotation represent?

  1. 1
    First arrangement

    Fold A is the validation portion, while folds B and C form the training portion.

  2. 2
    Second arrangement

    Fold B becomes the validation portion, while folds A and C form the training portion.

  3. 3
    Third arrangement

    Fold C becomes the validation portion, while folds A and B form the training portion.

Result

Each fold takes a turn in the validation role, while the other folds take the training role.

Shuffling and Repeating Folds

Iterated K-fold validation with shuffling extends K-fold validation in two ways. It adds shuffling to the dataset arrangement, and it repeats the K-fold approach. Repetition with different fold assignments creates additional fold-based training and validation arrangements. The defining distinction is therefore not merely that folds are used, but that shuffling and repetition are added to the K-fold process.

Datasetinitial arrangementK-fold arrangementfold assignment 1K-fold arrangementfold assignment 2Shuffled datasetnew arrangementorganize into foldsshuffle and repeatorganize into folds again
What changes when the dataset is shuffled and K-fold validation is repeated with different fold assignments?
RecipeData arrangementDefining featureWhen the source suggests considering it
Simple hold-out validationOne training portion and one held-out validation portionA single splitWhen a single split is enough
K-fold validationSeveral fold-based training and validation arrangementsValidation rotates across foldsWhen a single split is not the preferred arrangement
Iterated K-fold validation with shufflingRepeated fold-based arrangements after shufflingShuffling and repetition extend K-fold validationWhen little data is available and a more advanced splitting method is useful

The table compares the structural distinctions described in the source material.

Choosing with Limited Data

The source material gives a practical starting point: a single split may be enough in some situations, but more advanced splitting methods can be useful when little data is available. Use the method names to describe the arrangement you need. Choose simple hold-out validation when one training-validation split is appropriate. Choose K-fold validation when you want several fold-based training and validation arrangements. Choose iterated K-fold validation with shuffling when you want K-fold validation extended by shuffling and repetition.

Simple hold-outsingle splitLimited dataconsider advanced methodsK-foldmultiple arrangementsIterated K-foldshuffle and repeatmay be enoughcan be usefulcan be useful
How do simple hold-out, K-fold, and iterated K-fold validation differ in their data arrangements and their use when data is limited?

Common Recognition Errors

  • Common mistake

    Treating model evaluation as a scoring step only.

    What it looks like

    A learner begins by calculating a score without first deciding which records belong to training, validation, and test roles.

    Why it fails

    The source defines the division of available data as the starting point of evaluation.

    Do this instead

    Describe the data roles and splitting recipe before discussing the score.

  • Common mistake

    Calling every split K-fold validation.

    What it looks like

    A single training portion and a single held-out validation portion is labeled K-fold validation.

    Why it fails

    Simple hold-out validation is the basic single-split recipe, while K-fold validation uses multiple folds to form several arrangements.

    Do this instead

    Use simple hold-out for one split and K-fold for multiple fold-based arrangements.

  • Common mistake

    Forgetting what iterated K-fold adds.

    What it looks like

    A repeated method is described as ordinary K-fold validation even though shuffling is performed between repetitions.

    Why it fails

    Iterated K-fold validation with shuffling specifically adds shuffling and repetition to K-fold validation.

    Do this instead

    Mention both additions when identifying the iterated method.

  • Common mistake

    Assuming limited data automatically dictates one named recipe.

    What it looks like

    A learner claims that every limited dataset must use iterated K-fold validation.

    Why it fails

    The source says advanced methods can be useful when little data is available, while also noting that a single split may be enough in some situations.

    Do this instead

    Consider whether a single split is enough, then distinguish K-fold from iterated K-fold according to the arrangement required.

Check Your Choice

MEDIUM

A dataset is limited. You first consider one training portion and one held-out validation portion. Then you consider several fold-based arrangements. Finally, you consider shuffling the dataset and repeating the fold-based process. Name the three evaluation recipes in the order described and state the defining change from one recipe to the next.

Hints
  • The first recipe is the basic splitting recipe.
  • The second recipe uses multiple folds.
  • The third recipe adds shuffling and repetition to the second recipe.

What do you think happens?

Which description identifies iterated K-fold validation with shuffling?

  • One training portion and one held-out validation portion

  • Several fold-based training and validation arrangements

  • Shuffling the dataset and repeating the K-fold approach

Reveal answer
Answer:

Shuffling the dataset and repeating the K-fold approach

The source describes iterated K-fold validation with shuffling as K-fold validation extended by shuffling and repetition.

Key Takeaways

  1. 1
    Model evaluation begins by deliberately dividing available data into training, validation, and test roles.
  2. 2
    Simple hold-out validation uses one training-validation split.
  3. 3
    K-fold validation uses multiple folds to form several training and validation arrangements, with the validation role rotating across folds.
  4. 4
    Iterated K-fold validation with shuffling adds shuffling and repetition to the K-fold approach.
  5. 5
    When data is limited, a single split may be enough in some situations, while more advanced splitting methods can also be useful.

At a glance

  • Evaluation starts with a deliberate data division, not with score calculation.
  • Training, validation, and test sets have separate roles in the evaluation process.
  • Simple hold-out uses one split; K-fold uses several fold-based arrangements.
  • Iterated K-fold validation with shuffling adds shuffling and repetition to K-fold validation.
  • For limited data, consider whether a single split is enough or whether an advanced fold-based method is useful.

Related Topics

Related concepts

More concepts starting with M