Concepts / Parameter Tuning

Parameter Tuning

Cross-validation is used to choose the parameter, not to produce the final model that is ready for use.

  • Programming
Interactive lab

Try it: K-Fold Cross-Validation and Model Selection

How hold-out and k-fold cross-validation choose among candidate models: each fold takes a turn as the validation set, the fold errors are averaged, the lowest average picks the model, and only then is the untouched test set used once — while tuning on the test set would report an optimistic number.

How it works

  1. Set 10 test points aside; they take no part in any choice.
  2. Optionally shuffle the 20 development points (seeded), then cut them into k folds (hold-out: one chunk validates, the rest trains).
  3. For every round, train each candidate polynomial on the other folds and measure its validation MSE on the held-out fold.
  4. Average each candidate's fold errors (with repeated shuffling: average the p run scores; p × k fits per candidate).
  5. Select the candidate with the lowest average, retrain it on all development points, and evaluate it once on the test set.
  6. For contrast, show which candidate the test set itself would have picked — that figure is leaked and optimistic.

Default run (12 steps): 20 development points (sorted by x as they arrived) and 10 test points set aside. Candidates: polynomial degree 0, 1, 2, 3, 4, 5, 6. Plan: 5-fold cross-validation, shuffled with seed 1 → 5 fits per candidate (p × k = 1 × 5), 35 in total. … 5-fold cross-validation picks degree 3 (score 0.0768); its honest test MSE is 0.0889. Picking by test error would report degree 4 at 0.0868 — a leaked, optimistic figure.

Simplified: One fixed toy dataset (20 development + 10 test points from sin(2πx) with noise) and polynomial least squares of degree 0–6 as the candidate models. Shuffling uses a seeded mulberry32 Fisher–Yates; folds are contiguous chunks like scikit-learn's KFold. Hold-out uses the first of k chunks as the validation set.

Educational simulation

Loading the simulation…

The Scarce-Data Problem

A validation procedure usually needs some data to be held aside so that a model can be checked. When the available data is scarce, permanently reserving a separate validation set means that those examples cannot contribute to training. k-Fold Cross Validation addresses this problem by repeatedly using different parts of the original training set for validation while using the remaining parts for training.

Permanentvalidation setheld asideRotating validationfoldsdifferent examples eachfoldTraining setremaining examplesRotating trainingfoldsremaining examples eachfoldreplacereuse examples
How does k-Fold Cross Validation let the same scarce examples contribute to both model training and error estimation?

One Complete Cross-Validation Pass

Choose a value for k and divide the original training set into k folds. The procedure then repeats k times. During one repetition, one fold serves as the error-estimation fold, while all the other folds are used for training. On the next repetition, a different fold serves as the error-estimation fold. Each fold serves as the error-estimation fold exactly once.

Fold 1train on 2,3,...,k;estimate on 1Fold 2train on 1,3,...,k;estimate on 2Fold ktrain on 1,2,...,k-1;estimate on krotate held-out foldcontinue
How do the training and validation sets change across each fold, and how is every example used for both training and error estimation?
Key point

The important state change from one fold to the next is not a change to the original data. It is a change in which fold is used to estimate error and which folds are used to train. Across the complete pass, every example belongs to a training set during some folds and to an error-estimation fold once.

From Fold Errors to True Error

Each fold produces an error measurement for a model trained on the other folds. The average of the k fold errors is the cross-validation estimate of the model's true error. This estimate uses the repeated error measurements from the different held-out parts instead of relying on one permanently unused validation set.

Fold 1 errorheld-out fold 1Average fold errorcross-validation estimateTrue errorestimatedFold 2 errorheld-out fold 2Fold k errorheld-out fold kcombinecombinecombineestimate
How do the individual validation errors from the folds combine into an average that estimates the model's error on unseen data?
Worked example

Comparing Two Parameter Values

A learner evaluates two possible parameter values using three folds. Parameter A produces fold errors of 2, 4, and 3. Parameter B produces fold errors of 5, 2, and 4. Which parameter has the lower cross-validation error?

  1. 1
    Collect the fold errors

    For each parameter, retain the error produced when each of the three folds serves as the error-estimation fold.

  2. 2
    Average parameter A

    The three errors for parameter A average to 3.

  3. 3
    Average parameter B

    The three errors for parameter B average to 11 divided by 3, which is approximately 3.67.

  4. 4
    Compare the estimates

    The lower average is the lower estimated true error for this comparison.

Result

Parameter A is selected because its cross-validation error is lower than parameter B's.

Selecting a Parameter

k-Fold Cross Validation can be used for model selection, also called parameter tuning. Give the procedure a training set, a set of possible parameter values, a choice of k, and a learning algorithm that accepts both a training set and a parameter. Run cross-validation for each possible parameter value, compare the resulting cross-validation errors, and choose the parameter with the lowest estimated error.

Parameter Aaverage error: 3Parameter AselectedParameter Baverage error: 3.67lower estimatecompare
How are errors for different parameter values compared across folds to choose the parameter with the lowest estimated error?
Candidate parameterFold 1 errorFold 2 errorFold 3 errorCross-validation error
Parameter A2433
Parameter B524Approximately 3.67

Illustrative comparison of candidate parameter values using three folds.

Retraining After Selection

Cross-validation is used to choose the parameter, not to produce the final model that is ready for use. After the best parameter has been chosen, retrain the learning algorithm with that parameter on the entire original training set. This final training step allows the selected model to use all of the original training examples.

Training set andcandidatespossible parameter valuesk-Fold CrossValidationestimate each candidateBest parameterlowest estimated errorRetrain algorithmentire original trainingsetFinal modelselected parameterevaluatechoose lowest estimatepass parameterproduce
What happens to the selected parameter after cross-validation, and how does the final model get retrained on the entire training set?

Leave-One-Out Cross Validation

Definition

Leave-One-Out Cross Validation is the special case of k-Fold Cross Validation in which k equals m, where m is the number of examples. In this arrangement, the fold structure makes each example the single error-estimation case in turn, while the other examples are used for training.

Example 1validation caseRemaining examplestraining casesExample 2validation caseExample mvalidation caseall other examples trainall other examples trainall other examples train
What changes in the fold structure when k equals the number of examples, and how does each example become the single validation case in turn?
Key point

The defining feature is the value of k: it equals the number of examples. The validation fold therefore contains one example at a time, and the held-out example changes until every example has served as the error-estimation case.

Common Tuning Mistakes

  • Common mistake

    Keeping one permanently unused validation set when the training data is scarce.

    What it looks like

    A learner removes a separate validation portion and never allows those examples to contribute to training.

    Why it fails

    The held-aside examples cannot help train the model, even though the available data is limited.

    Do this instead

    Use k-Fold Cross Validation so different parts of the original training set take turns serving as the error-estimation fold.

  • Common mistake

    Using only one fold as the validation fold.

    What it looks like

    A learner trains on most folds, estimates error on one fold, and stops.

    Why it fails

    The procedure requires each fold to serve as the error-estimation fold exactly once.

    Do this instead

    Complete all k repetitions and combine the k fold errors.

  • Common mistake

    Choosing a parameter from one fold error instead of the cross-validation error.

    What it looks like

    A candidate parameter looks best on the first held-out fold, so it is selected immediately.

    Why it fails

    A single fold does not provide the full cross-validation estimate described by the procedure.

    Do this instead

    Average the errors from all folds for each candidate parameter, then compare those averages.

  • Common mistake

    Failing to retrain after selecting the parameter.

    What it looks like

    A learner keeps one of the fold-trained models as the final model.

    Why it fails

    The stated model-selection procedure ends with retraining on the entire original training set.

    Do this instead

    Use the selected parameter with the learning algorithm on all original training examples.

Practice the Procedure

MEDIUM

A training set is divided into four folds. A candidate parameter produces four fold errors: 6, 3, 5, and 2. Describe what each error represents, explain what should be done with the four errors, and state what happens after this candidate is compared with the other possible parameter values.

Hints
  • Identify which fold is used for error estimation during each repetition.
  • The other folds are used for training during that repetition.
  • The four errors are combined into the cross-validation estimate for this candidate.
  • After comparison, retrain with the selected parameter on the entire original training set.

What do you think happens?

Suppose k equals the number of examples. What does each error-estimation fold contain?

  • All examples

  • Half of the examples

  • Exactly one example

  • No examples

Reveal answer
Answer:

Exactly one example

When k equals m, the procedure is Leave-One-Out Cross Validation. Each example serves as the single error-estimation case in turn, while the other examples are used for training.

Key Takeaways

  1. 1
    k-Fold Cross Validation estimates true error without setting aside a permanently unused validation set.
  2. 2
    Each fold is used for error estimation exactly once, while the other folds are used for training.
  3. 3
    The average of the fold errors is the cross-validation estimate of true error.
  4. 4
    Leave-One-Out Cross Validation is the special case where k equals the number of examples.
  5. 5
    For parameter tuning, compare candidate parameters by their cross-validation errors, select the lowest estimate, and retrain on the entire original training set.

At a glance

  • k-Fold Cross Validation is useful when data is scarce because examples rotate between training and error estimation rather than remaining permanently unused.
  • Across k folds, each fold serves as the error-estimation fold once and the remaining folds serve as training data.
  • The average fold error estimates the model's true error.
  • Leave-One-Out Cross Validation occurs when k equals the number of examples.
  • Parameter tuning uses these estimates to select a parameter before retraining on the entire original training set.

Related Topics

Related concepts

More concepts starting with P