Concepts / Validation Techniques

Validation Techniques

The central question is not simply whether a model becomes more complex. The important question is whether validation error continues to improve as complexity increases.

  • Programming
Interactive lab

Try it: Polynomial Fit: Overfitting and Ridge

How raising the polynomial degree drives training error down while validation error eventually rises (overfitting), and how a ridge penalty λ‖w‖² tames a high-degree fit.

How it works

  1. Split the noisy points into a training set and a validation set.
  2. Build the design matrix with columns (, a rescaled ).
  3. Solve (XᵀX + λI)w = Xᵀy on the training points by Gaussian elimination with partial pivoting.
  4. Measure the mean squared error on the training points and, separately, on the validation points.
  5. Repeat for every degree 0–9: the degree with the lowest validation error is the one validation selects.

Default run (18 steps): 14 noisy points: 10 for training, 4 held out for validation. Fit a degree-3 polynomial with λ = 0. … Degree 3: training MSE 0.0282, validation MSE 0.0308. The lowest validation error is at degree 4. Training error keeps falling with degree; validation error is what reveals overfitting.

Simplified: One input, 14 fixed noisy samples of sin(2πx) and a single train/validation split (no cross-validation). The penalty λ‖w‖² here includes the constant term and is added to the sum (not the mean) of squared errors.

Educational simulation

Loading the simulation…

Why Complexity Needs Validation

A more complex model does not automatically produce a better model. To study the effect of complexity, compare how training error and validation error change across models with different levels of complexity. This comparison is called a model-selection curve. It helps reveal when increasing complexity is associated with overfitting.

Key point

The central question is not simply whether a model becomes more complex. The important question is whether validation error continues to improve as complexity increases.

Reading the Model-Selection Curve

A model-selection curve compares two error measures across model complexity. Training error measures the model's error as it is fit with increasing complexity, while validation error shows how the model performs on validation data. Reading the two curves together is essential because they follow different trends.

Model complexityincreasesTraining errordecreasesOverfittingvalidation error risesValidation errordecreases, then increaseshigher complexityhigher complexityafter the minimum
How do training error and validation error change together as model complexity increases?

The visual pattern has two parts. Training error decreases as complexity increases. Validation error first decreases, but after complexity continues to increase, validation error starts to increase. That contrast is the key evidence used to diagnose overfitting.

Following Training Error

The training-error trend is monotonic in this comparison: as model complexity increases, training error decreases. A more complex model can therefore look continuously better when judged only by its training error.

Lower complexityhigher training errorMore complexitylower training errorHigher complexitystill lower training errorincrease complexityincrease complexity
What happens to training error as the model becomes more complex?

Finding the Validation Minimum

Validation error behaves differently from training error. As complexity begins to increase, validation error first decreases. If complexity continues to increase, validation error eventually starts to increase. The lowest point in this pattern marks the point before the later worsening begins.

Lower complexityhigher validation errorValidation minimumerror stops decreasingHigher complexityvalidation error increasesincrease complexitycontinue increasing
What happens to validation error as model complexity increases, and where does it reach its minimum?
Worked example

Comparing Three Complexity Levels

A model-selection comparison includes a lower-complexity model, a more complex model, and a still more complex model. The training error decreases at each step. The validation error decreases from the first model to the second, then increases for the third. What does the comparison show?

  1. 1
    Compare training error

    Training error decreases as complexity increases, so the third model appears better if training error is the only measure considered.

  2. 2
    Compare validation error

    Validation error improves from the first model to the second, but worsens for the third model.

  3. 3
    Interpret the contrast

    The later increase in validation error, despite the continuing decrease in training error, is the pattern that signals overfitting.

Result

The comparison identifies the later, more complex model as showing the overfitting pattern. The validation curve must be interpreted together with the training curve.

Recognizing Overfitting

Overfitting is signaled when training error continues to decrease while validation error starts to increase as model complexity increases. The model is improving according to training error but worsening according to validation error. The evidence is the rising validation error, not complexity by itself.

Validationimprovingtraining error decreasesOverfittingvalidation error increasesValidationworseningtraining error stilldecreasescomplexity increasesvalidation error rises
What pattern in the training and validation error curves shows that the model has started fitting noise rather than generalizing?
Key point

The diagnostic sequence is: training error decreases, validation error first decreases, and then validation error increases while training error continues to decrease. That final change in validation error is the signal to identify.

Lower complexityvalidation error decreasesnextValidation minimumlowest point beforeincreaseHigher complexityoverfitting patterncomplexity increasescomplexity continues increasing
How does increasing model complexity move a model from higher error toward a validation minimum and then toward overfitting?

Common Reading Mistakes

  • Common mistake

    Choosing the most complex model because it has the lowest training error.

    What it looks like

    The training-error curve keeps decreasing as complexity increases.

    Why it fails

    Training error alone does not reveal the later increase in validation error.

    Do this instead

    Compare training error and validation error together.

  • Common mistake

    Assuming that a complex model is automatically better.

    What it looks like

    A later increase in complexity is treated as an improvement without checking validation error.

    Why it fails

    A more complex model can be associated with increasing validation error.

    Do this instead

    Look for the point where validation error first decreases and then begins to increase.

  • Common mistake

    Calling a model overfit only because it is complex.

    What it looks like

    The model has high complexity, but no validation trend is considered.

    Why it fails

    The source identifies the increase in validation error as the evidence of overfitting.

    Do this instead

    Diagnose overfitting from the pattern of rising validation error as complexity increases.

  • Common mistake

    Ignoring the validation curve after finding that training error is lower.

    What it looks like

    The later model is preferred because its training error is lower even though its validation error has increased.

    Why it fails

    The two curves must be interpreted together.

    Do this instead

    Use the validation trend to check whether additional complexity is associated with a worsening result.

Practice Diagnosis

EASY

A model-selection curve shows that training error decreases steadily as model complexity increases. Validation error decreases at first, reaches its lowest point, and then increases. State the diagnosis and identify the evidence that supports it.

Hints
  • Compare the direction of the training-error trend with the direction of the validation-error trend after the validation minimum.
  • The diagnosis depends on the increase in validation error, not merely on the model being complex.

What do you think happens?

If training error continues to decrease but validation error begins to increase as complexity rises, what pattern should you identify?

  • The model-selection curve shows overfitting

  • The model is automatically improving

  • Training and validation error are following the same trend

Reveal answer
Answer:

The model-selection curve shows overfitting.

The defining evidence is that validation error increases while training error continues to decrease.

Key Takeaways

  1. 1
    A model-selection curve compares training error and validation error across different levels of model complexity.
  2. 2
    Training error decreases as model complexity increases.
  3. 3
    Validation error first decreases and then increases as complexity continues to increase.
  4. 4
    The increase in validation error is the key signal of overfitting.
  5. 5
    Training and validation error must be interpreted together; training error alone can make increasing complexity look continuously beneficial.

At a glance

  • A model-selection curve compares training and validation error across model complexity.
  • Training error decreases as complexity increases.
  • Validation error first decreases, then increases.
  • Overfitting is indicated when validation error rises while training error continues to fall.

More concepts starting with V