Training and Validation Sets
The central workflow is: use the training set to create candidate predictors, use a fresh validation set to compare their errors, and select the candidate with the smallest validation error.
Try it: Polynomial Fit: Overfitting and Ridge
How raising the polynomial degree drives training error down while validation error eventually rises (overfitting), and how a ridge penalty λ‖w‖² tames a high-degree fit.
How it works
- Split the noisy points into a training set and a validation set.
- Build the design matrix with columns (, a rescaled ).
- Solve (XᵀX + λI)w = Xᵀy on the training points by Gaussian elimination with partial pivoting.
- Measure the mean squared error on the training points and, separately, on the validation points.
- Repeat for every degree 0–9: the degree with the lowest validation error is the one validation selects.
Default run (18 steps): 14 noisy points: 10 for training, 4 held out for validation. Fit a degree-3 polynomial with λ = 0. … Degree 3: training MSE 0.0282, validation MSE 0.0308. The lowest validation error is at degree 4. Training error keeps falling with degree; validation error is what reveals overfitting.
Simplified: One input, 14 fixed noisy samples of sin(2πx) and a single train/validation split (no cross-validation). The penalty λ‖w‖² here includes the constant term and is added to the sum (not the mean) of squared errors.
Loading the simulation…
The Selection Problem
Training does not always produce one unavoidable model. Starting with the same training set, you may use different algorithms or use one algorithm with different parameter settings. These choices can produce several candidate predictors. Model selection begins after those candidates exist: the task is to choose one of them. A fresh validation set provides a separate basis for making that choice.
The central workflow is: use the training set to create candidate predictors, use a fresh validation set to compare their errors, and select the candidate with the smallest validation error.
One Training Set, Several Candidates
The training set is used to produce a collection of candidate predictors. The candidates can arise because different learning algorithms are tried, or because one algorithm is run with different parameter choices. In either case, the common training set is the input to the procedures that produce the candidates. The result is not yet the final selected predictor; it is a collection from which one predictor can later be chosen.
Creating a Candidate Collection
Suppose the same training set is supplied to two different algorithms and to one algorithm with a different parameter setting. What exists before validation is used?
- 1Use the common training set
The training set is supplied to each learning procedure or configuration.
- 2Produce candidates
The procedures produce Predictor A, Predictor B, and Predictor C.
- 3Prepare for selection
The candidate collection now exists, but no single candidate has been selected yet.
Model selection starts with several already-produced candidate predictors.
Keeping Validation Separate
The candidate collection depends on the training set because the training set is used to create the candidates. The validation set is fresh and independent of that collection. This separation gives model selection a separate basis for comparing the candidates rather than using the same data that produced the collection.
Comparing Validation Errors
Once the candidate collection has been formed, each candidate predictor is evaluated on the fresh validation set. The resulting validation errors are compared. Empirical risk minimization over the validation set means selecting the candidate with the smallest validation error. In this context, ERM is therefore a model-selection rule: it chooses one predictor from the candidates rather than creating the original candidate collection.
Selecting by Smallest Validation Error
Three candidate predictors are evaluated on the same fresh validation set. Predictor A has validation error 8, Predictor B has validation error 5, and Predictor C has validation error 11. Which predictor does validation-based ERM select?
- 1Evaluate every candidate
The validation set is used to obtain an error for Predictor A, Predictor B, and Predictor C.
- 2Compare the errors
The values are 8, 5, and 11. The smallest value is 5.
- 3Apply the selection rule
ERM over the validation set retains the candidate with the smallest validation error.
Predictor B is selected because its validation error is the smallest of the three.
Training Versus Validation
| Training set | Validation set |
|---|---|
| Used to create the candidate predictors | Used to compare already-produced candidates |
| The candidate collection depends on it | The fresh set is independent of the candidate collection |
| Feeds different algorithms or parameter settings | Provides errors used for selection |
The two sets have different roles in validation-based model selection.
Common mistake Treating training as if it must produce only one model
What it looks likeAssuming that one training set leads to one unavoidable predictor
Why it failsDifferent algorithms or different parameter settings can produce several candidate predictors from the same training set.
Do this insteadRecognize that model selection begins after a candidate collection has been produced.
Common mistake Using the validation set to describe how the candidates were created
What it looks likeTreating validation as another input to the original learning procedures
Why it failsThe validation set is used after the candidates exist, to provide a separate basis for comparing them.
Do this insteadUse the training set to create candidates, then use a fresh validation set to evaluate and select among them.
Common mistake Selecting a candidate without comparing validation errors
What it looks likeChoosing a candidate before evaluating every candidate on the validation set
Why it failsValidation-based ERM selects the candidate with the smallest validation error.
Do this insteadEvaluate every candidate, compare the resulting validation errors, and retain the candidate with the smallest one.
Common mistake Confusing the candidate collection with the selected predictor
What it looks likeCalling the entire collection the final model
Why it failsThe collection contains several candidates; model selection retains one of them.
Do this insteadKeep the stages separate: collection first, selection second.
Practice the Workflow
A training set is used with three learning procedures, producing three candidate predictors. A fresh validation set is then used to evaluate all three. Candidate X has the smallest validation error, Candidate Y has a larger validation error, and Candidate Z has the largest validation error. Describe the complete model-selection process and identify which candidate is retained.
Hints
- Start with the role of the training set.
- Then explain what the validation set measures.
- Apply the smallest-validation-error rule.
Practice Answer
Explain the workflow for Candidates X, Y, and Z when Candidate X has the smallest validation error.
- 1Create the collection
Use the training set with the learning procedures to produce Candidates X, Y, and Z.
- 2Evaluate independently
Use the fresh validation set to evaluate all three already-produced candidates.
- 3Select by ERM
Compare the validation errors and choose the candidate with the smallest one.
Candidate X is retained because validation-based ERM selects the candidate with the smallest validation error.
Key Takeaways
- 1One common training set can produce several candidate predictors through different algorithms or parameter settings.
- 2The validation set is used after the candidate collection has been formed.
- 3Validation-based empirical risk minimization selects the candidate with the smallest validation error.
- 4The candidate collection depends on the training set, while the fresh validation set is independent of that collection.
- 5Model selection is the process of choosing one predictor from the already-produced candidates.
At a glance
- Training can produce several candidate predictors from one common training set.
- A fresh validation set compares the candidates after they have been produced.
- ERM over the validation set means selecting the candidate with the smallest validation error.
- The validation set is independent of the hypothesis collection produced from the training set.