Model Selection
Model selection is therefore not merely asking which algorithm to use.
Why Validation Is Needed
A learning algorithm produces an output predictor, but producing that predictor is not the end of the task. We also want to know how well the predictor will perform in terms of its true risk. The available evidence may not provide an estimate that is as informative as we would like, so a separate validation procedure can provide additional information.
Validation uses some of the original training data as a validation set. The output predictor is evaluated on that set, and its observed success supplies information for estimating true risk. In this way, validation gives us evidence about the predictor beyond simply knowing that a learning algorithm was run.
Tracing a Validation Estimate
Comparing Two Candidate Predictors
Suppose a practical problem produces two candidate output predictors. How can validation help decide which candidate to prefer?
- 1Produce candidates
Learning procedures produce two candidate output predictors for the same practical problem.
- 2Use the same validation procedure
Evaluate both candidates using the same validation procedure and validation set.
- 3Compare validation success
Compare the observed validation success of the candidates.
- 4Estimate true risk
The validation results provide information for estimating the candidates' true risks.
- 5Prefer the stronger result
The candidate with stronger validation success can be preferred because validation gives a more informative estimate than relying only on a broad, potentially pessimistic bound.
Validation turns observed performance on the validation set into evidence that can support a model choice.
Why Broad Bounds Can Be Limited
One way to reason about a hypothesis class is to use bounds on estimation error. Such bounds state that, for every hypothesis in the class, true risk is not very far from empirical risk. Their limitation is that they must cover all hypotheses and all possible data distributions. Because they provide this broad coverage, the bounds can be loose and pessimistic.
Validation changes the source of the estimate. Instead of relying only on a broad bound, the procedure evaluates the particular output predictor on the validation set. The observed result is then used as information for estimating that predictor's true risk. This more targeted evidence is why validation is useful when selecting among candidate models.
The Two Choices in Model Selection
Model selection is the task of choosing an algorithm for a particular problem and setting its parameters.
The definition contains two connected decisions. First, choose which learning algorithm should be used. Second, decide how that algorithm should be configured by choosing its parameter settings. Model selection is therefore not merely asking which algorithm to use. It also asks which settings fit the practical problem.
| Decision | Question it answers | Place in model selection |
|---|---|---|
| Algorithm choice | Which learning algorithm should address the problem? | One part of selecting a model |
| Parameter choice | How should the chosen algorithm be configured? | The other part of selecting a model |
A selected model is defined by both an algorithm choice and parameter choices.
AdaBoost and Parameter T
AdaBoost illustrates why parameter setting belongs inside model selection. Its parameter T controls the bias-complexity tradeoff. Therefore, deciding how to set T is not a minor step that happens after model selection. It is part of selecting the model itself.
The source does not identify one universally best value for T. The appropriate setting is tied to the practical problem at hand. This is precisely why T belongs in the selection question: different parameter choices represent different candidate configurations whose suitability can be considered using validation.
Balancing Bias and Complexity
Parameter selection can involve a bias-complexity tradeoff. In the source, AdaBoost's T is the concrete parameter associated with this tradeoff. The important practical lesson is not to treat parameter values as an afterthought. A parameter setting helps define the model, so choosing it is part of model selection.
Common Reasoning Mistakes
Common mistake Treating model selection as only the choice of an algorithm.
What it looks likeA learner asks which algorithm to use but ignores the settings available within that algorithm.
Why it failsModel selection includes both choosing an algorithm and setting its parameters.
Do this insteadAsk two questions: which algorithm fits the problem, and how should that algorithm be configured?
Common mistake Treating validation as the same thing as a broad estimation-error bound.
What it looks likeA learner assumes that a bound covering every hypothesis and every possible data distribution is the only evidence available.
Why it failsBroad bounds can be loose and pessimistic because they must provide coverage so widely.
Do this insteadRecognize that validation evaluates the particular output predictor on a validation set and supplies information for a more accurate true-risk estimate.
Common mistake Treating AdaBoost's T as an implementation detail outside model selection.
What it looks likeA learner chooses AdaBoost first and assumes T can be set later without affecting the model-selection decision.
Why it failsT controls the bias-complexity tradeoff, so deciding its value is part of choosing the model.
Do this insteadInclude T among the parameter choices considered during model selection.
Common mistake Claiming that one parameter setting is universally best.
What it looks likeA learner states that the same value of T should always be selected.
Why it failsThe appropriate selection is tied to the practical problem at hand.
Do this insteadUse validation evidence and the problem context when comparing candidate configurations.
Apply the Selection Logic
A practical problem has two candidate predictors. You have set aside some original training data as a validation set. In your own words, explain what the validation results contribute to the decision, and identify the two kinds of choices that together define the selected model.
Hints
- Start with what is evaluated on the validation set.
- Connect the observed validation success to an estimate of true risk.
- Name both the algorithm decision and the parameter decision.
What do you think happens?
A learner says, “I have finished model selection once I choose AdaBoost; choosing T is separate.” Is that statement correct?
Reveal answer
No, because T is a parameter whose setting is part of selecting the model.
AdaBoost's T controls the bias-complexity tradeoff. Since model selection includes setting parameters, deciding T belongs to the model selection problem.
Key Takeaways
- 1Validation sets use some of the original training data to evaluate an output predictor.
- 2Validation success supplies information for estimating true risk and can be more informative than broad, loose, or pessimistic bounds.
- 3Model selection means choosing both a learning algorithm and its parameter settings.
- 4AdaBoost's parameter T belongs to model selection because it controls the bias-complexity tradeoff.
- 5Parameter choices should be considered in relation to the particular practical problem; no universally best setting is given.
At a glance
- Validation evaluates a predictor on some training data set aside as a validation set.
- The validation result provides a more informative estimate of true risk than relying only on broad, potentially pessimistic bounds.
- Model selection includes choosing an algorithm and setting its parameters.
- AdaBoost's T is part of model selection because it controls the bias-complexity tradeoff.
- The best selection depends on the particular practical problem.