Bias-Complexity Tradeoff
The bias-complexity tradeoff is a consideration that arises when parameter settings control the relationship between a model's bias and its complexity.
The Decision Before the Model
When a practical problem arrives, the challenge is not always to write down one algorithm. Several algorithms may be reasonable candidates, and each algorithm may allow several parameter settings. The practical task is deciding which algorithm and which settings fit the problem. That combined decision is called model selection.
Model selection is the task of choosing an algorithm for a particular problem and setting its parameters.
Two Parts of One Choice
Model selection has two connected decisions. First, choose an algorithm that could address the particular problem. Second, decide how to set that algorithm's parameters. A selected model is therefore defined by both the algorithm choice and the parameter choices.
| Decision | Question it answers | Why it belongs to model selection |
|---|---|---|
| Algorithm choice | Which algorithm should be considered for this particular problem? | Several algorithms may be reasonable candidates. |
| Parameter setting | How should the chosen algorithm be configured? | Each algorithm may offer several parameter settings. |
| Combined model choice | Which algorithm and settings fit the problem? | The selected model is defined by both decisions. |
Model selection combines algorithm choice with parameter setting.
Tracing AdaBoost's T
AdaBoost makes the role of parameter setting concrete. Its parameter T controls the bias-complexity tradeoff. Therefore, deciding how to set T is not a minor implementation detail that happens after model selection. It is part of model selection itself.
Treating T as a Model-Selection Decision
A practical problem can be addressed with AdaBoost. What decisions must be included when selecting the model?
- 1Choose the algorithm
AdaBoost is one algorithm being considered for the particular problem.
- 2Choose T
T is a parameter of AdaBoost, so its setting must also be decided.
- 3Consider the tradeoff
Because T controls the bias-complexity tradeoff, the choice of T is connected to the relationship between the model's bias and its complexity.
- 4Classify the decision
The algorithm choice and the T choice together form part of the model-selection problem.
Selecting AdaBoost without deciding how to set T leaves part of the model-selection decision unfinished.
Understanding the Tradeoff
The bias-complexity tradeoff is a consideration that arises when parameter settings control the relationship between a model's bias and its complexity. In the source's concrete example, AdaBoost's T controls this tradeoff. The important point is not that one value of T is always best. The appropriate choice is tied to the particular practical problem.
Avoiding False Shortcuts
Common mistake Treating model selection as only an algorithm choice.
What it looks likeChoosing AdaBoost and stopping without deciding how to set T.
Why it failsModel selection combines algorithm choice with parameter setting.
Do this insteadRecord both the algorithm being considered and the parameter values that configure it.
Common mistake Treating parameters as an afterthought.
What it looks likeCalling the algorithm choice the complete model-selection decision and postponing T as a minor implementation detail.
Why it failsParameters are part of the model-selection decision, not an afterthought.
Do this insteadInclude parameter setting in the model-selection question from the beginning.
Common mistake Claiming that one algorithm or one parameter setting is universally best.
What it looks likeDeclaring a particular algorithm or value of T correct for every practical problem.
Why it failsThe selection question is tied to the particular practical problem at hand.
Do this insteadTreat algorithm and parameter choices as decisions that must fit the problem.
Common mistake Calling the bias-complexity tradeoff an algorithm.
What it looks likeListing the bias-complexity tradeoff as though it were another candidate algorithm beside AdaBoost.
Why it failsThe tradeoff is a consideration related to parameter settings.
Do this insteadUse the tradeoff to describe what parameter selection must consider.
Practice the Decision
A team is considering several algorithms for a practical problem. One candidate is AdaBoost. Explain, in your own words, why the team has not completed model selection merely by naming AdaBoost. Then explain why choosing T should be discussed as part of the same decision.
Hints
- Start with the two decisions that define model selection.
- Connect T to the bias-complexity tradeoff.
- Do not claim that one value of T is universally best.
Checking a Model-Selection Explanation
Which explanation is complete: “Model selection means choosing AdaBoost,” or “Model selection means choosing an algorithm and setting its parameters for a particular problem”?
- 1Inspect the first explanation
It names an algorithm but leaves out parameter setting.
- 2Inspect the second explanation
It includes both algorithm choice and parameter choices.
- 3Apply the AdaBoost example
Because T controls the bias-complexity tradeoff, setting T belongs in the second explanation.
The second explanation is complete.
Key Takeaways
- 1Model selection is choosing an algorithm for a particular problem and setting its parameters.
- 2A selected model is defined by both the algorithm choice and the parameter choices.
- 3AdaBoost's parameter T controls the bias-complexity tradeoff.
- 4Choosing T is part of model selection, not merely a minor implementation detail.
- 5The bias-complexity tradeoff is a model-selection consideration rather than a separate algorithm.
At a glance
- Model selection combines choosing an algorithm with setting its parameters.
- The selected model depends on both decisions.
- AdaBoost's T demonstrates why parameter setting matters: T controls the bias-complexity tradeoff.
- There is no universally best parameter setting stated here; the choice is tied to the practical problem.
- Bias-complexity tradeoff describes a consideration within model selection, not a separate algorithm.