Decision Stumps
A decision stump is a simple threshold-based classifier.
From One Cut to Three Regions
A decision stump is a simple threshold-based classifier. To understand its role here, compare it with a more structured class called 3-piece classifiers. A 3-piece classifier operates on the real line and uses two ordered thresholds, theta_1 and theta_2, satisfying theta_1 < theta_2, together with a sign parameter b chosen from {+1, -1}. The two thresholds divide the real line into three intervals. A decision stump uses only one threshold theta and a sign parameter b, so it has a simpler structure.
A 3-Piece Classifier Step by Step
Reading a 3-Piece Specification
Suppose a classifier is specified by two real thresholds with theta_1 < theta_2 and by a sign parameter b in {+1, -1}. What structural information does this specification provide?
- 1Locate the first boundary
The first threshold, theta_1, marks a boundary on the real line.
- 2Locate the second boundary
The second threshold, theta_2, lies to the right of theta_1 because theta_1 < theta_2.
- 3Identify the regions
The two ordered thresholds divide the real line into three intervals.
- 4Account for the sign parameter
The parameter b is part of the classifier specification and can take one of two values: +1 or -1.
The classifier is described by two ordered cut points, three resulting intervals, and a binary sign parameter. The source specification identifies these structural ingredients; the exact label assignment rule for each interval is not specified here.
How a Decision Stump Uses One Threshold
A decision stump uses one threshold, theta, and a sign parameter b. The single threshold divides the real line into two regions: inputs on one side of theta and inputs on the other side. The stump then applies its binary classification rule to those regions. Compared with a 3-piece classifier, the stump has one fewer threshold and one fewer interval.
What do you think happens?
A classifier changes from two thresholds to one threshold. What structural change should you expect?
Reveal answer
The real line is divided into fewer regions.
Two ordered thresholds divide the real line into three intervals, whereas one threshold gives the simpler two-region structure of a decision stump.
Comparing the Two Classifier Families
| Property | 3-piece classifier | Decision stump |
|---|---|---|
| Operating domain | The real line | The real line |
| Number of thresholds | Two ordered thresholds, theta_1 and theta_2 | One threshold, theta |
| Ordering requirement | theta_1 < theta_2 | Only one threshold is used |
| Number of intervals | Three | Two regions |
| Sign parameter | b in {+1, -1} | A sign parameter b |
| Relative structure | More structured classifier family | Simpler classifier family |
The structural distinction is the number of thresholds and the resulting number of regions.
Why a Simpler Learner Can Still Help
The class of decision stumps is called B in this example. The class of 3-piece classifiers is the more structured class H. Because a stump uses only one threshold, it cannot express the full two-threshold structure of a 3-piece classifier. However, the learning question is not whether a stump has exactly the same structure as a 3-piece classifier. The question is whether the simpler stump family can still learn something useful about the more structured family. In this example, decision stumps are used as a weak learner for 3-piece classifiers.
Interpreting the 1/12 Guarantee
The statement that ERM_B is a gamma-weak learner for H with gamma equal to 1/12 identifies ERM_B as the learning procedure associated with the decision-stump class B and states the specified weak-learning parameter for the target class H. In this example, the value of that parameter is 1/12.
The important interpretation is comparative: ERM_B is being evaluated as a learner for H, even though B contains the simpler decision-stump structure. The gamma value records the strength of the weak-learning guarantee in this example. The source statement supplies gamma = 1/12, but it does not provide the detailed error inequality or derivation behind that guarantee, so this article treats 1/12 as the stated parameter rather than deriving a numerical bound.
Mistakes to Avoid
Common mistake Treating a 3-piece classifier as if it used one threshold.
What it looks likeDescribing the 3-piece class using only theta.
Why it failsA 3-piece classifier is specified using two ordered thresholds, theta_1 and theta_2, with theta_1 < theta_2.
Do this insteadTrack both thresholds and remember that they create three intervals.
Common mistake Assuming that a decision stump and a 3-piece classifier have the same expressive structure.
What it looks likeCalling a stump a 3-piece classifier because both operate on the real line.
Why it failsA stump uses one threshold, while a 3-piece classifier uses two ordered thresholds.
Do this insteadCompare the number of thresholds and resulting regions, not only the domain on which the classifiers operate.
Common mistake Interpreting weak learner as meaning identical classifier family.
What it looks likeConcluding that decision stumps must contain the full three-region structure of H.
Why it failsThe source presents decision stumps as the simpler learner class B and 3-piece classifiers as the more structured class H.
Do this insteadRead weak learner as a useful learning role for the more structured target class, not as structural equality.
Common mistake Ignoring the sign parameter.
What it looks likeListing only the thresholds when specifying a 3-piece classifier.
Why it failsThe classifier specification also includes b in {+1, -1}.
Do this insteadRecord the thresholds and the binary sign parameter together.
Common mistake Treating gamma = 1/12 as a derivation rather than a stated guarantee parameter.
What it looks likeClaiming an error formula that is not supplied by the source.
Why it failsThe source states the weak-learning parameter but does not give the detailed error inequality or its derivation.
Do this insteadState that ERM_B is identified as a 1/12-weak learner for H, without inventing an unsupported formula.
Check Your Understanding
Explain, in your own words, why two ordered thresholds produce a 3-piece classifier while one threshold produces a decision stump. Then explain what the statement ERM_B is a 1/12-weak learner for H tells you about the relationship between the stump learner class B and the 3-piece target class H.
Hints
- Start by counting the intervals created on the real line.
- Name the parameters used by each classifier family.
- Emphasize that B is simpler than H but is still used as a weak learner for H.
- Treat 1/12 as the stated weak-learning parameter because no error inequality is provided here.
Key Takeaways
- 1A 3-piece classifier operates on the real line and is specified by theta_1, theta_2, and b, with theta_1 < theta_2 and b in {+1, -1}.
- 2The two ordered thresholds divide the real line into three intervals.
- 3A decision stump uses one threshold and a sign parameter, giving it a simpler two-region structure.
- 4Decision stumps form the learner class B and are used as a weak learner for the more structured 3-piece class H.
- 5The statement that ERM_B is a 1/12-weak learner for H gives the specified weak-learning parameter gamma = 1/12.
At a glance
- Two ordered thresholds create three intervals on the real line, defining the structure of a 3-piece classifier.
- A decision stump uses one threshold and is therefore structurally simpler.
- The decision-stump class B can still serve as a weak learner for the 3-piece class H.
- ERM_B being a 1/12-weak learner for H means that 1/12 is the stated weak-learning parameter for this relationship.
- The exact label assignment and error inequality are not specified in the provided source, so they should not be inferred from the structural description alone.