Learning Protocols
The learning protocol separates hypothesis construction from performance estimation.
Separating Construction from Evaluation
A learning protocol organizes learning into two different jobs: constructing a hypothesis and estimating how that hypothesis performs. The sequence T is used for construction. After the hypothesis h_T has been produced, a separate fresh sequence V is used to calculate its error. Keeping these jobs separate is the central idea of the protocol.
The Three Protocol Stages
- 1Use the sequence T, containing k examples, as the construction sequence.
- 2Construct the hypothesis h_T from T.
- 3Use the fresh sequence V, containing m-k examples, to calculate the error of h_T and apply Bernstein's inequality to obtain a related bound.
The notation records the dependency clearly. The subscript in h_T indicates that the hypothesis was constructed from T. It does not indicate that h_T is evaluated on T. The later evaluation uses V, a separate sequence that is fresh and independent of T.
Following the Named Objects
Suppose a protocol has a construction sequence T with k examples and a separate sequence V with m-k examples. Identify the role of each named object.
- 1T
T is the sequence available during hypothesis construction.
- 2h_T
h_T is the hypothesis produced from T.
- 3V
V is the fresh sequence used after construction to calculate the error of h_T.
- 4Final quantity
The protocol focuses on the error of h_T on V, rather than the error of h_T on the sequence that produced it.
The dependency chain is T produces h_T, and h_T is then evaluated on V.
T and V in the Full Sequence
T and V are two different parts of the protocol's examples. T contains k examples and supplies the information used to construct h_T. V contains m-k examples and is fresh and independent of T. A schematic index arrangement can place the k construction examples before the m-k evaluation examples, but the essential distinction is their role: T is used to construct, while V is used afterward to evaluate.
Why Evaluation Uses V
Evaluating h_T on V preserves the separation between construction and performance estimation. The hypothesis was constructed using T, while V is fresh and independent of T. By contrast, evaluating on T would use the same sequence both to construct the hypothesis and to estimate its performance. That is not the separated protocol described here.
| Object | Role | Relationship to h_T |
|---|---|---|
| T | Construction sequence | Used to produce h_T |
| h_T | Constructed hypothesis | Produced from T |
| V | Fresh evaluation sequence | Used to calculate the error of h_T |
The protocol keeps the construction role of T separate from the evaluation role of V.
Bernstein's Inequality in the Protocol
After h_T has been constructed, the protocol calculates its error using V. The independence of V and T is important because h_T was constructed from T. The source states that Bernstein's inequality can then be applied to obtain a result related to the error of h_T on V. In this protocol, Bernstein's inequality is therefore used after construction, as part of the performance-estimation stage.
Mistakes in Reading the Protocol
Common mistake Treating T and V as interchangeable names for the same examples.
What it looks likeUsing T to construct h_T and then describing V as if it were just another name for T.
Why it failsT contains k construction examples, while V is a separate fresh sequence of m-k examples.
Do this insteadTrack the roles separately: T constructs h_T, and V is used afterward to calculate its error.
Common mistake Assuming the subscript in h_T identifies the evaluation sequence.
What it looks likeReading h_T as a hypothesis that must be evaluated on T.
Why it failsThe subscript identifies the sequence from which h_T was constructed.
Do this insteadRead h_T as the hypothesis produced from T, then check its error on V.
Common mistake Applying Bernstein's inequality before constructing h_T.
What it looks likeTreating the inequality as part of the process that creates the hypothesis.
Why it failsThe protocol applies Bernstein's inequality after h_T has been constructed and its error on V has been considered.
Do this insteadFollow the order T, then h_T, then error on V, then the Bernstein-related bound.
Common mistake Ignoring independence between T and V.
What it looks likeDescribing V as fresh but not recognizing why that matters.
Why it failsThe source identifies the independence of T and V as supporting the application of Bernstein's inequality.
Do this insteadMention that h_T comes from T while V is fresh and independent when explaining the bound.
Protocol Check
Explain the protocol in three ordered statements. Your explanation must identify the role of T, the origin of h_T, and the purpose of V.
Hints
- Start with the number of examples in T.
- Use the subscript in h_T to state where the hypothesis comes from.
- End by explaining why V is used and where Bernstein's inequality enters.
What do you think happens?
A learner says: h_T is evaluated on T because the subscript T tells us where evaluation happens. Is that interpretation consistent with the protocol?
Reveal answer
No, because the subscript identifies the construction sequence, while evaluation uses fresh V.
T is used to construct h_T. The protocol then uses the fresh sequence V to calculate the error of h_T, and the independence of T and V supports applying Bernstein's inequality.
Protocol Summary
- 1T contains k examples and is used to construct h_T.
- 2V is a fresh, independent sequence containing m-k examples.
- 3The protocol evaluates h_T on V rather than on T to separate construction from performance estimation.
- 4Bernstein's inequality is applied after evaluation on V to obtain a result related to the error of h_T on V.
At a glance
- The learning protocol separates hypothesis construction from performance estimation.
- T is the sequence of k examples used to construct h_T.
- V is a fresh sequence of m-k examples used to calculate the error of h_T.
- The independence of T and V supports applying Bernstein's inequality to obtain a bound related to that error.