Generalizing the Learning Model

The learner's goal and the learner's available evidence are not the same thing: the goal concerns true error, while the directly available training data supports empirical-risk evaluation.

Programming

From Training Performance to Generalization

A prediction rule h can be evaluated in two related ways. One view asks how likely h is to make an error on labeled points drawn according to the probability distribution D. This is the true error, also called the risk. The other view checks errors on the training data S. This is empirical risk. The distinction is central because the learner wants a rule with low true error, but does not know D and can directly access only S.

Key point

The learner's goal and the learner's available evidence are not the same thing: the goal concerns true error, while the directly available training data supports empirical-risk evaluation.

Following D to Labeled Points

How does the distribution D provide labeled points for evaluating the prediction rule h?

D is the probability distribution associated with the labeled points used to evaluate h. True error considers how likely h is to make an error on labeled points drawn according to D. Therefore, true error is not limited to the particular examples already placed in S; it concerns the error behavior of h with respect to points associated with the distribution.

Tracing the Two Evaluation Sources

What does the learner want to minimize, and what quantity can the learner compute from the training data?

Measure	Evaluation source	Role for the learner
True error, or risk	Labeled points drawn according to D	The error measure the learner wants to minimize
Empirical risk	Training data S	The error measure the learner can evaluate from available data

Both measures evaluate the same prediction rule h, but they use different sources of evaluation.

The prediction rule h is the common object in both evaluations. True error uses labeled points associated with the unknown distribution D. Empirical risk uses the training data S that the learner has. The difference is not which prediction rule is being judged; it is where the evaluation points come from.

A Worked Evaluation Scenario

Worked example

One Rule, Two Error Questions

Consider a prediction rule h, a probability distribution D, and a training set S. Identify which evaluation source is used by each question.

1
Question about future distribution-based behavior
Ask how likely h is to make an error on labeled points drawn according to D. This is a question about true error, or risk.
2
Question about the available examples
Check the errors made by h on the training data S. This is a question about empirical risk.
3
Compare the information available
The learner has access to S but does not know D. Therefore, the learner can directly evaluate empirical risk but cannot directly use knowledge of D to evaluate true error.

Result

True error uses D as its evaluation source; empirical risk uses S. The learner wants low true error but can directly work with the training data S.

Mistakes About What Can Be Minimized

Common mistake
Saying that the learner directly minimizes true error using D.
What it looks like
The learner knows D and selects h by evaluating its true error.
Why it fails
The learner does not know the probability distribution D.
Do this instead
State that the learner wants a prediction rule with low true error, while the learner has direct access to the training data S and can use it to measure empirical risk.
Common mistake
Calling error on the training data true error.
What it looks like
The errors of h on S are described as the risk of h with respect to D.
Why it fails
Errors measured on S are the basis of empirical risk, not the distribution-based evaluation called true error.
Do this instead
Use empirical risk when the evaluation source is S, and true error or risk when the evaluation source is labeled points drawn according to D.
Common mistake
Ignoring the prediction rule h when comparing the two measures.
What it looks like
True error evaluates D, while empirical risk evaluates S.
Why it fails
Both measures evaluate the same kind of object: a prediction rule h.
Do this instead
Say that both measures evaluate h and differ in the source used for that evaluation.

Check the Evaluation Path

EASY

A learner has a prediction rule h and access to a training set S, but does not know D. For each statement, identify whether it describes true error or empirical risk: measuring errors on S; asking how likely h is to make an error on labeled points drawn according to D; choosing a goal for the quality of h; using information the learner can directly access.

Hints

Look first for the evaluation source: D or S.
Separate the learner's desired goal from the quantity the learner can directly evaluate.

Your attempt (not saved)

1
True error, or risk, asks how likely a prediction rule h is to make an error on labeled points drawn according to D. Empirical risk measures the error of h using the training data S. The learner wants low true error but does not know D. Because the learner does have access to S, empirical risk is the directly available evaluation source. The two measures concern the same prediction rule and differ in where their evaluation points come from.

Key Takeaways

True error, or risk, evaluates a prediction rule h with respect to labeled points drawn according to D.
Empirical risk evaluates the prediction error of h using the training data S.
The learner wants a prediction rule with low true error.
The learner does not know D but does have access to S.
True error and empirical risk evaluate the same rule h through different evaluation sources.

Related concepts

More concepts starting with G