Learning Problems and Hypothesis Classes
The direction of the problem matters: explanatory variables are used as inputs, while the real-valued outcome is the quantity the predictor aims to estimate.
From Information to an Estimate
Suppose you want to estimate a baby's weight using information such as her age and weight at birth. The age and birth weight are the explanatory variables: they provide the information used to make the estimate. The baby's current weight is the outcome: it is the real-valued quantity being estimated. Linear regression is a statistical tool for modeling this kind of relationship.
The direction of the problem matters: explanatory variables are used as inputs, while the real-valued outcome is the quantity the predictor aims to estimate.
The Learning-Problem View
Linear regression can be viewed as a learning problem. Its input domain, written as X, is a subset of the real-coordinate space R to the power d for some d. Each input is therefore a feature vector with d components, or a member of the allowed collection of inputs. Its label set, written as Y, is the set of real numbers. A label is the real-valued outcome associated with an input.
Classifying the Parts of an Example
Consider the baby-weight estimation task described in the source.
- 1Identify the inputs
Age and weight at birth are information used to make the estimate, so they belong with the explanatory variables.
- 2Identify the outcome
The baby's weight being estimated is the outcome. It is a real-valued quantity.
- 3Place them in the learning formulation
The input belongs to the input domain X, while the estimated outcome belongs to the real-valued label set Y.
The task pairs an input made from explanatory variables with a real-valued outcome.
Choosing the Hypothesis Class
A hypothesis class is the collection of candidate functions that a learning problem is allowed to consider. In linear regression, the hypothesis class is restricted to linear functions. The learning task is to select or learn a suitable member of that class so it can approximate the relationship between the explanatory variables and the outcome.
Generated example: imagine that one candidate linear function predicts an outcome using a feature vector with one set of weights and an intercept, while another candidate uses different weights and an intercept. These are different hypotheses, but both remain inside the linear-function hypothesis class. Learning consists of choosing a suitable candidate for the relationship represented by the data.
Following the Learned Predictor
The goal is not merely to name or describe the variables. A learned function, written as h, maps an input from R to the power d to a real number. Its purpose is to best approximate the relationship between the explanatory variables and the outcome. In this sense, linear regression provides a way to learn a predictor from the relationship being modeled.
Tracing a Prediction Task
Separate the roles of the variables in a task that estimates a baby's weight from age and weight at birth.
- 1Start with the information
Age and weight at birth are selected as explanatory variables because they provide the input information.
- 2Specify the target
The baby's weight is the real-valued outcome that the task aims to estimate.
- 3Restrict the candidates
The learner considers linear functions as candidate predictors because linear regression uses a linear-function hypothesis class.
- 4Learn a predictor
The learned function h maps the input to a real number intended to approximate the relationship between the explanatory variables and the baby's weight.
The complete learning problem moves from explanatory-variable inputs, through a learned linear predictor, to an approximated real-valued outcome.
Mistakes in Problem Formulation
Common mistake Treating the outcome as an explanatory variable.
What it looks likeCalling the baby's estimated weight an input rather than the quantity being estimated.
Why it failsThe explanatory variables provide information for the estimate, while the outcome is the real-valued quantity the predictor aims to approximate.
Do this insteadAsk which quantities are supplied to make the estimate and which quantity is the target.
Common mistake Describing linear regression only as a way to summarize data.
What it looks likeStopping after saying that the variables have a relationship.
Why it failsThe learning formulation emphasizes learning a predictor that maps an input to a real number.
Do this insteadInclude the learned function and its role in approximating the relationship.
Common mistake Confusing the hypothesis class with one learned hypothesis.
What it looks likeUsing the phrase hypothesis class to refer to only the final function selected by the learner.
Why it failsA hypothesis class is the collection of candidate functions, while the learned predictor is a suitable member selected from that collection.
Do this insteadUse hypothesis class for the full collection and learned predictor for the selected function.
Common mistake Forgetting the restriction to linear functions.
What it looks likeDescribing the learning task as choosing any possible function.
Why it failsLinear regression restricts its candidate functions to linear functions.
Do this insteadState that the hypothesis class consists of linear functions.
| Role | Meaning in the learning problem | Baby-weight example |
|---|---|---|
| Explanatory variables | Inputs used to make an estimate | Age and weight at birth |
| Outcome | Real-valued quantity being estimated | Baby's weight |
| Hypothesis class | Collection of candidate functions | Linear functions |
| Learned predictor | Selected function that approximates the relationship | Function h mapping inputs to a real number |
The roles are related but not interchangeable.
Check Your Formulation
A learning task uses two measurements as input and aims to estimate one real-valued quantity. Explain which measurements belong to the explanatory variables, what belongs to the label set, what the input domain represents, and why the candidate predictors belong to a hypothesis class.
Hints
- Start by identifying what information is available before the estimate is made.
- The target is the real-valued quantity being estimated.
- The input domain contains allowed inputs, and the label set contains real-valued labels.
- For linear regression, the hypothesis class is restricted to linear functions.
What do you think happens?
A task estimates a baby's weight from age and weight at birth. Which item is the outcome?
Reveal answer
The baby's weight being estimated
Age and weight at birth are explanatory variables used as input information. The baby's weight is the real-valued outcome that the learned predictor aims to approximate.
Key Takeaways
- 1Linear regression models relationships between explanatory variables and a real-valued outcome.
- 2The explanatory variables form the input, while the outcome is the quantity being estimated.
- 3The input domain X is a subset of R to the power d, and the label set Y is the set of real numbers.
- 4A hypothesis class is a collection of candidate functions; linear regression uses a class of linear functions.
- 5The learned predictor is a linear function that maps an input to a real number and aims to approximate the modeled relationship.
At a glance
- Linear regression is a statistical tool for modeling a relationship between explanatory variables and a real-valued outcome.
- In the learning formulation, inputs come from a domain X that is a subset of R to the power d, and labels come from the real numbers.
- The hypothesis class is the collection of candidate functions permitted by the learning problem.
- For linear regression, the hypothesis class consists of linear functions.
- A learned predictor maps an input to a real number and approximates the relationship between the inputs and outcome.