Modeling Relationships Between Variables

The direction of the problem matters: explanatory variables provide the input, while the real-valued outcome is what the learned predictor aims to approximate.

Programming

From Information to an Estimate

Suppose you want to estimate a baby's weight. You might use information such as her age and weight at birth. The information used to make the estimate consists of explanatory variables. The quantity being estimated is the outcome. Linear regression is a statistical tool for modeling this kind of relationship.

Which variables are used as inputs, and which single real-valued quantity is the model trying to predict?

Key point

The direction of the problem matters: explanatory variables provide the input, while the real-valued outcome is what the learned predictor aims to approximate.

The Learning Problem

When linear regression is viewed as a learning problem, its input domain X is a subset of R^{^d} for some d. An input is therefore represented as a point in a d-dimensional real-valued space. Its label set Y is the set of real numbers. The outcome is a real-valued label associated with an input.

How does an input from the domain map to a real-valued label in the learning problem?

Definition

Linear regression is used to model relationships between explanatory variables and a real-valued outcome.

Choosing a Function

The goal is to learn a linear function h that maps an input from R^{^d} to a real number. The function h is a predictor: it takes the explanatory-variable input and produces an estimated outcome. The learning goal is not to choose any possible kind of function. It is to find a function that belongs to the allowed family and best approximates the relationship between the inputs and the outcome.

Definition

A hypothesis class is the collection of candidate functions that a learning problem is allowed to consider. In linear regression, the hypothesis class is restricted to linear functions.

What family of possible functions can the learner choose from before selecting a suitable predictor?

Following the Predictor

Worked example

Estimating a Baby's Weight

Identify the inputs, outcome, domain, label set, and learned predictor in a linear-regression learning problem about estimating a baby's weight.

1
Identify the explanatory variables
The baby's age and weight at birth are information used to make the estimate, so they serve as explanatory variables.
2
Identify the outcome
The baby's weight is the quantity being estimated, so it is the real-valued outcome.
3
Describe the domain
The combined input belongs to the input domain X, which is a subset of R^{^d} for some d.
4
Describe the label set
The outcome belongs to Y, the set of real numbers.
5
Describe the learned predictor
The learning task is to learn a linear function h that maps the input to a real number and best approximates the relationship between the explanatory variables and the baby's weight.

Result

The explanatory variables form the input, the baby's weight is the real-valued outcome, and the learned linear predictor maps the input to an estimated real number.

How does the learned predictor approximate the underlying relationship between explanatory variables and the observed outcome?

The learned predictor is the result of the modeling process. It is not the explanatory variable and it is not the observed outcome. It is a learned linear function that uses the input to produce a real-number estimate, with the aim of approximating the relationship represented by the data.

Mistakes in Problem Direction

Common mistake
Treating the outcome as an explanatory variable
What it looks like
Calling the baby's weight the input when the problem is estimating the baby's weight from age and weight at birth.
Why it fails
The explanatory variables provide the input, while the real-valued outcome is what the predictor aims to approximate.
Do this instead
First identify what information is used to make the estimate. Then identify the single quantity being estimated.
Common mistake
Describing linear regression as only a description of data
What it looks like
Stopping after saying that the variables are related, without identifying a function that produces an estimate.
Why it fails
Linear regression is also a way to learn a predictor from the relationship being modeled.
Do this instead
Include the learned function h and its role in mapping an input to a real number.
Common mistake
Assuming the hypothesis class contains every possible function
What it looks like
Treating any candidate relationship as an acceptable hypothesis for linear regression.
Why it fails
The hypothesis class in linear regression is restricted to linear functions.
Do this instead
State that the learner selects or learns a suitable member of the linear-function class.
Common mistake
Confusing the domain with the label set
What it looks like
Describing both the input and the outcome as belonging to the same set.
Why it fails
The input domain X is a subset of R^{^d} for some d, while the label set Y is the set of real numbers.
Do this instead
Keep the input space X and the real-valued label set Y distinct when describing the learning problem.

Best practice

Before discussing a linear model, state the direction of the problem in words: these explanatory variables are the input, and this real-valued quantity is the outcome. This simple step prevents the model's inputs, labels, and learned predictor from being mixed together.

Practice and Summary

MEDIUM

A learning problem uses several explanatory variables to estimate one real-valued quantity. Explain which part is the input, which part is the outcome, what X and Y represent, and why the learned predictor belongs to the linear-function hypothesis class.

Hints

Start by identifying the information used to make the estimate.
Then identify the quantity being estimated.
Use X for the input domain and Y for the real-valued label set.
Finish by explaining that the learned predictor is a suitable linear function.

Your attempt (not saved)

1
Linear regression models relationships between explanatory variables and a real-valued outcome. Its input domain X is a subset of R^d for some d, and its label set Y is the set of real numbers. The hypothesis class contains linear functions. The learning goal is to learn a suitable linear function h that maps an input to a real number and best approximates the relationship between the inputs and the outcome.

At a glance

Explanatory variables provide the input to a linear-regression problem.
The outcome is a real-valued quantity that the predictor aims to approximate.
The input domain X is a subset of R^{^d} for some d, while the label set Y is the set of real numbers.
The hypothesis class in linear regression consists of linear functions.
A learned predictor h maps an input to a real number and approximates the relationship between the explanatory variables and the outcome.

More concepts starting with M