Mean Squared Error

When squared loss is used across a data set, the resulting empirical risk function is called the Mean Squared Error.

Programming

Interactive lab

Try it: Least-Squares Line

How least squares fits a line: the slope and intercept that minimise the mean squared error come from solving the normal equations, and no hand-drawn line can do better.

How it works

Each point's residual is y − ŷ, the vertical gap to the line; its square is the squared error.
Add up n, Σx, Σy, Σx² and Σxy.
Write the normal equations (XᵀX)w = Xᵀy for w = (slope a, intercept b) and solve the 2×2 system.
Measure every residual of the fitted line and average the squares (MSE).
Compare with your own line: its MSE is always at least the least-squares MSE.

Default run (13 steps): 8 points (x, y). Your line ŷ = 0.2x + 4 has mean squared error 1.78. Least squares finds the line with the smallest possible MSE. … MSE = 1.744 / 8 = 0.218. Your line: 1.78 — 1.562 worse. No other line can beat least squares on this data.

Simplified: One input feature and at most 12 points; the 2×2 system is solved in closed form. If every x is equal the system is singular and the lab stops.

Educational simulation

Loading the simulation…

Why Error Size Matters

A regression model rarely produces only perfectly correct or completely incorrect answers. Suppose the actual value is 3 kg. A prediction of 3.00001 kg and a prediction of 4 kg are both different from the actual value, but they are not equally bad. Regression therefore needs a rule that converts the discrepancy between a prediction and the actual value into a numerical penalty.

What happens to a regression loss as the prediction moves from exactly correct to slightly wrong to substantially wrong?

Two Penalty Rules

A loss function evaluates one prediction against its target. The prediction and target provide the same discrepancy in either calculation, but the loss function determines how that discrepancy becomes a penalty. Two common choices for regression are squared loss and absolute value loss.

Loss choice	How the discrepancy is transformed	What changes when the discrepancy changes
Squared loss	Square the discrepancy	The penalty follows the squared-loss rule
Absolute value loss	Take the absolute value of the discrepancy	The penalty follows the absolute-value rule

Both choices begin with the prediction-target discrepancy but apply different penalty rules.

How do squared loss and absolute value loss respond differently as the prediction moves farther from the actual value?

Applying Loss to One Example

Worked example

One Prediction, Two Losses

A model predicts 5 for a target whose actual value is 3. Apply squared loss and absolute value loss.

1
Find the discrepancy
Subtract the actual value from the prediction: 5 - 3 = 2.
2
Apply squared loss
Square the discrepancy: 2 squared = 4.
3
Apply absolute value loss
Take the absolute value of the discrepancy: absolute value of 2 = 2.

Result

For this generated example, squared loss is 4 and absolute value loss is 2. The two values differ because the same discrepancy was passed through two different penalty rules.

How is one numerical prediction-target discrepancy transformed into squared loss or absolute value loss?

What do you think happens?

A prediction is 7 and the target is 3. Which loss is larger for this example?

Squared loss
Absolute value loss
They are always equal

Reveal answer

Answer:

Squared loss

The discrepancy is 4. Squared loss applies the squaring rule to that discrepancy, while absolute value loss applies the absolute-value rule. The penalty values therefore differ.

From One Loss to Mean Squared Error

A loss function evaluates one example at a time. When squared loss is used across a data set, the resulting empirical risk function is called the Mean Squared Error. The individual squared-loss expression describes one prediction's penalty; Mean Squared Error names the empirical risk associated with using squared loss across the sample.

Worked example

Individual Loss Versus Empirical Risk

Consider two generated prediction-target pairs. The first pair has prediction 5 and target 3. The second pair has prediction 4 and target 5. Compare the squared loss on the first example with the empirical risk obtained by averaging the two squared losses.

1
Evaluate the first example
Its discrepancy is 5 - 3 = 2, so its squared loss is 4.
2
Evaluate the second example
Its discrepancy is 4 - 5 = -1, so its squared loss is 1 when the discrepancy is squared.
3
Combine the sample losses
The empirical risk associated with squared loss averages the squared losses across the data set: (4 + 1) divided by 2 = 2.5.
4
Compare the quantities
The first example's squared loss is 4. The sample-level empirical risk is 2.5. They are different quantities because one describes one example and the other averages losses across the sample.

Result

Individual squared loss: 4. Mean Squared Error as the empirical risk for this two-example sample: 2.5.

How does the loss for one prediction differ from the empirical risk obtained by averaging losses across an entire sample?

Discretization for Analysis

Loss functions also matter when studying how much data is needed for learning guarantees. Linear regression is not a binary prediction task, so its sample complexity cannot be analyzed using the VC-dimension in the way described for binary classification. One possible route is the discretization trick.

The basic idea is to replace a continuous collection of possible linear hypotheses with a finite collection by discretizing possible parameter or prediction values. Instead of allowing every value in a continuous range, the analysis considers selected discrete values. This produces a finite hypothesis class that can be used in a sample-complexity analysis.

How does discretizing possible parameter or prediction values change a continuous set of linear hypotheses into a finite collection?

Mistakes to Avoid

Common mistake
Treating every wrong regression prediction as equally bad.
What it looks like
Calling predictions of 3.00001 kg and 4 kg equally bad simply because both differ from an actual value of 3 kg.
Why it fails
Regression loss is intended to quantify the discrepancy, not merely record whether the prediction is different.
Do this instead
Use a loss function that converts the size of the discrepancy into a numerical penalty.
Common mistake
Confusing squared loss with absolute value loss.
What it looks like
Using the squared-loss rule when the chosen setup specifies absolute value loss.
Why it fails
The two choices begin with the same discrepancy but apply different penalty rules.
Do this instead
Identify the selected loss function before transforming the discrepancy.
Common mistake
Calling the loss on one example Mean Squared Error.
What it looks like
Reporting the squared loss for one prediction as the data-set-level empirical risk.
Why it fails
Mean Squared Error is the empirical risk function associated with squared loss across a data set.
Do this instead
Use squared loss for one example and Mean Squared Error for the empirical-risk calculation across the sample.
Common mistake
Assuming binary-classification analysis applies directly to linear regression.
What it looks like
Trying to analyze linear-regression sample complexity with the VC-dimension approach described for binary classification.
Why it fails
Linear regression is not a binary prediction task.
Do this instead
Recognize discretization as one possible route to making the linear-regression hypothesis class finite for analysis.

Check Your Understanding

MEDIUM

A model makes two predictions. For the first example, the prediction is 6 and the target is 4. For the second, the prediction is 3 and the target is 4. Calculate the squared loss for each example, then calculate the empirical risk associated with squared loss by averaging the two values. Finally, state whether your individual loss for the first example is the same quantity as the empirical risk.

Hints

Compute each prediction-target discrepancy first.
Apply the squaring rule to each discrepancy.
Average the two squared losses for the empirical-risk value.
Keep the one-example loss separate from the sample-level average.

Your attempt (not saved)

MEDIUM

In your own words, explain why replacing continuous parameter or prediction values with discrete choices can help make a linear-regression hypothesis class finite for sample-complexity analysis.