Squared Loss
For linear regression with squared loss, least squares solves the empirical risk minimization problem.
Try it: Least-Squares Line
How least squares fits a line: the slope and intercept that minimise the mean squared error come from solving the normal equations, and no hand-drawn line can do better.
How it works
- Each point's residual is y − ŷ, the vertical gap to the line; its square is the squared error.
- Add up n, Σx, Σy, Σx² and Σxy.
- Write the normal equations (XᵀX)w = Xᵀy for w = (slope a, intercept b) and solve the 2×2 system.
- Measure every residual of the fitted line and average the squares (MSE).
- Compare with your own line: its MSE is always at least the least-squares MSE.
Default run (13 steps): 8 points (x, y). Your line ŷ = 0.2x + 4 has mean squared error 1.78. Least squares finds the line with the smallest possible MSE. … MSE = 1.744 / 8 = 0.218. Your line: 1.78 — 1.562 worse. No other line can beat least squares on this data.
Simplified: One input feature and at most 12 points; the 2×2 system is solved in closed form. If every x is equal the system is singular and the lab stops.
Loading the simulation…
From Prediction to Optimization
Linear regression chooses a predictor from a class of possible predictors. The training data determine which predictor is best through empirical risk minimization: choose the predictor whose loss on the training set is smallest. When the loss is squared loss, least squares is the algorithm used to perform this search for the linear regression class.
The important transition is from many training examples to one objective function. Each example contributes to the squared-loss objective, and least squares searches for the regression weights that minimize that objective.
Residuals Become Loss
A regression predictor produces a prediction for each training instance. The difference between a target and its prediction is a residual error. Under squared loss, the contribution of that error is squared before the contributions are combined into the objective that least squares minimizes.
Tracing One Training Problem
A linear regression model must be selected from a class of possible predictors, and squared loss is used on the training set. What sequence of ideas turns this selection problem into a least-squares problem?
- 1Evaluate predictors
Use the training data to evaluate the loss produced by possible linear predictors.
- 2Form the objective
Use the squared-loss contributions from the training instances to define the squared-loss objective.
- 3Choose least squares
Least squares carries out the empirical risk minimization search for the linear regression class when the loss is squared loss.
- 4Find the weights
The regression weights are obtained by deriving the gradient condition and rewriting it as a linear system.
The original predictor-selection problem becomes the task of finding regression weights that minimize the squared-loss objective.
Why the Gradient Vanishes
Least squares minimizes the squared-loss objective. To locate an optimal set of weights, the derivation calculates the gradient of that objective and compares it with zero. The zero-gradient condition is not merely a diagnostic checked after training. It is the mathematical condition used to derive the system that determines the weights.
The System A w = b
After the gradient condition has been formed, the empirical risk minimization problem can be rewritten as the linear system A w = b. In this compact expression, A and b are determined by the problem formulation, while w is the vector of regression weights that must be found.
Reading the Linear System
Interpret the roles of A, w, and b in A w = b after the gradient condition has been derived.
- 1Identify A
A is determined by the formulation of the regression problem.
- 2Identify b
b is also determined by the formulation of the regression problem.
- 3Identify w
w is the vector of regression weights that must be found.
- 4Solve the system
The solution of A w = b gives the regression weights for the least-squares problem.
A w = b is the central computational form: A and b describe the formed system, and w is the unknown weight vector.
When A Is Invertible
Once the gradient condition has been converted into A w = b, the next question concerns matrix A. If A is invertible, the empirical risk minimization problem has the corresponding direct linear-algebra solution for the weights. The optimization problem has therefore been reduced to solving the system under the invertible-matrix case.
When A Is Not Invertible
A different tool is needed when A is not invertible. This situation can occur when the training instances do not span the entire space of R d. Eigenvalue decomposition provides the needed linear-algebra tools for analyzing this case.
The non-invertible case does not discard the original system. The system remains A w = b, and in this setting it still has a solution because b is in the range of A. Eigenvalue decomposition supplies the linear-algebra perspective needed to work with the structure that makes A non-invertible.
Mistakes to Avoid
Common mistake Treating least squares as separate from empirical risk minimization.
What it looks likeDescribing least squares only as an equation-solving method and ignoring the predictor-selection problem.
Why it failsFor linear regression with squared loss, least squares is the algorithm used to carry out empirical risk minimization.
Do this insteadStart with the training objective: choose the linear predictor whose squared loss on the training set is smallest.
Common mistake Skipping the gradient condition.
What it looks likeJumping directly from the squared-loss objective to A w = b.
Why it failsThe gradient condition is the mathematical step used to derive the linear system.
Do this insteadCalculate the objective gradient, compare it with zero, and then rewrite that condition as A w = b.
Common mistake Assuming A is always invertible.
What it looks likeUsing the direct invertible-matrix treatment without checking the matrix case.
Why it failsThe source distinguishes an invertible case from a non-invertible case, which requires eigenvalue-decomposition tools.
Do this insteadIdentify the status of A before choosing the linear-algebra treatment.
Common mistake Thinking that non-invertibility removes the system.
What it looks likeConcluding that A w = b cannot be used when A is not invertible.
Why it failsThe system remains the central form, and in the described setting it still has a solution because b is in the range of A.
Do this insteadKeep the system and use eigenvalue decomposition to analyze the non-invertible case.
Practice the Derivation Path
Explain the full reasoning path for a linear regression problem with squared loss. Your explanation should begin with empirical risk minimization, identify the role of least squares, state why the objective gradient is compared with zero, interpret A w = b, and finish by distinguishing the invertible and non-invertible cases for A.
Hints
- Describe what is minimized on the training set.
- Treat the zero-gradient condition as the bridge from optimization to linear algebra.
- State which symbols are known from the problem formulation and which vector must be found.
- Mention eigenvalue decomposition for the non-invertible case.
What do you think happens?
After the squared-loss objective has been formed, what mathematical condition is used to derive the system for the regression weights?
Reveal answer
Compare the objective's gradient with zero
The gradient condition is the step that is rewritten as the linear system A w = b.
The Complete Picture
- 1Empirical risk minimization selects the linear predictor with the smallest training loss; with squared loss, least squares performs this search.
- 2The gradient of the squared-loss objective is compared with zero because that condition is used to derive the weights.
- 3The derived condition is rewritten as the central linear system A w = b, where w contains the regression weights.
- 4If A is invertible, the system has the corresponding direct linear-algebra solution.
- 5If A is not invertible, eigenvalue decomposition provides the needed tools; in the described setting, the system still has a solution because b is in the range of A.
Key Takeaways
- Least squares is empirical risk minimization for the linear regression class when the loss is squared loss.
- The zero-gradient condition transforms the optimization problem into a solvable mathematical condition.
- The central computational form is A w = b, with w representing the regression weights.
- An invertible A supports the corresponding direct solution, while a non-invertible A calls for eigenvalue-decomposition tools.
- When A is not invertible in the described setting, b lies in the range of A, so the system still has a solution.