Linear Regression for Modeling Relationships
For linear regression with squared loss, least squares solves the empirical risk minimization problem.
Try it: Least-Squares Line
How least squares fits a line: the slope and intercept that minimise the mean squared error come from solving the normal equations, and no hand-drawn line can do better.
How it works
- Each point's residual is y − ŷ, the vertical gap to the line; its square is the squared error.
- Add up n, Σx, Σy, Σx² and Σxy.
- Write the normal equations (XᵀX)w = Xᵀy for w = (slope a, intercept b) and solve the 2×2 system.
- Measure every residual of the fitted line and average the squares (MSE).
- Compare with your own line: its MSE is always at least the least-squares MSE.
Default run (13 steps): 8 points (x, y). Your line ŷ = 0.2x + 4 has mean squared error 1.78. Least squares finds the line with the smallest possible MSE. … MSE = 1.744 / 8 = 0.218. Your line: 1.78 — 1.562 worse. No other line can beat least squares on this data.
Simplified: One input feature and at most 12 points; the 2×2 system is solved in closed form. If every x is equal the system is singular and the lab stops.
Loading the simulation…
From Observations to a Predictor
Linear regression begins with a class of possible predictors and training data. The goal is to select the predictor that performs best on the observed training examples. When performance is measured with squared loss, least squares is the algorithm that carries out this search.
Empirical Risk and Squared Loss
Empirical risk minimization means choosing the predictor whose loss on the training set is smallest. In the linear regression setting described here, the loss is squared loss. Therefore, least squares minimizes the squared-loss objective over the class of linear regression predictors. The important connection is that least squares is not a separate goal from empirical risk minimization in this setting; it is the method used to perform empirical risk minimization for linear regression with squared loss.
Following the ERM Choice
Suppose a linear regression model must be selected from a class of possible predictors, and the training data are evaluated with squared loss.
- 1Identify the candidates
The possible predictors form the linear regression class from which a model will be selected.
- 2Evaluate training loss
Each candidate is evaluated by its loss on the observed training set.
- 3Use squared loss
Because the chosen loss is squared loss, the objective is the squared-loss objective.
- 4Select the least-squares solution
Least squares performs the search for the linear regression predictor with the smallest training loss.
Least squares solves the empirical risk minimization problem for linear regression with squared loss.
Why the Gradient Becomes Zero
After defining the squared-loss objective, the derivation looks for an optimal set of regression weights. The stated procedure is to calculate the objective's gradient and compare it with zero. A zero gradient is the condition used to identify a stationary solution of the objective, so it is part of the derivation of the weights rather than a diagnostic added after training.
The Central System A w = b
Once the gradient condition has been formed, the empirical risk minimization problem is rewritten as the linear system A w = b. The symbols have distinct roles: A and b are determined by the problem formulation, while w is the vector of regression weights that must be found. Solving this system therefore gives the weights selected by the least-squares derivation.
Tracing the Algebraic Form
Trace how a least-squares problem becomes a system for the regression weights without inserting particular numerical data.
- 1Start with the objective
Least squares minimizes the squared-loss objective for the linear regression class.
- 2Calculate the gradient
The derivation calculates the gradient of that objective.
- 3Compare with zero
The gradient condition is set equal to zero to identify the stationary solution condition.
- 4Rewrite the condition
The zero-gradient condition is rewritten in the compact form A w = b.
- 5Solve for w
The unknown vector w contains the regression weights, so solving the system gives the weights.
The optimization problem has been converted into a linear-system problem whose central unknown is the regression-weight vector w.
Invertibility and Eigenvalue Decomposition
The next step is to determine what kind of matrix A appears in A w = b. If A is invertible, the system has the corresponding direct linear-algebra solution for the weights. In this case, the gradient-derived system can be handled through the invertible-matrix case.
If A is not invertible, a direct inverse is not the appropriate tool. The source account connects this case with training instances that do not span the entire space of R d and identifies eigenvalue decomposition as the needed linear-algebra tool. In the stated setting, A w = b still has a solution because b is in the range of A. Eigenvalue decomposition supplies a way to reason about the matrix when its invertibility must be handled explicitly.
Mistakes in the Derivation
Common mistake Treating least squares and empirical risk minimization as unrelated procedures.
What it looks likeDescribing empirical risk minimization as the model-selection goal but presenting least squares as a separate algorithm with no connection to that goal.
Why it failsFor linear regression with squared loss, least squares is the algorithm used to carry out empirical risk minimization.
Do this insteadState the connection explicitly: choose the linear predictor with the smallest training squared loss, and least squares performs that search.
Common mistake Skipping the gradient condition.
What it looks likeJumping directly from the squared-loss objective to A w = b without explaining where the system comes from.
Why it failsThe gradient condition is the mathematical step used to derive the system that determines the weights.
Do this insteadCalculate the objective gradient, compare it with zero, and then rewrite that condition as A w = b.
Common mistake Confusing A and b with the unknown weights.
What it looks likeSaying that every symbol in A w = b is an unknown parameter vector.
Why it failsA and b are determined by the problem formulation, while w is the vector of regression weights to be found.
Do this insteadRead A w = b as a system whose known formulation determines A and b and whose unknown is w.
Common mistake Using the invertible-matrix reasoning without checking A.
What it looks likeApplying the direct-solution case even when A is not invertible.
Why it failsThe non-invertible case requires eigenvalue decomposition as the relevant linear-algebra tool.
Do this insteadSeparate the invertible case from the non-invertible case before choosing the linear-algebra method.
Check Your Understanding
Explain the complete reasoning chain in your own words: a linear regression class is evaluated on training data with squared loss; least squares minimizes that empirical risk; the gradient of the objective is compared with zero; the resulting condition is rewritten as A w = b; and the matrix case determines whether the direct solution or eigenvalue-decomposition tools are appropriate.
Hints
- Begin with the model-selection goal, not with the matrix equation.
- Identify w as the unknown regression-weight vector.
- Mention why the non-invertible case cannot be handled by the same direct inverse reasoning.
- 1Linear regression with squared loss is an empirical risk minimization problem. Least squares minimizes the training squared-loss objective. Setting the objective gradient equal to zero supplies the condition used to derive the regression weights. That condition is rewritten as A w = b, where w is the unknown weight vector and A and b come from the problem formulation. If A is invertible, the corresponding direct linear-algebra solution applies. If A is not invertible, eigenvalue decomposition provides the needed tool, and in the stated setting b lies in the range of A so the system still has a solution.
Key Takeaways
- Least squares solves empirical risk minimization for linear regression with squared loss.
- The zero-gradient condition is the derivation step that identifies the stationary least-squares solution.
- The condition is rewritten as A w = b, with w as the regression-weight vector.
- An invertible A supports the corresponding direct solution, while a non-invertible A requires eigenvalue-decomposition tools.
- When A is non-invertible in the stated setting, b is in the range of A, so the system still has a solution.