Ridge Regression
Ridge Regression is connected to this idea because its setup includes feature normalization and a learning rule derived from linear regression.
Try it: Polynomial Fit: Overfitting and Ridge
How raising the polynomial degree drives training error down while validation error eventually rises (overfitting), and how a ridge penalty λ‖w‖² tames a high-degree fit.
How it works
- Split the noisy points into a training set and a validation set.
- Build the design matrix with columns (, a rescaled ).
- Solve (XᵀX + λI)w = Xᵀy on the training points by Gaussian elimination with partial pivoting.
- Measure the mean squared error on the training points and, separately, on the validation points.
- Repeat for every degree 0–9: the degree with the lowest validation error is the one validation selects.
Default run (18 steps): 14 noisy points: 10 for training, 4 held out for validation. Fit a degree-3 polynomial with λ = 0. … Degree 3: training MSE 0.0282, validation MSE 0.0308. The lowest validation error is at degree 4. Training error keeps falling with degree; validation error is what reveals overfitting.
Simplified: One input, 14 fixed noisy samples of sin(2πx) and a single train/validation split (no cross-validation). The penalty λ‖w‖² here includes the constant term and is added to the sum (not the mean) of squared errors.
Loading the simulation…
From Features to a Learning Input
A learning algorithm does not work directly with an abstract description of a problem. It works with a feature vector: a representation built from the original features. Feature manipulation changes that representation before it is given to the learner. Ridge Regression is connected to this idea because its setup includes feature normalization and a learning rule derived from linear regression.
Feature manipulation is not automatically good or bad. Its value depends on the learning algorithm being used and on the prior assumptions made about the problem.
Why Transform Features
Feature manipulation transforms each original feature to create a resulting feature vector. There are two stated purposes for doing this. A transformation may reduce approximation or estimation errors, or it may produce a faster algorithm. These purposes are different: one concerns the errors produced by the learning process, while the other concerns how efficiently the algorithm operates.
Two Possible Reasons for One Transformation
Suppose a learner receives several original features. Why might a practitioner transform those features before learning?
- 1Purpose one
The transformation may be chosen to reduce approximation or estimation errors.
- 2Purpose two
The transformation may instead be chosen to obtain a faster algorithm.
- 3Selection principle
The transformation should be considered together with the learning algorithm and the prior assumptions about the problem. There is no claim that one transformation is universally best.
Feature manipulation is useful only in relation to the problem, its assumptions, and the algorithm that will process the resulting feature vector.
Normalization and the Regression Setup
Normalization is introduced as a type of feature manipulation. Its motivation comes from a linear regression problem using squared loss. In the stated setup, the data is represented by a matrix whose rows are instance vectors and by a vector containing target values.
The rows of the matrix X are instance vectors, and y is a vector of target values. Normalization changes the feature representation before the learning rule is applied. In this discussion, the normalized features remain part of the regression input, while the target values describe what the learner is trying to use the input to model.
Building the Ridge Rule
Ridge Regression is best understood as a construction. Start with linear regression in the squared-loss setting. Apply the RLM rule with Tikhonov regularization. The resulting learning rule is called Ridge Regression.
Classifying a Learning Rule by Construction
A description mentions linear regression, squared loss, the RLM rule, and Tikhonov regularization. What learning rule does this combination identify?
- 1Identify the starting model
The starting problem is linear regression.
- 2Identify the loss
The regression problem uses the squared loss.
- 3Identify the rule
The RLM rule is applied to that problem.
- 4Identify the regularization
Tikhonov regularization is used in obtaining the result.
- 5Name the result
The full combination identifies Ridge Regression.
Ridge Regression is recognized from the complete construction, not from any one ingredient considered alone.
Tikhonov regularization is the technique used in obtaining Ridge Regression. It is not, by itself, the complete name of the resulting learning rule.
Original and Regularized Objectives
The original linear regression setup is described through the model and the squared loss. Ridge Regression is the resulting regularized learning rule after the RLM rule is applied with Tikhonov regularization. The distinction is therefore a distinction between the starting regression problem and the rule produced after regularization is introduced.
| Aspect | Original linear regression | Ridge Regression |
|---|---|---|
| Starting point | Linear regression | Linear regression |
| Loss setting | Squared loss | Squared loss with regularization |
| Additional construction | No stated regularization step | RLM rule with Tikhonov regularization |
| Result | Original regression setup | Regularized learning rule called Ridge Regression |
The comparison emphasizes the construction that turns the original setup into the Ridge Regression rule.
Common Classification Mistakes
Common mistake Treating Tikhonov regularization as the complete name of the learning rule.
What it looks likeA description mentions Tikhonov regularization, so the learner immediately labels the result Ridge Regression.
Why it failsTikhonov regularization is a technique used in obtaining Ridge Regression. The full identification also requires the linear regression setup, squared loss, and RLM rule.
Do this insteadClassify the complete combination of model, loss, RLM rule, and regularization technique.
Common mistake Calling every feature transformation universally useful.
What it looks likeA feature transformation is described without mentioning the learner or the problem assumptions.
Why it failsThe correct transformation depends on the learning algorithm and the prior assumptions about the problem.
Do this insteadEvaluate a transformation in the context of the algorithm that will receive the resulting feature vector.
Common mistake Confusing the original regression setup with the regularized learning rule.
What it looks likeLinear regression with squared loss and Ridge Regression are treated as identical descriptions.
Why it failsRidge Regression is obtained after applying the RLM rule with Tikhonov regularization to the stated linear regression problem.
Do this insteadTrace the construction from the starting regression problem to the resulting regularized rule.
Common mistake Ignoring the representation used by the learner.
What it looks likeThe abstract problem description is treated as if it were directly supplied to the algorithm.
Why it failsThe algorithm works with a feature vector built from the original features.
Do this insteadIdentify the feature manipulation and the resulting feature vector before discussing the learning rule.
Trace the Construction
A learning problem is represented by a matrix whose rows are instance vectors and by a vector of target values. The features are normalized. The learner uses linear regression with squared loss, applies the RLM rule, and uses Tikhonov regularization. Identify the resulting learning rule and explain the role of each ingredient.
Hints
- Begin with the model and loss: linear regression with squared loss.
- Then identify the rule being applied: the RLM rule.
- Finally identify the regularization technique and use the complete construction to name the result.
Practice Solution
Use the stated ingredients to identify the rule.
- 1Representation
The matrix rows are instance vectors, and the target vector contains target values. Normalization is a feature manipulation applied within this setup.
- 2Starting problem
The model is linear regression with squared loss.
- 3Rule and regularization
The RLM rule is applied with Tikhonov regularization.
- 4Classification
The full construction identifies the resulting learning rule as Ridge Regression.
The answer is Ridge Regression because all of the defining construction steps occur together.
Summary
- 1Feature manipulation transforms original features into a resulting feature vector used by a learning algorithm.
- 2Its purposes can include reducing approximation or estimation errors or obtaining a faster algorithm.
- 3A transformation should be chosen in relation to the learning algorithm and the prior assumptions about the problem.
- 4Normalization is motivated through linear regression with squared loss, where matrix rows are instance vectors and a separate vector contains target values.
- 5Ridge Regression is constructed by applying the RLM rule with Tikhonov regularization to linear regression with squared loss.
At a glance
- Feature manipulation changes the representation supplied to a learner.
- Normalization is discussed as feature manipulation within a squared-loss linear regression setting.
- Ridge Regression is identified by its complete construction: linear regression, squared loss, the RLM rule, and Tikhonov regularization.
- Tikhonov regularization is a technique in the construction, while Ridge Regression is the resulting learning rule.
- The key reasoning method is to trace the process from the original regression setup to the regularized rule.