Polynomial Regression
Polynomial regression illustrates the general pattern of constructing features first and training a linear predictor on top of them.
Try it: Polynomial Fit: Overfitting and Ridge
How raising the polynomial degree drives training error down while validation error eventually rises (overfitting), and how a ridge penalty λ‖w‖² tames a high-degree fit.
How it works
- Split the noisy points into a training set and a validation set.
- Build the design matrix with columns (, a rescaled ).
- Solve (XᵀX + λI)w = Xᵀy on the training points by Gaussian elimination with partial pivoting.
- Measure the mean squared error on the training points and, separately, on the validation points.
- Repeat for every degree 0–9: the degree with the lowest validation error is the one validation selects.
Default run (18 steps): 14 noisy points: 10 for training, 4 held out for validation. Fit a degree-3 polynomial with λ = 0. … Degree 3: training MSE 0.0282, validation MSE 0.0308. The lowest validation error is at degree 4. Training error keeps falling with degree; validation error is what reveals overfitting.
Simplified: One input, 14 fixed noisy samples of sin(2πx) and a single train/validation split (no cross-validation). The penalty λ‖w‖² here includes the constant term and is added to the sum (not the mean) of squared errors.
Loading the simulation…
From an Instance to a Representation
A machine learning method cannot work directly with an unspecified instance space. It needs each instance to be expressed in a usable feature representation. Feature learning addresses this preparation step by learning how an instance should be represented instead of requiring the representation to be completely designed in advance.
Feature learning learns a function ψ that maps instances from an instance space X into d-dimensional feature vectors.
The important change occurs at the representation level. Before the mapping, an item is described only as an element of X. After the mapping, that same item is available as a vector with d coordinates. The function ψ creates the description that a later learning method can use.
Polynomial Terms as Constructed Features
Polynomial regression illustrates the general pattern of constructing features first and training a linear predictor on top of them. For an input x, the constructed feature vector can contain the monomial terms 1, x, x², through xᵈ. The representation is created before the later predictor operates on it.
Constructing a Degree-3 Representation
Represent the input x = 2 using the monomial construction through degree 3.
- 1Start with the input
The instance supplied to the representation step is x = 2.
- 2Construct the terms
The terms through degree 3 are 1, x, x², and x³.
- 3Evaluate the terms
For x = 2, these terms are 1, 2, 4, and 8.
- 4Separate representation from prediction
The resulting vector is the constructed representation. A later linear predictor is a separate part of the learning system.
The constructed feature vector is [1, 2, 4, 8].
This example separates two roles that are easy to confuse. The monomial construction defines a representation, while the linear predictor operates after that representation has been created. Feature learning concerns the transformation into the representation; the later predictor is a separate part of the learning system.
Degree and Feature-Space Size
The degree controls which monomial terms are included in the constructed representation. Through degree d, the vector is described as [1, x, x², ..., xᵈ]. Increasing the degree therefore adds higher-order constructed terms to the representation. The meaning of the feature space changes because the later predictor receives a richer collection of coordinates.
Learning versus Choosing Features
Feature selection and feature transformation begin with a predefined feature space, described in the source as Rᵈ. Feature selection chooses some of the available features. Feature transformation changes individual features. Feature learning starts one step earlier: it learns a function that creates the representation from instances in X.
| Approach | Starting point | Main operation |
|---|---|---|
| Feature learning | Instances in X | Learn a function that creates the representation |
| Feature selection | Predefined feature space Rᵈ | Choose some available features |
| Feature transformation | Predefined feature space Rᵈ | Change individual features |
The distinction is whether the representation is learned from instances or is already predefined before selection or transformation.
Why Prior Knowledge Matters
The No-Free-Lunch theorem implies that prior knowledge about the data distribution is necessary for building a good feature representation. A representation is not useful merely because it is automated. Its usefulness is connected to whether it produces a suitable hypothesis class for the task and the distribution of the data.
Common Conceptual Mistakes
Common mistake Calling polynomial regression only a linear-predictor procedure
What it looks likeTreating the later linear predictor as if it were the entire method
Why it failsPolynomial regression illustrates a two-part pattern: first construct features, then train a linear predictor on those features.
Do this insteadSeparate the monomial construction from the predictor that operates after the representation exists.
Common mistake Treating feature learning as ordinary feature selection
What it looks likeAssuming that learning ψ simply chooses some coordinates from a predefined Rᵈ
Why it failsFeature selection begins with a predefined feature space, whereas feature learning starts with instances in X and learns how to create their representation.
Do this insteadAsk whether the representation already exists before the method acts. If it does not, the task is at the feature-learning level.
Common mistake Assuming automation eliminates the need for prior knowledge
What it looks likeBelieving that any learned representation must be suitable because it was learned automatically
Why it failsThe No-Free-Lunch theorem implies that prior knowledge about the data distribution is necessary for building a good feature representation.
Do this insteadConnect the representation to the data distribution and the hypothesis class needed for the task.
Check Your Understanding
An instance belongs to X, and a method learns a function ψ that maps it into a vector with d coordinates. Is this feature selection, feature transformation, or feature learning? Explain what happens before and after the mapping.
Hints
- Identify whether the feature space was predefined before the method acted.
- Use the distinction between an instance in X and a vector in a predefined Rᵈ.
For x = 3, write the constructed monomial feature vector through degree 2. Then state which part of the polynomial-regression pattern is represented by that vector and which part is still separate.
Hints
- List the terms 1, x, and x².
- The vector is the representation; the later linear predictor is separate.
What do you think happens?
If the polynomial degree increases from 2 to 3, what changes in the constructed representation?
Reveal answer
A higher-order term is added.
The representation through degree 2 contains 1, x, and x². Through degree 3, it additionally contains x³.
Key Takeaways
- 1Feature learning learns a function ψ that maps instances from X into d-dimensional feature vectors.
- 2Polynomial regression illustrates feature construction followed by a linear predictor.
- 3The monomial construction creates the representation; the later predictor is a separate system component.
- 4Feature selection and feature transformation begin with a predefined feature space, while feature learning starts by learning how instances should be represented.
- 5Prior knowledge about the data distribution is necessary for constructing a good feature representation.
At a glance
- Feature learning maps an instance from X into a d-dimensional feature vector through a learned function ψ.
- Polynomial regression demonstrates how constructed features such as 1, x, x², through xᵈ can be created before prediction.
- The feature-construction step and the later linear predictor play different roles.
- Feature selection and transformation operate on a predefined feature space, whereas feature learning learns the representation itself.
- Prior knowledge about the data distribution remains necessary for choosing or learning a useful representation.