Feature Selection and Model Design

Feature selection is also a model-design decision: changing the selected features changes the size of the model's input space.

Programming

Choosing the Inputs

When designing a learning algorithm, choosing the algorithm itself is only part of the decision. You must also decide which available patient features the algorithm will use. In the heart-attack prediction example, the available features are blood pressure, body-mass index, age, level of physical activity, and income. Selecting a subset of these features determines the dimensionality of the feature space.

Key point

Feature selection is also a model-design decision: changing the selected features changes the size of the model's input space.

Two Input Paths

Worked example

Two possible heart-attack prediction designs

Compare an algorithm that uses blood pressure and body-mass index with an algorithm that uses blood pressure, body-mass index, age, physical activity, and income.

1
Select two features
The first design uses blood pressure and body-mass index. Its feature space is two-dimensional.
2
Select five features
The second design uses blood pressure, body-mass index, age, physical activity, and income. Its feature space is five-dimensional.
3
Compare the designs
The first design works in a smaller feature space. The second works in a larger feature space, so the comparison becomes a question about the advantages and demands associated with different levels of complexity.

Result

The two algorithms do not receive the same feature set: one receives two selected features, while the other receives all five listed features.

How do the same patient measurements flow into a two-feature model and a five-feature model?

The diagram shows the central design difference. Both models can be used for the same prediction task, but the two-feature algorithm receives only blood pressure and body-mass index. The five-feature algorithm receives those two measurements plus age, physical activity, and income. Because their inputs differ, their feature spaces differ as well.

Dimensionality in the Feature Space

Dimensionality here means the number of selected features used as coordinates in the feature space. Selecting blood pressure and body-mass index creates a two-dimensional feature space. Adding age, physical activity, and income creates a five-dimensional feature space. The dimensionality therefore changes directly with the number of selected features.

What changes in the model's input space when it uses two features instead of five?

Feature selection	Selected features	Feature-space dimensionality
Smaller selection	Blood pressure and body-mass index	2D
Larger selection	Blood pressure, body-mass index, age, physical activity, and income	5D

The number of selected features determines the dimensionality in this heart-attack prediction example.

Complexity and Bias

Feature selection is connected to the bias-complexity tradeoff because the selected feature set determines part of the model's complexity. The two-feature design works in a smaller feature space, while the five-feature design works in a larger feature space. Moving from two features to five features therefore changes the complexity side of the tradeoff. The design question is not whether the larger model is automatically better; it is what is gained and what becomes more demanding when the feature space grows.

How does adding or removing features change model simplicity, bias, and potential overfitting?

Practical Design Tradeoffs

Design choice	Practical advantage	Practical disadvantage
Fewer features	The model works in a smaller feature space.	The model does not use the other listed features, such as age, physical activity, and income.
More features	The model uses all five listed patient features rather than only two.	The model works in a larger feature space, making the complexity side of the design more demanding.

These are design tradeoffs in the supplied heart-attack prediction example, not a claim that one choice is always superior.

What is gained and what becomes more demanding when a model uses fewer or more patient features?

Mistakes in Model Comparison

Common mistake
Treating the number of features as unrelated to dimensionality.
What it looks like
Describing both the blood-pressure-and-BMI model and the five-feature model as having the same feature-space size.
Why it fails
The number of selected features determines the dimensionality. The first model uses a two-dimensional space, while the second uses a five-dimensional space.
Do this instead
Count the selected features before comparing the models.
Common mistake
Assuming that using more features automatically makes the model better.
What it looks like
Declaring the five-feature algorithm the winner solely because it uses all five available measurements.
Why it fails
The comparison is about the pros and cons of different complexity levels, not an automatic winner.
Do this instead
Explain both the larger feature space and the demands associated with greater complexity.
Common mistake
Discussing the prediction task without discussing feature selection.
What it looks like
Talking only about heart-attack prediction while ignoring which patient measurements enter the algorithm.
Why it fails
The central design question includes both the algorithm choice and the feature set.
Do this instead
Name the selected features and connect them to the resulting feature-space dimensionality.

Check Your Design Reasoning

EASY

A heart-attack prediction algorithm uses blood pressure, body-mass index, and age. What is the dimensionality of its feature space? Then compare it with the two-feature design and explain one way the complexity discussion changes.

Hints

Count the selected features.
The two-feature design uses blood pressure and body-mass index.
Use the language of smaller or larger feature spaces and the bias-complexity tradeoff.

Your attempt (not saved)

Key point

A complete answer should identify the three-dimensional feature space, compare it with the two-dimensional space, and explain that changing the feature count changes the complexity side of the bias-complexity tradeoff.

Feature Selection Summary

1
Selecting features determines the dimensionality of the feature space.
2
Using blood pressure and body-mass index creates a two-dimensional feature space.
3
Using blood pressure, body-mass index, age, physical activity, and income creates a five-dimensional feature space.
4
Fewer features create a smaller feature space, while more features create a larger feature space and a more demanding complexity comparison.
5
Feature selection is part of the bias-complexity tradeoff, so models should be compared by explaining their respective advantages and disadvantages rather than naming an automatic winner.

At a glance

The selected feature set determines the dimensionality of a learning algorithm's feature space.
The blood-pressure-and-BMI design uses two dimensions; the design using all five listed patient features uses five dimensions.
Fewer features produce a smaller feature space but leave out some available measurements.
More features include more of the listed measurements but create a larger and more demanding feature space.
Feature selection should be analyzed as part of the bias-complexity tradeoff, not as a decision with an automatic winner.

Related concepts

More concepts starting with F