Predictive Models and Quality Measures

Suppose a dataset contains many candidate features, but a later predictive model should use only a smaller set. A filter method addresses this by examining each feature on its own, assigning the feature a quality score, and then using the scores to choose candidates. The important idea is that assessment happens before the features are combined in a larger predictive model.

A filter method separates two decisions. First, it assesses each feature independently. Second, it compares the resulting scores and selects features. This turns one large feature-selection question into a collection of single-feature quality judgments.

To score a particular feature, the filter method takes the values of that feature across the training examples and pairs them with the matching target values. The feature under assessment is the only input considered by the predictor used for that assessment.

A filter method does not prescribe one universal way to judge a feature. The quality measure supplies that judgment, and many quality measures are possible. One direct approach is predictive: train a predictor using only the feature under assessment, then use the predictor's error as the feature's score.

For the regression illustration, the feature values and their matching target values define the single-feature evaluation. The predictor is not trained from the whole candidate feature set for this step. It is trained from the one feature whose quality is being measured.

The source illustrates the error-based approach with linear regression and squared loss. For one feature, the predictor produces a prediction for each training example. Each prediction is compared with that example's actual target, the difference is squared, and the squared differences are averaged across the training examples. This average is the empirical squared loss for that single-feature assessment.

Squaring makes the score reflect the size of prediction errors without allowing positive and negative differences to cancel each other. In this loss-based scoring convention, a smaller average squared loss means the one-feature predictor made smaller errors on the assessed training examples, so the feature receives the more favorable loss score.

After every candidate feature has been assessed independently, the filter method has one score per feature. Selection happens at this later stage. The scores are compared, often by ranking them or applying a rule for retaining candidates, and the strongest candidates are kept according to the chosen quality measure.