Mastering Feature Representation for Machine Learning
Feature transformations should be selected according to the property that needs to change.
Try it: Feature Scaling: Min-Max and Standardization
How centering, min-max scaling and standardization change each feature's location and range, and why features on very different scales let one feature dominate Euclidean distances — so the nearest neighbour can change once the features are scaled. The scaler's statistics come from the training data and are then applied unchanged to new data.
How it works
- Fit the scaler per feature on the training points: mean (centering), min and max (min-max), or mean and standard deviation (standardization).
- Transform every training point with that feature's formula: x − mean, (x − min)/(max − min) (optionally mapped to [−1, 1]), or (x − mean)/std.
- Apply the same fitted transform to the test point — it may land outside [0, 1]; the scaler is not refitted on new data.
- Measure the Euclidean distance from the test point to every training point in the transformed space and rank them.
- Compare with the raw ranking: without scaling, the feature with the larger numbers dominates; centering alone changes nothing.
Default run (11 steps): 6 training points and one test point (100 sq m, 6 rooms). Area spans 40–180, rooms 1.5–7. Method: standardization, fitted on the training points only. … Nearest neighbour after standardization: p_2 (class B); ranking p_2 < p_4 < p_3 < p_6 < p_1 < p_5. With raw values it would be p_1 (class A) — the large-scale area feature dominated.
Simplified: Two features (area 20–200 sq m, rooms 1–8) and at most 8 training points on a grid; the effect is shown on a 1-nearest-neighbour lookup only. Standard deviation is the population value (ddof = 0) and a zero range or zero spread is replaced by 1, as scikit-learn does.
Loading the simulation…
Why Representation Matters
A feature is not always presented in the most useful numerical form. Its values may be shifted away from zero, cover an inconvenient range, contain unusually high or low values, or represent counts whose differences are not equally meaningful. Feature transformations change that representation. The key decision is not to transform automatically, but to identify which property needs to change first.
Following Location and Spread
Centering, scaling, and standardization reorganize the numerical position of a feature. They do not primarily change which observations are larger or smaller than one another. Instead, they alter where the values sit, how wide their range is, or how much variation the feature has around its center.
| Transformation | What changes | Source description |
|---|---|---|
| Centering | Feature location | Subtracts the empirical mean from every value |
| Scaling | Feature range | Commonly places values between 0 and 1 or between −1 and 1 |
| Standardization | Feature mean and variance | Subtracts the empirical mean and divides by the standard deviation |
The three location-and-spread transformations perform different jobs.
Choosing among centering, scaling, and standardization
A feature has values that are far from zero, span an inconvenient range, and vary by different amounts around their center. Which transformation matches each problem?
- 1Values are shifted away from zero
Centering is the direct choice because it subtracts the empirical mean from every value and changes the feature's location.
- 2The range is inconvenient
Scaling is the direct choice because it adjusts the feature's range, commonly to 0 through 1 or to −1 through 1.
- 3Both location and variance need adjustment
Standardization combines a mean adjustment with division by the standard deviation, producing zero mean and unit variance.
Identify the property first: location suggests centering, range suggests scaling, and both mean and variance suggest standardization.
Reshaping Value Differences
Clipping, sigmoidal transformation, and logarithmic transformation address the influence or meaning of values rather than simply repositioning a feature around a mean. These operations reshape how differences are expressed. Two values can remain ordered in the same general direction while their numerical separation and influence change.
Clipping limits high or low values to a specified range. It directly prevents values from extending beyond those limits. A sigmoidal transformation is softer: values close to zero are affected only slightly, while values far from zero behave similarly to clipping. A logarithmic transformation is especially useful for count features because it compresses the large-count end relative to the small-count end.
Handling Extreme and Skewed Values
Selecting a reshaping operation
Three features have different problems: one contains unusually high and low values, one has extreme values that should be softened rather than abruptly limited, and one records counts where small and large differences do not have equal importance.
- 1Unusually high or low values must stay within explicit limits
Use clipping because it limits high or low feature values to a specified range.
- 2Extreme values need a softer treatment
Use a sigmoidal transformation because values near zero are affected only slightly while far-from-zero values behave similarly to clipping.
- 3The feature represents counts
Use a logarithmic transformation when small and large count differences have unequal importance; it compresses the large-count end relative to the small-count end.
The observed behavior determines the choice: hard limits suggest clipping, softer extreme compression suggests sigmoid, and unequal importance across a count scale suggests logarithm.
Choosing by Feature Behavior
This selection process can also lead to a combination. For example, centering and scaling may be used together when both the feature's location and range need adjustment. The important discipline is to identify the desired property first and then choose the operation or combination that addresses it.
Common mistake Treating every transformation as if it did the same job
What it looks likeUsing scaling when the actual issue is that unusually high values need to be limited
Why it failsScaling adjusts range, while clipping addresses high or low values by limiting them to a specified range.
Do this insteadName the behavior that needs to change before selecting the transformation.
Common mistake Confusing centering with standardization
What it looks likeAssuming that subtracting the empirical mean also creates unit variance
Why it failsCentering changes location only. Standardization subtracts the empirical mean and divides by the standard deviation.
Do this insteadUse standardization when both mean and variance need adjustment.
Common mistake Using clipping whenever extreme values appear
What it looks likeApplying hard limits when a softer treatment of far-from-zero values is desired
Why it failsA sigmoidal transformation provides a softer alternative: near-zero values are affected only slightly and extreme values are compressed.
Do this insteadChoose clipping for specified limits and sigmoid for softer extreme-value behavior.
Common mistake Treating all count differences as equally meaningful
What it looks likeViewing the difference between zero and one occurrence as equivalent to the difference between 1000 and 1001 occurrences
Why it failsFor count features, small and large differences may have unequal importance.
Do this insteadConsider a logarithmic transformation when the large-count end should be compressed relative to the small-count end.
Practice the Selection
For each situation, choose the most appropriate transformation and explain which feature property it changes: a feature is shifted away from zero; a feature covers an inconvenient range; a feature needs both its mean and variance adjusted; high and low values must be limited; extreme values should be compressed softly; or a count feature has unequal meaning across small and large values.
Hints
- Centering addresses location.
- Scaling addresses range.
- Standardization addresses mean and variance together.
- Clipping uses specified limits, while sigmoid provides a softer treatment of extremes.
- Logarithm compresses the large-count end relative to the small-count end.
What do you think happens?
A feature records word counts. Moving from zero occurrences to one occurrence is more meaningful than moving from 1000 occurrences to 1001 occurrences. Which transformation best matches this behavior?
Reveal answer
Logarithmic transformation
A logarithmic transformation compresses the large-count end relative to the small-count end, matching a count feature whose differences do not have equal importance across the scale.
Practical Takeaways
- 1Centering changes a feature's location by subtracting its empirical mean from every value.
- 2Scaling changes a feature's range, commonly placing it between 0 and 1 or between −1 and 1.
- 3Standardization adjusts both mean and variance by subtracting the empirical mean and dividing by the standard deviation.
- 4Clipping and sigmoid both address extreme values, but clipping imposes specified limits while sigmoid provides softer compression.
- 5Logarithmic transformation is useful for count features when small and large differences have unequal importance.
At a glance
- Select a feature transformation according to the property that needs to change.
- Use centering for location, scaling for range, and standardization for both mean and variance.
- Use clipping for explicit limits and sigmoid for softer treatment of extreme values.
- Use logarithmic transformation when a count feature gives unequal importance to small and large differences.
- Transformations can be combined, but the desired change should be identified first.