Concepts / Feature Manipulation and Normalization

Feature Manipulation and Normalization

Feature transformations should be selected according to the property that needs to change.

  • Programming
Interactive lab

Try it: Feature Scaling: Min-Max and Standardization

How centering, min-max scaling and standardization change each feature's location and range, and why features on very different scales let one feature dominate Euclidean distances — so the nearest neighbour can change once the features are scaled. The scaler's statistics come from the training data and are then applied unchanged to new data.

How it works

  1. Fit the scaler per feature on the training points: mean (centering), min and max (min-max), or mean and standard deviation (standardization).
  2. Transform every training point with that feature's formula: x − mean, (x − min)/(max − min) (optionally mapped to [−1, 1]), or (x − mean)/std.
  3. Apply the same fitted transform to the test point — it may land outside [0, 1]; the scaler is not refitted on new data.
  4. Measure the Euclidean distance from the test point to every training point in the transformed space and rank them.
  5. Compare with the raw ranking: without scaling, the feature with the larger numbers dominates; centering alone changes nothing.

Default run (11 steps): 6 training points and one test point (100 sq m, 6 rooms). Area spans 40–180, rooms 1.5–7. Method: standardization, fitted on the training points only. … Nearest neighbour after standardization: p_2 (class B); ranking p_2 < p_4 < p_3 < p_6 < p_1 < p_5. With raw values it would be p_1 (class A) — the large-scale area feature dominated.

Simplified: Two features (area 20–200 sq m, rooms 1–8) and at most 8 training points on a grid; the effect is shown on a 1-nearest-neighbour lookup only. Standard deviation is the population value (ddof = 0) and a zero range or zero spread is replaced by 1, as scikit-learn does.

Educational simulation

Loading the simulation…

Choosing the Property to Change

A feature is not always presented in the most useful numerical form. Its values may be shifted away from zero, cover an inconvenient range, contain unusually high or low values, or represent counts whose differences are not equally meaningful. Feature manipulation and normalization provide several ways to change that representation. The central rule is simple: identify the property that needs to change first, then select the transformation that addresses it.

Key point

Do not choose a transformation merely because it is familiar. Choose it according to whether the problem concerns location, spread, extreme values, or the meaning of differences across a count scale.

Tracing Location and Spread

Centering, scaling, and standardization all reorganize the numerical position of a feature, but they do different jobs. Centering changes location by subtracting the empirical mean from every value. Scaling changes the range, commonly placing values between 0 and 1 or between −1 and 1. Standardization combines a location adjustment with a variance adjustment: it subtracts the empirical mean and divides by the standard deviation, producing a feature with zero mean and unit variance.

Raw featureoriginal location and rangeCentered featuremean adjusted to zeroScaled featurerange commonly 0 to 1 or −1to 1Standardized featurezero mean and unit variancesubtract meanadjust rangeadjust mean and variance
What happens to a feature's values, center, and spread when it is centered, scaled, or standardized?
TransformationPrimary property changedTypical result
CenteringLocationThe empirical mean becomes zero
ScalingRangeValues are commonly placed between 0 and 1 or between −1 and 1
StandardizationLocation and varianceThe feature has zero mean and unit variance

The different jobs performed by location-and-spread transformations

A Small Feature Trace

Worked example

Selecting a location or spread adjustment

Suppose a generated feature contains values whose average is far from zero, and a separate feature contains values across an inconvenient numerical range. Which transformation matches each problem?

  1. 1
    Inspect the first feature

    The problem is its location: the values are shifted away from zero. Centering is appropriate because it subtracts the empirical mean from every value.

  2. 2
    Inspect the second feature

    The problem is its range. Scaling is appropriate because it changes the range, commonly placing values between 0 and 1 or between −1 and 1.

  3. 3
    Consider both properties

    If both the location and the variance need adjustment, standardization combines subtraction of the empirical mean with division by the standard deviation.

Result

Use centering for a location problem, scaling for a range problem, and standardization when both mean and variance need adjustment.

Location and spreadcenter, scale, standardizeCentermeanValue reshapingclip, sigmoid, logarithmRangescaleExtreme influenceclip or sigmoidCount differenceslogarithmrepositionsrescaleschanges influencecompresses large counts
What is the difference between moving or rescaling a distribution and changing its shape by clipping, applying a sigmoid, or taking a logarithm?

Reshaping Extreme and Count Values

Clipping, sigmoidal transformation, and logarithmic transformation do not primarily reposition a feature around its mean or place it inside a fixed range. They alter how values and differences are represented, especially when extreme values or unevenly meaningful count differences are a concern.

Input valuesordinary and extreme valuesClipped valueslimited to a specifiedrangeSigmoidal valuessoft treatment of extremesLogarithmic valueslarge counts compressedlimit high or low valuessoften extreme influencecompress large-count differences
How do clipping, sigmoid, and logarithmic transformations change individual input values across their ranges?
Definition

Clipping limits high or low feature values to a specified range. It directly prevents values beyond that range from remaining as large or small as they originally were.

Definition

A sigmoidal transformation is a softer alternative for extreme values. Values close to zero are affected only slightly, while values far from zero behave similarly to clipping.

Definition

A logarithmic transformation is especially useful for count features when equal numerical gaps do not carry equal meaning. It compresses the large-count end relative to the small-count end.

Count Features in Context

Worked example

Why a count may need reshaping

Consider a generated word-count feature. Compare the meaning of moving from zero occurrences to one occurrence with moving from 1000 occurrences to 1001 occurrences.

  1. 1
    Compare the numerical gaps

    Both changes increase the count by one, but the first change can matter much more than the second.

  2. 2
    Identify the mismatch

    Equal numerical gaps do not necessarily represent equal importance across the count scale.

  3. 3
    Select a transformation

    A logarithmic transformation is useful because it compresses the large-count end relative to the small-count end.

Result

For a count feature with unevenly meaningful differences, consider a logarithmic transformation rather than treating every numerical gap as equally important.

What do you think happens?

A feature contains a few unusually high values, and you want a direct limit on how high or low values may be. Which transformation best matches that goal?

  • Centering

  • Clipping

  • Logarithmic transformation

  • Standardization

Reveal answer
Answer:

Clipping

Clipping limits high or low feature values to a specified range. A sigmoidal transformation would be a softer alternative when extreme values should be treated less strongly rather than directly limited.

Matching Problems to Transformations

Observed problemwhat needs to change?Shifted locationmean away from zeroCenteringInconvenient rangevalues need rescalingScalingMean and varianceboth need adjustmentStandardizationExtreme valuesinfluence needs changingClippingUneven countimportancelarge gaps less meaningfulSigmoidaltransformationLogarithmictransformationlocationrangemean and varianceextreme influencecount differencessubtract meanadjust rangeadjust mean and variancedirect limitsofter treatmentcompress large counts
How does the observed problem with a feature map to an appropriate transformation?
  • Use centering when the feature's location, represented by its empirical mean, needs adjustment.
  • Use scaling when the feature's range is inconvenient.
  • Use standardization when both the empirical mean and variance need adjustment.
  • Use clipping when high or low values should be limited to a specified range.
  • Use a sigmoidal transformation when extreme values need a softer treatment: values near zero should change little, while far-from-zero values should behave similarly to clipping.
  • Use a logarithmic transformation for count features when large-count differences should be compressed relative to small-count differences.
Raw featureobserved valuesDiagnose behaviorlocation, range, extremes,or countsCenter or scalereposition or rescaleCombinedtransformationwhen more than one propertyneeds changeFinal featureselected representationClip, sigmoid, or logreshape value influenceinspectlocation or rangeextremes or countsoptional combinationoptional combinationapply selected operations
How does a raw feature move through a selected transformation to become the final feature?

Common Selection Mistakes

  • Common mistake

    Treating centering, scaling, and standardization as interchangeable.

    What it looks like

    Choosing scaling when the real issue is that the empirical mean is shifted away from zero.

    Why it fails

    Centering adjusts location, scaling adjusts range, and standardization adjusts both mean and variance.

    Do this instead

    Name the property that needs to change before choosing the operation.

  • Common mistake

    Using a location-and-spread adjustment to solve a value-meaning problem.

    What it looks like

    Standardizing a count feature when large and small count differences have unequal importance.

    Why it fails

    Standardization changes mean and variance, while a logarithmic transformation reshapes the count scale by compressing the large-count end.

    Do this instead

    Use a logarithmic transformation when the issue is the relative meaning of differences across a count range.

  • Common mistake

    Assuming all extreme-value transformations behave in the same way.

    What it looks like

    Describing a sigmoidal transformation as an exact hard limit.

    Why it fails

    Clipping directly limits high or low values to a specified range, whereas a sigmoid is a softer alternative: values near zero change little and far-from-zero values behave similarly to clipping.

    Do this instead

    Choose clipping for a direct limit and a sigmoid for softer treatment of extremes.

  • Common mistake

    Applying a transformation without diagnosing the feature.

    Why it fails

    Different transformations address different properties, and transformations can also be combined when more than one property needs adjustment.

    Do this instead

    Inspect whether the issue concerns location, range, variance, extreme influence, or unevenly meaningful count differences.

Practice the Selection Rule

MEDIUM

For each generated situation, select the transformation that best matches the property needing change: a feature shifted away from zero; a feature covering an inconvenient range; a feature requiring both mean and variance adjustment; a feature with unusually high and low values that must be limited; a feature with extreme values that should be softened rather than directly limited; and a count feature where large-count differences matter less than small-count differences.

Hints
  • Separate location and range problems from problems involving extreme-value influence.
  • For the two extreme-value cases, distinguish a direct limit from a softer response.
  • For the count case, consider whether equal numerical gaps have equal meaning.
  1. 1
    Feature transformation is a property-matching task. Centering changes location by subtracting the empirical mean. Scaling changes the range. Standardization adjusts both mean and variance. Clipping directly limits extreme values, while a sigmoidal transformation treats extremes more softly. A logarithmic transformation is useful for count features when the large-count end should be compressed relative to the small-count end. Transformations may be combined, but the desired change should be identified first.

Key Takeaways

  • Choose a feature transformation according to the property that needs to change.
  • Centering adjusts location, scaling adjusts range, and standardization adjusts both mean and variance.
  • Clipping and sigmoidal transformation address extreme values differently: clipping imposes a direct limit, while a sigmoid provides a softer response.
  • Logarithmic transformation compresses the large-count end when count differences are not equally meaningful across the scale.
  • Transformations can be combined when more than one property needs adjustment.

Related Topics

Related concepts

More concepts starting with F