Preprocessing Data for Neural Networks
A neural network cannot learn directly from raw sound, images, text, or an unprepared table.
Try it: Feature Scaling: Min-Max and Standardization
How centering, min-max scaling and standardization change each feature's location and range, and why features on very different scales let one feature dominate Euclidean distances — so the nearest neighbour can change once the features are scaled. The scaler's statistics come from the training data and are then applied unchanged to new data.
How it works
- Fit the scaler per feature on the training points: mean (centering), min and max (min-max), or mean and standard deviation (standardization).
- Transform every training point with that feature's formula: x − mean, (x − min)/(max − min) (optionally mapped to [−1, 1]), or (x − mean)/std.
- Apply the same fitted transform to the test point — it may land outside [0, 1]; the scaler is not refitted on new data.
- Measure the Euclidean distance from the test point to every training point in the transformed space and rank them.
- Compare with the raw ranking: without scaling, the feature with the larger numbers dominates; centering alone changes nothing.
Default run (11 steps): 6 training points and one test point (100 sq m, 6 rooms). Area spans 40–180, rooms 1.5–7. Method: standardization, fitted on the training points only. … Nearest neighbour after standardization: p_2 (class B); ranking p_2 < p_4 < p_3 < p_6 < p_1 < p_5. With raw values it would be p_1 (class A) — the large-scale area feature dominated.
Simplified: Two features (area 20–200 sq m, rooms 1–8) and at most 8 training points on a grid; the effect is shown on a 1-nearest-neighbour lookup only. Standard deviation is the population value (ddof = 0) and a zero range or zero spread is replaced by 1, as scikit-learn does.
Loading the simulation…
From Raw Data to Usable Input
A neural network cannot learn directly from raw sound, images, text, or an unprepared table. Preprocessing changes raw data into numerical forms that a neural network can accept and learn from. The main preparation steps in this topic are vectorization, normalization, reshaping, and handling missing values.
Think of preprocessing as preparing the representation of information, not changing the underlying information itself. A useful representation has a suitable tensor structure, numerical values that are easier for the network to use, and a clear treatment of missing entries.
Reshaping Without Losing Data
Reshaping reorganizes a tensor's rows and columns into a target shape. The target shape is the new arrangement requested by the reshape operation. Reshaping changes the tensor's structure, but it does not change the total number of coefficients stored in the tensor. It is therefore best understood as a change in arrangement rather than a change in content.
tensor.reshape((new_shape))Rearranging a six-coefficient tensor
A tensor has shape 2 by 3 and contains the coefficients 1, 2, 3, 4, 5, and 6. Consider reshaping it to a target shape of 3 by 2.
- 1Read the original shape
The original arrangement has 2 rows and 3 columns, so it contains 2 multiplied by 3, or 6, coefficients.
- 2Read the target shape
The requested arrangement has 3 rows and 2 columns, so it also contains 3 multiplied by 2, or 6, coefficients.
- 3Compare the totals
Because both shapes account for six coefficients, the target shape preserves the tensor's total amount of data.
- 4Interpret the result
The rows and columns are reorganized. The coefficients remain the same, so the operation changes arrangement rather than content.
A 2 by 3 tensor can be reshaped into a 3 by 2 tensor because both shapes contain six coefficients.
Checking the Target Shape
Before accepting a proposed target shape, multiply its dimensions and compare that result with the number of coefficients in the original tensor. The target arrangement is valid only when it accounts for the same total number of coefficients. Different rows and columns are expected; a different total is the problem.
Vectorizing Data for the Network
Vectorization produces tensors from sources such as text, images, and tabular data. It changes raw information into numerical structure that can be supplied to a neural network. In a digit-image example, reshaping is part of preprocessing before the data enters the network. The important question is what the requested dimensions represent: one dimension can represent the collection of images, while another represents the contents of one image after rearrangement.
Neural-network inputs and targets must be represented as tensors because preprocessing and network operations work with tensor structure. The input tensor carries the prepared examples, while the target tensor carries the corresponding outputs used during learning. Their shapes must represent the same collection of examples and their associated target values.
Keeping Numerical Values Usable
Normalization keeps values small and helps avoid problems caused by incompatible feature ranges. Neural-network data should usually contain small, relatively homogeneous values so that one feature's scale does not make the other features difficult to use.
| Preparation method | Main idea | What to remember |
|---|---|---|
| Scaling | Place values into a small range | The focus is on keeping numerical values small and making feature ranges more compatible |
| Standardizing | Transform each feature to a mean of 0 and a standard deviation of 1 | The focus is on the distribution of each feature |
Scaling and standardizing both prepare numerical data, but they describe different transformations.
Handling Missing Values
A dataset may lack a value for a feature in some training or test records. One safe approach is to represent a missing value as 0 only when 0 does not already have a meaningful interpretation for that feature. If 0 is meaningful, using it as the missing marker would make two different situations look identical: a genuine zero and an absent value.
Common Preprocessing Mistakes
Common mistake Assuming that a different shape means the tensor contains different data
What it looks likeTreating a 2 by 3 tensor and a 3 by 2 tensor as if one must contain more coefficients
Why it failsReshaping reorganizes rows and columns while preserving the total number of coefficients when the target shape is valid.
Do this insteadCompare the products of the dimensions before interpreting the visual difference.
Common mistake Accepting a target shape without checking its coefficient count
What it looks likeTrying to reshape a tensor with six coefficients into a 2 by 2 arrangement
Why it failsThe proposed target accounts for only four coefficients.
Do this insteadMultiply the target dimensions and verify that the result equals the original coefficient count.
Common mistake Sending raw data directly to the network
What it looks likeTreating an image, text source, or unprepared table as ready for neural-network processing
Why it failsRaw data must first be changed into numerical tensor forms that the network can accept and learn from.
Do this insteadVectorize the data and prepare its tensor structure before it enters the network.
Common mistake Using zero as a missing marker when zero is meaningful
What it looks likeRepresenting both a genuine zero and a missing feature value with 0
Why it failsThe network cannot distinguish the two situations from their representation.
Do this insteadUse a representation that distinguishes missing entries from genuine zero values.
Common mistake Treating scaling and standardizing as identical descriptions
What it looks likeDescribing both operations only as making numbers smaller
Why it failsScaling places values into a small range, while standardizing gives each feature a mean of 0 and a standard deviation of 1.
Do this insteadName the preparation goal precisely.
Practice the Preparation Check
A tensor has shape 4 by 5. A proposed preprocessing step asks for a target shape of 2 by 10. Decide whether the reshape preserves the total number of coefficients. Then identify whether the operation changes the tensor's content or only its arrangement. Finally, state one question you would ask before representing a missing value as 0.
Hints
- Multiply the original dimensions and then multiply the target dimensions.
- Reshaping is described as a change in arrangement rather than a change in content.
- Ask whether 0 already has a meaningful interpretation for the feature.
Practice solution
Compare the original shape 4 by 5 with the proposed target shape 2 by 10.
- 1Count the original coefficients
The original shape contains 4 multiplied by 5, which is 20 coefficients.
- 2Count the target coefficients
The target shape contains 2 multiplied by 10, which is also 20 coefficients.
- 3Interpret the reshape
Because the totals match, the target shape preserves the tensor's total number of coefficients. The rows and columns are reorganized rather than the content being changed.
- 4Check the missing marker
Before using 0 for a missing value, check whether 0 is already a meaningful value for that feature.
The reshape is valid under the coefficient-preservation rule, and 0 is safe as a missing marker only when it is not already meaningful for the feature.
Preparation Checklist
- 1Convert raw sources such as text, images, and tables into numerical tensor forms through vectorization.
- 2Choose a tensor arrangement that suits the neural network, using reshaping when the target shape preserves the total number of coefficients.
- 3Keep numerical values small and relatively homogeneous through appropriate normalization, scaling, or standardizing.
- 4Represent inputs and targets as tensors whose shapes describe corresponding examples and outputs.
- 5Handle missing values consistently, using 0 as a missing marker only when 0 has no meaningful interpretation for that feature.
Key Takeaways
- Reshaping reorganizes rows and columns into a target shape while preserving the total number of coefficients.
- A proposed target shape must account for exactly the same total number of coefficients as the original tensor.
- Vectorization changes raw text, images, and tabular data into numerical tensor forms that a neural network can use.
- Scaling keeps values small, while standardizing gives each feature a mean of 0 and a standard deviation of 1.
- Use 0 for missing values only when 0 is not already a meaningful value for that feature.