Text Data Preprocessing
The important boundary is between human-readable text and numeric model input. Tokenization identifies the sequence units; vectorization gives those units numeric representations; the resulting vectors are packed into sequence tensors.
Try it: Text Vectorization: Tokens to Vectors
The text-to-tensor pipeline: standardize, split into tokens, build a frequency-ranked vocabulary with padding and [UNK], map tokens to integer ids, pad or truncate, then one-hot or multi-hot encode and look the ids up in an embedding table.
How it works
- Standardize: lowercase and delete punctuation.
- Split on whitespace into tokens.
- Build the vocabulary: 0 = padding, 1 = [UNK], then words by frequency (ties: first appearance), capped at max tokens.
- Map each token to its id (1 if it did not make the vocabulary), then pad with 0 or truncate to the sequence length.
- Encode: one-hot rows per position or one multi-hot row per text; embed: id k selects row k of the embedding table.
Default run (18 steps): Raw text (59 characters): "The movie was great. The plot was great, the acting wasn't!". A model cannot read strings, so it is turned into numbers step by step. … Embedding lookup: id k selects row k of a fixed 8 x 3 table, giving a (8, 3) dense matrix instead of (8, 8) sparse one-hots. Row 0 (padding) is looked up too; a trained Embedding layer would learn these rows.
Simplified: Bounded ASCII text (up to 200 characters, 40 tokens), whitespace splitting only, and a small FIXED embedding table (a real Embedding layer learns its rows). The multi-hot vector keeps the padding slot (always 0) so its positions line up with the ids.
Loading the simulation…
From Words to Model Input
A neural network does not receive raw text directly. Before a model can process a sentence, the text must be represented as numeric data. This gives sequence processing two broad stages: create a numerical representation of the sequence, then apply a model designed to process ordered data.
The important boundary is between human-readable text and numeric model input. Tokenization identifies the sequence units; vectorization gives those units numeric representations; the resulting vectors are packed into sequence tensors.
Token Units and Numeric Values
Tokenization is the step that chooses and identifies the units in a sequence. For text, those units may be words, subwords, or characters. Tokenization does not by itself create the numeric vectors needed by a neural network.
Text vectorization is the step that supplies each generated token with a numeric vector. Those vectors can then be packed into sequence tensors and passed to a deep neural network.
A Three-Token Sequence
Illustrate the difference between tokenization and vectorization for the generated phrase "Text data flows".
- 1Tokenize
Treat the phrase as three word tokens: "Text", "data", and "flows". Tokenization has identified the ordered units, but the result is still text.
- 2Vectorize
Associate each token with a numeric vector. The particular numeric values are implementation choices in this illustration; the important point is that every token receives a numeric representation.
- 3Pack
Place the vectors in their sequence order to form a sequence tensor that can be supplied to a neural network.
Tokenization determines what the units are. Vectorization turns those units into numeric representations. Packing preserves their order in the model input.
Ways to Represent Tokens
Two major ways to associate vectors with tokens are one-hot encoding and token embedding. The source material presents token embedding as typically being used for words, where it is also called word embedding. Both belong to the vectorization stage because they provide numeric representations for the generated tokens.
| Representation approach | Role in preprocessing | Source terminology |
|---|---|---|
| One-hot encoding | Associates a numeric representation with each token | One of the two major approaches |
| Token embedding | Associates a vector with each token | Typically used for words; also called word embedding |
Both approaches provide numeric representations after tokenization.
When describing a text pipeline, state both decisions explicitly: which units are tokens and which numeric representation is assigned to each token. This makes it clear how raw text becomes a sequence tensor.
Sliding Windows over Sequences
Once a sequence has become numeric input, a one-dimensional convnet applies the one-dimensional convolutional approach to sequence processing. Its broad role is to examine local regions of an ordered sequence as a sliding operation. This makes it suitable for detecting patterns that appear within nearby sequence positions.
The sequence remains ordered while the local region being examined moves across it. The central idea is not simply that the input is numeric; it is that the numeric input still represents an ordered sequence whose local structure can be examined.
Convolutional and Recurrent Roles
One-dimensional convnets and recurrent neural networks are presented as two fundamental deep-learning approaches for sequence processing. A one-dimensional convnet broadly scans local regions of the sequence to detect local patterns. An RNN broadly processes sequence information recurrently over time. The distinction is about the model's way of processing the ordered input, not about whether preprocessing is required: both approaches still need numeric sequence tensors rather than raw text.
Where Local Patterns Matter
A one-dimensional convnet is a reasonable sequence-processing choice when the task can benefit from detecting patterns in nearby positions of an ordered input. Text and timeseries are both examples of sequence data, and other ordered inputs can be treated in the same general way. The model choice should follow the structure of the sequence and the kind of pattern the task requires the model to process.
Consider a text task in which nearby words or characters form useful patterns. After the text has been tokenized and vectorized, a one-dimensional convnet can scan the resulting sequence to detect local features. The same broad reasoning can apply to timeseries data because timeseries are also organized over ordered positions.
Common Preprocessing Mistakes
Common mistake Treating raw text as if it were already neural-network input
What it looks likeDescribing a sentence as the final input without explaining how it becomes numeric.
Why it failsA neural network needs a numeric tensor rather than raw text.
Do this insteadState the tokenization step, the vectorization step, and the packing of vectors into a sequence tensor.
Common mistake Calling tokenization and vectorization the same operation
What it looks likeSaying that splitting a sentence into words has already produced the numeric representation.
Why it failsTokenization identifies the units, while vectorization supplies each token with a numeric vector.
Do this insteadDescribe tokenization first and vectorization second.
Common mistake Ignoring order in sequence data
What it looks likeDiscussing text as though it were an unordered collection of values.
Why it failsText, timeseries, and other sequence data are ordered inputs.
Do this insteadPreserve the sequence positions when describing the numeric tensor and the model's processing.
Common mistake Describing a one-dimensional convnet as recurrent processing
What it looks likeExplaining a convolutional scan as though it carried information recurrently over time.
Why it failsThe broad roles differ: a one-dimensional convnet scans local regions, while an RNN processes sequence information recurrently over time.
Do this insteadUse local scanning and local-pattern detection for the convnet, and recurrent processing over time for the RNN.
Practice Check
A sentence is split into word units, and each word is then associated with a numeric vector. Explain which operation performed tokenization, which performed vectorization, and what must happen before the result can be passed to a neural network.
Hints
- Tokenization identifies the units in the sequence.
- Vectorization supplies a numeric vector for each generated token.
- The vectors are packed into a sequence tensor.
Choose between these descriptions: a model scans nearby positions to detect local patterns, or a model processes sequence information recurrently over time. Which description matches a one-dimensional convnet, and which matches an RNN?
Hints
- Look for the phrase local regions.
- Look for the phrase recurrently over time.
Key Takeaways
- Sequence data includes text, timeseries, and other ordered inputs.
- A neural network needs a numeric tensor rather than raw text.
- Tokenization identifies units such as words, subwords, or characters; vectorization assigns those units numeric vectors.
- One-hot encoding and token embedding are two major ways to associate vectors with tokens, with token embedding typically used for words and also called word embedding.
- A one-dimensional convnet scans local regions to detect patterns, while an RNN processes sequence information recurrently over time.