Tokenization

Tokenization is part of the transformation that makes this possible.

Programming

Interactive lab

Try it: Text Vectorization: Tokens to Vectors

The text-to-tensor pipeline: standardize, split into tokens, build a frequency-ranked vocabulary with padding and [UNK], map tokens to integer ids, pad or truncate, then one-hot or multi-hot encode and look the ids up in an embedding table.

How it works

Standardize: lowercase and delete punctuation.
Split on whitespace into tokens.
Build the vocabulary: 0 = padding, 1 = [UNK], then words by frequency (ties: first appearance), capped at max tokens.
Map each token to its id (1 if it did not make the vocabulary), then pad with 0 or truncate to the sequence length.
Encode: one-hot rows per position or one multi-hot row per text; embed: id k selects row k of the embedding table.

Default run (18 steps): Raw text (59 characters): "The movie was great. The plot was great, the acting wasn't!". A model cannot read strings, so it is turned into numbers step by step. … Embedding lookup: id k selects row k of a fixed 8 x 3 table, giving a (8, 3) dense matrix instead of (8, 8) sparse one-hots. Row 0 (padding) is looked up too; a trained Embedding layer would learn these rows.

Simplified: Bounded ASCII text (up to 200 characters, 40 tokens), whitespace splitting only, and a small FIXED embedding table (a real Embedding layer learns its rows). The multi-hot vector keeps the padding slot (always 0) so its positions line up with the ids.

Educational simulation

Loading the simulation…

From Words to Model Input

A sequence is data whose elements have an order. Text can be treated as a sequence of words or characters, just as a timeseries is another kind of ordered data. However, a deep-learning system does not receive raw words or characters directly. It receives numeric tensors arranged as sequences. Tokenization is part of the transformation that makes this possible.

What happens to raw text as it moves through tokenization and into a form that a neural network can process?

Tokenization and Vectorization

Tokenization and text vectorization are related but different operations. Tokenization selects the units that make up the sequence. Those units may be words, characters, or n-grams. Text vectorization then assigns numeric vectors to the selected units and forms sequence tensors. In short, tokenization answers which text units are present, while vectorization answers how those units are represented numerically for a model.

Operation	Main question	Result
Tokenization	Which units should form the sequence?	Words, characters, or n-grams
Text vectorization	How should those units be represented numerically?	Numeric vectors arranged into sequence tensors

The source distinguishes selecting sequence units from assigning their numeric representations.

What is the difference between splitting text into tokens and converting those tokens into numeric representations?

A Worked Text Transformation

Worked example

Tracing a Short Ordered Text

Trace the conceptual transformation of the text "deep models learn" when the text is treated as a sequence of words.

1
Select the units
Treat the text as a word sequence. The selected units are "deep", "models", and "learn". This is the tokenization step.
2
Retain the order
The units remain arranged in their sequence order: "deep" comes before "models", and "models" comes before "learn".
3
Assign numeric representations
Text vectorization assigns numeric vectors to the selected units.
4
Form the model input
The numeric representations are arranged as a sequence tensor, which is the kind of ordered numeric input a sequence-processing neural network can receive.

Result

The text has moved conceptually from ordered words to an ordered numeric representation. Tokenization selected the words; vectorization supplied the numeric representation.

How do token positions and their order remain represented after text has been converted into numbers?

Sequence Models After Vectorization

Once sequence information has been represented numerically, it can be supplied to a sequence-processing neural network. The supplied material presents recurrent neural networks and one-dimensional convolutional networks, or 1D convnets, as two fundamental deep-learning algorithms for sequence processing. A 1D convnet is described as the one-dimensional version of the 2D convolutional networks used for other data types.

How is each token associated with the numeric representation used in a sequence tensor?

Common Misunderstandings

Common mistake
Treating tokenization and vectorization as the same operation.
What it looks like
Calling the assignment of numeric vectors the tokenization step.
Why it fails
The source distinguishes selecting words, characters, or n-grams from assigning numeric vectors and forming sequence tensors.
Do this instead
Use tokenization for selecting the sequence units and text vectorization for creating their numeric representations.
Common mistake
Assuming a neural network can process raw words or characters directly.
What it looks like
Passing a sentence to a sequence model without a numeric representation.
Why it fails
The source states that deep-learning systems receive numeric tensors arranged as sequences, not raw words or characters.
Do this instead
Explain the transformation from text units to numeric vectors and then to an ordered sequence tensor.
Common mistake
Ignoring sequence order after converting text to numbers.
What it looks like
Describing a sequence tensor as merely an unordered collection of numeric values.
Why it fails
A sequence is defined by the order of its elements, and the model input is an ordered numeric representation.
Do this instead
Track the position of each token's numeric representation within the sequence.
Common mistake
Inventing a specific 1D pooling calculation from the general description of 1D convnets.
What it looks like
Claiming a particular output length or pooling result without the required technical parameters.
Why it fails
The supplied material does not contain enough information to calculate or diagram a specific 1D pooling transition.
Do this instead
State only that 1D convnets are a fundamental sequence-processing family unless additional pooling specifications are provided.

Check Your Understanding

MEDIUM

A sentence is being prepared for a sequence-processing neural network. Describe, in order, what tokenization does, what text vectorization does, and what form the final model input takes. Then state whether the supplied material gives enough information to calculate a specific 1D pooling transition.

Hints

Begin by naming the possible units selected from text.
Distinguish selecting units from assigning numeric vectors.
Use the phrase ordered numeric representation when describing the final input.
Separate what the source establishes about 1D convnets from details it does not specify.

Your attempt (not saved)

Worked example

Practice Answer

Explain the preparation of a sentence for a sequence-processing neural network.

1
Tokenization
Select words, characters, or n-grams as the units of the text sequence.
2
Text vectorization
Assign numeric vectors to the selected units.
3
Sequence tensor
Arrange the numeric representations as an ordered sequence tensor.
4
Pooling specification
Do not calculate a specific 1D pooling transition from this material alone, because the required technical details are not provided.

Result

The model receives an ordered numeric sequence tensor. Tokenization selects the units, vectorization supplies their numeric representations, and the supplied material does not specify enough information for a particular pooling calculation.

Key Takeaways

1
A sequence is data whose elements have an order, and text can be treated as a sequence of words or characters.
2
Tokenization selects words, characters, or n-grams; text vectorization assigns numeric vectors and forms sequence tensors.
3
Deep-learning systems process ordered numeric representations rather than raw words or characters.
4
Recurrent neural networks and 1D convnets are presented as fundamental deep-learning algorithms for sequence processing.
5
The supplied material does not specify enough technical detail to calculate or diagram a particular 1D pooling transition.

At a glance

Tokenization selects the units of a text sequence, such as words, characters, or n-grams.
Text vectorization assigns numeric vectors to those units and forms ordered sequence tensors.
Sequence order remains part of the numeric representation supplied to a sequence-processing model.
The source presents recurrent neural networks and 1D convnets as fundamental sequence-processing model families.
Specific 1D pooling behavior cannot be calculated from the supplied material without additional technical specifications.

Related concepts

More concepts starting with T