Token Embeddings

Token embedding is typically used for words and is also called word embedding.

Programming

Interactive lab

Try it: Text Vectorization: Tokens to Vectors

The text-to-tensor pipeline: standardize, split into tokens, build a frequency-ranked vocabulary with padding and [UNK], map tokens to integer ids, pad or truncate, then one-hot or multi-hot encode and look the ids up in an embedding table.

How it works

Standardize: lowercase and delete punctuation.
Split on whitespace into tokens.
Build the vocabulary: 0 = padding, 1 = [UNK], then words by frequency (ties: first appearance), capped at max tokens.
Map each token to its id (1 if it did not make the vocabulary), then pad with 0 or truncate to the sequence length.
Encode: one-hot rows per position or one multi-hot row per text; embed: id k selects row k of the embedding table.

Default run (18 steps): Raw text (59 characters): "The movie was great. The plot was great, the acting wasn't!". A model cannot read strings, so it is turned into numbers step by step. … Embedding lookup: id k selects row k of a fixed 8 x 3 table, giving a (8, 3) dense matrix instead of (8, 8) sparse one-hots. Row 0 (padding) is looked up too; a trained Embedding layer would learn these rows.

Simplified: Bounded ASCII text (up to 200 characters, 40 tokens), whitespace splitting only, and a small FIXED embedding table (a real Embedding layer learns its rows). The multi-hot vector keeps the padding slot (always 0) so its positions line up with the ids.

Educational simulation

Loading the simulation…

From Sequence to Tensor

A sequence is ordered data. Text, timeseries, and other ordered inputs all have meaningful positions. A neural network cannot receive raw text directly; it needs a numeric tensor. Sequence processing therefore begins before the model runs: the original sequence must first be converted into a numerical representation.

How does raw text or another ordered sequence become a numeric tensor that a neural network can process?

Key point

The key transformation is not simply “text becomes numbers.” The ordered units must be identified first, then each unit must receive a numeric representation, and those representations must remain arranged as a sequence.

Tokenization and Vectorization

Tokenization chooses the units in a sequence. For text, the units can be words or characters. Tokenization does not yet provide the neural network with the vectors it needs; it identifies the pieces that will later be represented numerically.

Vectorization turns the generated tokens into numeric representations. The source describes two broad ways to associate vectors with tokens: one-hot encoding and token embedding. Token embedding is typically used for words and is also called word embedding. After vectorization, the vectors are packed into sequence tensors that can be given to a deep neural network.

What changes when text is split into tokens, and how are those tokens converted into numeric representations?

Worked example

Following One Short Text Sequence

Trace the conceptual conversion of the ordered text “birds fly” into an input suitable for a neural network.

1
Choose units
Treat the two words as the tokens “birds” and “fly.” This is tokenization: deciding which units make up the sequence.
2
Assign representations
Give “birds” and “fly” numeric vectors through a vectorization method such as token embedding. The exact vector values are not important for this conceptual trace.
3
Preserve order
Keep the vector for “birds” before the vector for “fly,” because the input is an ordered sequence.
4
Pack the sequence
Place the ordered vectors into a sequence tensor, which can then be fed into a deep neural network.

Result

The text has become an ordered sequence of numeric vectors rather than remaining raw text.

Embedding Lookup

A token embedding associates a token with a numeric vector. Conceptually, you can think of a lookup from each token to its vector. Applying that lookup across an ordered token sequence produces an ordered sequence of vectors. The resulting collection is the representation that can be packed into a sequence tensor.

How does each token map to a learned vector, and what does the resulting sequence of vectors look like?

Sliding Through a Sequence

Once a sequence has been represented as a numeric tensor, a model designed for sequences can process it. A 1D convnet applies the one-dimensional convolutional approach to sequence processing. Its defining operation is a one-dimensional convolution applied across the ordered sequence.

How does a sliding 1D convolution examine neighboring positions in an ordered sequence?

The important conceptual connection is this: token embeddings provide numeric vectors at ordered positions, and a 1D convnet applies a sequence-processing operation to those positions. The model therefore works on the numerical sequence representation rather than on raw words or characters.

Convnets and Recurrent Networks

The source identifies two fundamental deep-learning approaches for sequence processing: 1D convnets and recurrent neural networks. A 1D convnet uses the one-dimensional convolutional approach on the sequence. A recurrent neural network is the other broad approach named for sequence processing. At this level, the main distinction is the model family being applied after the sequence has been converted into numeric tensors.

How does a 1D convnet differ from a recurrent neural network at the broad model-family level?

Best practice

When choosing terminology, do not call every step “embedding.” Embedding describes the numeric representation associated with tokens. A 1D convnet or recurrent neural network describes the sequence-processing model that can operate on the resulting tensor.

Where Sequence Models Fit

Sequence-processing models are appropriate when order is part of the input. Text is one example: it can be treated as a sequence of words or characters. Timeseries data is another: its values are organized over a sequence of positions. Other ordered inputs can also be represented as sequences before being processed by a deep-learning model.

Input type	How it is viewed	Initial representation step
Text	A sequence of words or characters	Tokenize, then vectorize
Timeseries	Values arranged over sequence positions	Create a numeric sequence representation
Other ordered input	A sequence whose positions have meaning	Represent it as a numeric tensor

Examples of ordered inputs that can be prepared for sequence processing

Mistakes to Avoid

Common mistake
Treating tokenization and vectorization as the same operation.
What it looks like
Saying that splitting text into words has already produced the numeric input for a neural network.
Why it fails
Tokenization identifies the units, while vectorization supplies numeric representations for those units.
Do this instead
Describe the stages separately: first choose tokens, then turn them into vectors and pack those vectors into a sequence tensor.
Common mistake
Feeding raw text directly into a neural network.
What it looks like
Passing a sentence to a model without converting it into a numeric tensor.
Why it fails
The source states that neural networks receive numeric tensors rather than raw text.
Do this instead
Create a numerical representation before applying a deep-learning model.
Common mistake
Calling a token embedding a sequence-processing model.
What it looks like
Describing the embedding itself as the 1D convnet.
Why it fails
The embedding supplies vectors for tokens; the 1D convnet is the model that applies one-dimensional convolution to sequence data.
Do this instead
Use “token embedding” for the representation and “1D convnet” for the sequence-processing approach.
Common mistake
Ignoring order after vectorization.
What it looks like
Treating the vectors for an ordered text sequence as an unarranged collection.
Why it fails
Sequence data is defined by meaningful order, and the vectors are packed into sequence tensors.
Do this instead
Preserve the sequence positions when assembling the numeric representation.

Practice Check

EASY

A learner says: “I tokenized the sentence, so it is ready for a 1D convnet.” Explain what step is still needed and why.

Hints

Ask what tokenization identifies.
Ask what a neural network requires as input.
Name the operation that supplies each token with a numeric vector.

Your attempt (not saved)

MEDIUM

Classify each item as a sequence representation step or a sequence-processing model: tokenization, token embedding, 1D convnet, recurrent neural network.