Timeseries Forecasting
Sequence data includes text, timeseries, and other ordered inputs.
From Ordered Data to Model Input
Sequence data is data whose elements occur in an order. Text, timeseries, and other ordered inputs all fit this description. A neural network cannot receive raw text directly. Before sequence data can be processed, it must be represented as numeric tensors. This gives sequence processing two broad stages: create a numerical representation, then apply a model designed for sequences.
Tracing a Sequence Representation
A Short Text Sequence
Trace what happens to the ordered text sequence "red sky" before it can be given to a neural network.
- 1Start with ordered text
The input is text, and its elements have an order. In this generated example, the units are words: "red" followed by "sky".
- 2Tokenize
Tokenization identifies the units in the sequence. Here, the generated tokens are "red" and "sky".
- 3Vectorize
Text vectorization supplies each generated token with a numeric representation. The exact numbers are not specified here; the important change is from tokens to numeric vectors.
- 4Pack the sequence
The vectors remain arranged in sequence order and are packed into a sequence tensor.
The neural network receives an ordered numeric tensor rather than the original raw text.
The same reasoning applies to a timeseries. Its values occupy ordered positions, and the resulting numerical representation must preserve that sequence organization. The model therefore receives numeric sequence data, not an unprocessed description of the original input.
Tokenization and Vectorization
Tokenization and vectorization solve different problems. Tokenization chooses or identifies the units in a sequence. For text, those units can be words or characters. Vectorization then gives each generated token a numeric vector. Only after this second step can the vectors be packed into sequence tensors for a deep neural network.
| Stage | Main question | Result |
|---|---|---|
| Tokenization | What are the units in this sequence? | Tokens such as words or characters |
| Vectorization | How is each token represented numerically? | A numeric vector for each token |
| Sequence tensor formation | How are the representations supplied to the model? | Vectors packed in sequence order |
Tokenization identifies units; vectorization supplies their numeric representations.
Two major ways to associate vectors with tokens are one-hot encoding and token embedding. Token embedding is typically used for words and is also called word embedding in that setting. These are ways of supplying numeric representations; they do not replace the need to identify the sequence units first.
Choosing a Sequence Model
Once sequence data has become numeric tensor data, a model designed for sequences can process it. The source identifies two fundamental deep-learning approaches for sequence processing: one-dimensional convolutional networks, or 1D convnets, and recurrent neural networks. A 1D convnet applies the one-dimensional convolutional approach to a sequence. At a broad level, it is useful when the task depends on detecting patterns across nearby positions in an ordered input. An RNN provides the contrasting sequential approach by carrying information through sequential states as it processes the sequence.
This distinction is about the broad processing role, not a rule that one model is always correct. A 1D convnet is a natural candidate when useful evidence appears as patterns across nearby sequence positions. A recurrent model is the contrasting choice when the explanation emphasizes information being carried through sequential states. Both still require an appropriate numeric sequence representation as input.
Where 1D Convnets Fit
For a text sequence, a 1D convnet can be considered when nearby words or characters form useful patterns. For a timeseries, it can be considered when nearby positions contain a useful temporal pattern. The same reasoning extends to other ordered inputs. In every case, the raw input must first be represented as numeric sequence data.
Mistakes Beginners Make
Common mistake Treating raw text as if it were already a neural-network input.
What it looks likePassing a sentence directly to a neural network without first identifying units and creating numeric representations.
Why it failsA neural network needs a numeric tensor rather than raw text.
Do this insteadSeparate the preparation stages: tokenize the text, vectorize the tokens, and pack the resulting vectors into a sequence tensor.
Common mistake Using tokenization and vectorization as if they meant the same thing.
What it looks likeSaying that tokenization turns each word into a numeric vector.
Why it failsTokenization identifies the units; vectorization supplies their numeric representations.
Do this insteadDescribe tokenization first and vectorization second.
Common mistake Ignoring sequence order after creating numeric vectors.
What it looks likeTreating the vectors as an unordered collection instead of packing them in sequence order.
Why it failsText and timeseries are ordered inputs, so their positions are part of the sequence representation.
Do this insteadKeep the generated vectors arranged as a sequence tensor.
Common mistake Describing a 1D convnet and an RNN as identical sequence models.
What it looks likeExplaining both models only as generic tools for sequence data.
Why it failsThe approaches have different broad roles: a 1D convnet applies one-dimensional convolution, while an RNN carries information through sequential states.
Do this insteadUse local pattern detection as the broad 1D-convnet intuition and sequential state as the broad RNN intuition.
Check Your Understanding
A dataset contains ordered text and you want to examine whether a 1D convnet is a suitable sequence-processing approach. Describe the preparation and model-selection reasoning in order. Your answer should name the units chosen during tokenization, explain what vectorization contributes, state what the neural network receives, and identify the kind of sequence pattern that would make a 1D convnet a sensible candidate.
Hints
- Start by separating tokenization from vectorization.
- Remember that the model receives a numeric tensor, not raw text.
- Use the broad role of a 1D convnet rather than claiming that it is suitable for every sequence task.
A Complete Reasoning Chain
Choose the correct description for a text sequence that will be processed by a sequence model.
- 1Identify the data type
Text is ordered sequence data. It may be treated as a sequence of words or characters.
- 2Prepare the representation
Tokenization identifies the chosen units. Vectorization gives each token a numeric vector, and those vectors are packed into a sequence tensor.
- 3Select a broad model role
A 1D convnet is a candidate when useful patterns occur across nearby sequence positions. An RNN represents the contrasting sequential-state approach.
The correct reasoning keeps representation and model selection separate: first create ordered numeric input, then choose a sequence-processing approach based on the pattern the task requires.
Key Takeaways
- 1Sequence data includes text, timeseries, and other ordered inputs.
- 2Neural networks receive numeric tensors, so raw sequence data must first be represented numerically.
- 3Tokenization identifies sequence units; vectorization assigns those units numeric vectors.
- 4One-hot encoding and token embedding are two major ways to associate vectors with tokens.
- 51D convnets and recurrent neural networks are two fundamental approaches to sequence processing, with different broad processing roles.
At a glance
- Sequence data is ordered data, including text and timeseries.
- A neural network needs numeric sequence tensors rather than raw text.
- Tokenization identifies units, while vectorization represents those units numerically.
- 1D convnets apply one-dimensional convolution to sequence processing and are useful to consider when nearby positions form meaningful patterns.
- RNNs provide a contrasting sequential-state approach to processing sequence data.