Working with Text and Sequence Data
A sequence is not always just a collection of separate inputs.
Why Order Matters
A sequence is not always just a collection of separate inputs. When information arrives in an order, an earlier item can affect how a later item is understood. This creates a challenge for a network that processes each input independently: after one input has been processed, the network has no maintained state to use when processing the next one.
Consider reading a sentence one word at a time. The meaning assigned to a new word is influenced by the words that came before it. The current word cannot always be interpreted as though it were unrelated to the preceding words.
Updating an Internal Model
Recurrent processing handles a sequence incrementally. At each step, the network receives the current item and uses an internal model that contains information carried from earlier items. The current item then updates that internal model, which is carried forward to the next step. In this way, earlier information remains available while later items are processed.
Tracing three sequence items
Follow what happens when three items arrive one at a time and the network maintains an internal model.
- 1First item
The first item is processed and contributes to the internal model.
- 2Second item
The second item is processed while the model from the first step is available. The model is updated again.
- 3Third item
The third item is processed using the model carried from the earlier steps, and the model is updated once more.
The process is incremental: each item contributes to an internal model, and the updated model is carried to the next item.
Feedforward and Recurrent Processing
| Processing approach | What happens between inputs | Use of earlier information |
|---|---|---|
| Feedforward | Inputs are processed independently | No state is maintained between inputs |
| Recurrent | An internal model is updated as items arrive | Information from earlier items is carried to later steps |
The central difference is whether information is retained between successive inputs.
A feedforward network can receive an entire sequence as one large input, but that is different from processing the sequence incrementally. In independent processing, one input does not leave a maintained state for the next input. In recurrent processing, the sequence is handled step by step, with an internal model preserving information from earlier items.
Context Changes Interpretation
When reading a sentence one word at a time, the meaning assigned to a new word can be influenced by the words that came before it. The later word is still the current input, but the internal model gives the network access to information from earlier positions. The same processing step can therefore use both the current item and carried context.
Reading in order
Explain why a later word in a sentence should not always be treated as an isolated input.
- 1Earlier words arrive
The earlier words contribute information to the internal model as the sentence is processed.
- 2A later word arrives
The later word is processed while the model contains information carried from the preceding words.
- 3Interpretation uses context
The earlier information can affect how the later word is understood.
Position and previous information can affect the interpretation of the current word.
Mistakes About Sequence Processing
Common mistake Assuming that a sequence is merely a collection of independent inputs.
What it looks likeTreating every word as though its interpretation were unrelated to the preceding words.
Why it failsPosition and previous information can affect how the current item is understood.
Do this insteadAsk what information from earlier items should remain available when the current item is processed.
Common mistake Assuming that a feedforward network automatically remembers an earlier input.
What it looks likeExpecting information from one input to be available during the next input without a maintained state.
Why it failsFeedforward networks process inputs independently because they do not maintain state between inputs.
Do this insteadDistinguish independent processing from recurrent processing, which carries an internal model forward.
Common mistake Assuming that calling data a sequence is enough to make a network understand it.
What it looks likeProviding ordered items but ignoring the procedure used to process them.
Why it failsThe important change is that information arrives incrementally, an internal model is maintained, and that model is updated.
Do this insteadFocus on the step-by-step process and on how the internal model is updated between items.
Check Your Understanding
A network processes three ordered items one at a time. After processing the first item, it keeps no information that can be used for the second item. After processing the second item, it keeps no information that can be used for the third item. Which processing approach does this describe, and what limitation does it create?
Hints
- Look for whether an internal model is maintained between inputs.
- Consider whether earlier items can influence the interpretation of later items.
Describe the three-step flow for recurrent processing using these terms: current item, internal model, update, and next item.
Hints
- Start with the current item and the model carried from earlier steps.
- End with the updated model being available when the next item arrives.
Key Takeaways
- 1Sequence data is challenging because the meaning of a current item can depend on earlier items and their positions. Feedforward networks process inputs independently and do not maintain state between them. Recurrent processing works incrementally: it maintains an internal model, updates that model when each item arrives, and carries the updated information forward. This carried context allows later items to be interpreted using information from what came before.
At a glance
- Sequence data has order, so earlier items can affect how later items are understood.
- Feedforward networks process inputs independently because they do not maintain state between inputs.
- Recurrent processing maintains an internal model that is updated as each item arrives.
- The updated internal model carries earlier information into the processing of later items.
- Processing an entire sequence as one large input is different from processing it incrementally.