Feedforward Neural Networks
Feedforward networks process inputs independently because they do not maintain state between inputs.
The Sequence Problem
A feedforward neural network processes inputs independently because it does not maintain state between inputs. This works differently from processing a sequence incrementally. When data arrives one item at a time, the interpretation of the current item may depend on items that appeared earlier. The central question is therefore not only what the current input is, but also what information from the sequence is still available when that input arrives.
A Running Sequence
Reading One Word at a Time
Imagine reading a sentence one word at a time. Consider what happens when a new word arrives after earlier words have already been read.
- 1First word
The first word is processed with no earlier word in the sentence available as context.
- 2Earlier information
As additional words arrive, information from the words that came before can contribute to how the new word is understood.
- 3Current word
The current word is not necessarily interpreted as an isolated input. Its position and the preceding words can affect its meaning.
A sequence is not merely a collection of separate inputs. Earlier information can matter when interpreting what comes next.
Feedforward Without State
The defining limitation in this context is the absence of maintained state. A feedforward network can process an entire sequence as one large input, but that is different from receiving the sequence incrementally and preserving information as each item arrives. If inputs are presented separately, the network does not maintain state between them. The processing of one input does not create an internal model that is carried into the processing of the next input.
| Processing approach | How inputs arrive | Information carried forward |
|---|---|---|
| Feedforward processing | Inputs are processed independently | No maintained state between inputs |
| Processing a sequence as one large input | The entire sequence is presented together | The sequence is available as one input |
| Recurrent processing | Items arrive incrementally | An internal model carries information from earlier items |
The distinction is between presenting a whole sequence at once and processing its items incrementally with carried context.
Carrying an Internal Model
Recurrent processing addresses the sequence challenge by preserving an internal model. When one item arrives, the model is updated. When the next item arrives, information from the earlier items is still available through that model. This makes processing incremental: information is not treated as disappearing after each independent input, but as context that can influence what comes next.
The important change is procedural: information arrives incrementally, an internal model is maintained, and that model is updated when another item arrives. A sequence does not automatically become understandable merely because it is called a sequence.
Why Position Matters
The Same Current Item in Different Contexts
Suppose a current item appears after two different sets of earlier items. Why might the network need the earlier items when interpreting the current one?
- 1Context A
The current item follows one set of earlier sequence items. Its position and preceding information form one context.
- 2Context B
The same current item follows a different set of earlier sequence items. Its position and preceding information now form another context.
- 3Interpretation
Because earlier information can influence the meaning assigned to a new item, the current item should not always be treated as an isolated input.
Earlier sequence items matter because the current item's interpretation can depend on what came before it.
Mistakes About Sequence Processing
Common mistake Assuming that calling data a sequence automatically gives a feedforward network sequence understanding.
What it looks likeItems are presented one at a time, but no information is maintained between those inputs.
Why it failsThe source describes sequence understanding as a procedural issue involving incremental arrival, an internal model, and updates to that model.
Do this insteadAsk whether earlier information is preserved and available when the next item is processed.
Common mistake Treating a whole sequence as identical to incremental processing.
What it looks likeAn entire sequence is presented as one large input, then compared with processing its items one at a time.
Why it failsPresenting the entire sequence at once is different from processing it incrementally.
Do this insteadDistinguish between one large input and a sequence of inputs connected by carried context.
Common mistake Ignoring the effect of earlier items on a later item.
What it looks likeThe current word is interpreted without considering the words that came before it.
Why it failsThe meaning assigned to a new word can be influenced by preceding words.
Do this insteadInclude position and previous information when reasoning about ordered data.
Practice the Distinction
For each situation, decide whether it describes independent feedforward processing or recurrent processing: a whole sequence is supplied as one large input; items arrive one at a time and an internal model is updated after each item; a new item is interpreted using information from earlier items; each input is processed without maintained state.
Hints
- Look for whether inputs are processed independently or incrementally.
- Look for an internal model that carries information from earlier items.
- A whole sequence presented as one large input is distinct from a sequence processed item by item.
- 1A feedforward network processes inputs independently because it does not maintain state between inputs. Sequence data creates a challenge when the meaning of a current item depends on earlier items or on position. Recurrent processing addresses this challenge by maintaining an internal model, updating it as each item arrives, and using the carried information when processing what comes next.
Key Takeaways
- Feedforward networks process inputs independently and do not maintain state between inputs.
- Presenting an entire sequence as one large input differs from processing its items incrementally.
- Recurrent processing maintains an internal model that carries information from earlier items.
- Earlier sequence items can influence how a later item is interpreted.
- The key change for ordered data is the procedure of maintaining and updating context as items arrive.