Text Generation with LSTM
The initial text is called conditioning data.
From Prediction to Generation
A language model learns the statistical structure of language by modeling possible next tokens. Its basic task is local: given tokens that have already appeared, it assigns probabilities to tokens that could appear next. Text generation extends this task through repetition. The model predicts one token, that prediction is appended to the sequence, and the enlarged sequence is used for another prediction.
The central idea is a repeated feedback cycle: predict, append, and predict again.
Tracing the First Prediction
A Short Character Sequence
Suppose a character-level model receives the characters c, a, and t as its current sequence. What happens when the model generates more text?
- 1Start with the current sequence
The model receives c, a, and t as the conditioning sequence. In a character-level setup, these individual characters are the tokens.
- 2Produce the next-character distribution
The LSTM character model produces a probability distribution over possible next characters. That distribution represents how plausible each possible next character is after c, a, and t.
- 3Select a next character
A next character is selected using the model's probabilities. The selected character becomes part of the generated sequence.
- 4Feed the enlarged sequence back
The current sequence now contains c, a, t, and the selected character. This enlarged sequence supplies more context for the next prediction.
The model does not need to produce the entire continuation in one step. It produces one next character, attaches it to the sequence, and repeats the process.
Tokens and Language Models
The word token does not necessarily mean a whole word. In language modeling, a token can be a word or a character. More generally, the model uses the preceding sequence to assign probabilities to possible next tokens. This makes language modeling a task of representing the statistical structure of language through next-token possibilities.
| Language modeling in general | Character-level language modeling |
|---|---|
| Models possible next tokens from preceding tokens. | Uses individual characters as the tokens. |
| The token may refer to a word or a character. | Receives a string containing N characters and predicts character N + 1. |
| The output provides probabilities for possible next tokens. | The output is a softmax over possible characters, forming a probability distribution for the next character. |
Character-level modeling is one particular choice of prediction unit within language modeling.
Character-level modeling changes the prediction unit, not the overall generation mechanism. The model still predicts, appends, and predicts again.
Following the Feedback Cycle
Consider a generated sequence that begins with a short conditioning string. At the first step, the LSTM uses that string to produce probabilities for possible next characters or words. After one output is selected and attached, the model receives a longer sequence at the next step. The second prediction therefore uses the original conditioning data together with the newly appended token. The same relationship continues at every later step.
What do you think happens?
After the model predicts and appends one token, what sequence is used for the next prediction?
Reveal answer
The enlarged sequence containing the earlier context and the appended token
Generation uses repeated feedback. The output from one step is added to the sequence, giving the next prediction more context than the previous prediction.
Common mistake Treating text generation as a single prediction of an entire sentence.
What it looks likeAssuming the model receives a short prompt and immediately writes the complete continuation in one step.
Why it failsThe described sequence generator predicts one next token, appends it, and uses the enlarged sequence for another prediction.
Do this insteadTrace generation as a repeated cycle of prediction, appending, and feedback.
Common mistake Forgetting the role of the conditioning data.
What it looks likeDescribing the first prediction as if it were made without an initial sequence.
Why it failsThe initial text supplies the context for the model's first next-token prediction.
Do this insteadIdentify the initial text as conditioning data and connect it to the first prediction.
Common mistake Assuming that token always means word.
What it looks likeSaying that a character-level model predicts a whole word at each step.
Why it failsA character-level model uses individual characters as its tokens.
Do this insteadState which prediction unit is being used: a character, a word, or another token type.
Common mistake Confusing a probability distribution with a complete continuation.
What it looks likeDescribing the LSTM character model's output as one finished string.
Why it failsThe model produces a probability distribution over possible next characters.
Do this insteadSeparate the distribution over possible next outputs from the selected token that is appended.
Practice the Generation Logic
Explain the generation process for a character-level LSTM when its current sequence contains N characters. Your explanation should name the model output, describe what happens to the selected character, and state what sequence is used at the next step.
Hints
- Begin with the model's prediction unit.
- Describe the output as a probability distribution over possible next characters.
- Explain how appending the selected character changes the sequence used for the next prediction.
Checking a Generation Trace
A model starts with conditioning data and generates three additional tokens. What should appear in a correct trace?
- 1First step
The conditioning data is used to produce a distribution over possible next tokens. One token is selected and appended.
- 2Second step
The model uses the enlarged sequence, which now includes the first selected token, to produce the next distribution.
- 3Third step
The second selected token is appended, and the resulting sequence is used to produce the third prediction.
A correct trace shows three prediction steps and two feedback updates before the third prediction is made, with every later step using the sequence enlarged by earlier outputs.
Key Takeaways
- 1A language model represents the statistical structure of language by assigning probabilities to possible next tokens based on preceding tokens.
- 2Text generation is iterative: the model predicts a token, appends it to the sequence, and predicts again.
- 3The initial sequence is conditioning data, and each generated token becomes part of the context for later predictions.
- 4A character-level model uses individual characters as tokens and produces a probability distribution over possible next characters.
- 5The same feedback mechanism can continue repeatedly to produce sequences of arbitrary length.
At a glance
- A language model predicts possible next tokens from previous tokens.
- An LSTM sequence generator creates text one token at a time through repeated feedback.
- Conditioning data determines the context for the first prediction.
- Each appended output enlarges the context used for the next prediction.
- Character-level language modeling uses individual characters as the prediction units.