Experiments Leading to the Reward Prediction Error Hypothesis
The hypothesis is not simply that dopamine signals the presence of a reward. Its focus is the change between an old expected-reward estimate and a new estimate.
The Central Puzzle
The reward prediction error hypothesis begins with a change in an estimate. A learning system first has an old estimate of the reward expected in the future. It then forms a new estimate. The difference between those estimates is treated as an error signal. The hypothesis proposes that one function of phasic activity in dopamine-producing neurons is to deliver this error to target areas throughout the brain.
The hypothesis is not simply that dopamine signals the presence of a reward. Its focus is the change between an old expected-reward estimate and a new estimate.
Tracking Two Estimates
The old estimate is the reward expectation held before new information is incorporated. The new estimate is the expectation formed after the system has processed a reward or outcome. The prediction error is the difference between these two estimates. This makes the error a learning quantity: it describes how much the system's expectation needs to change.
A Three-Unit Prediction Error
Suppose the old estimate of expected future reward is 5 and the new outcome-based estimate is 8. What is the difference between the estimates?
- 1Identify the old estimate
The system expected a reward amount of 5 before incorporating the new information.
- 2Identify the new estimate
After the outcome is processed, the new estimate is 8.
- 3Compare the estimates
The difference is 8 minus 5, which equals +3.
The prediction error is +3. The new estimate is three units higher than the old estimate.
From TD Error to Dopamine
Reinforcement learning describes signals that help an agent learn from the consequences of its behavior. The TD error is a reinforcement-learning quantity that belongs to a learning process unfolding over time. Researchers noticed that the behavior of TD errors showed striking parallels with the phasic activity of dopamine neurons. The reward prediction error hypothesis links these two patterns by proposing that phasic dopamine-producing neuron activity represents, or delivers, a reward prediction error.
This connection explains why TD errors matter to dopamine research. A TD error is not merely a label for a reward. It is part of a computational account of how expectations change during learning. If dopamine neuron activity shows similar patterns, that activity could function as a reinforcement signal reaching multiple brain areas and supporting learning.
Schultz and the Experimental Pattern
Wolfram Schultz and colleagues conducted influential experiments in the late 1980s and 1990s. Their experiments revealed parallels between dopamine neuron activity and TD errors. These neural-computational parallels helped lead scientists to the reward prediction error hypothesis.
The learning pattern can be described in terms of three elements: an unexpected reward, a predictive cue, and the development of a reward expectation. Early in learning, phasic activity is associated with the unexpected reward. As learning progresses and the cue becomes predictive, the relevant phasic activity is associated with the predictive cue. This pattern parallels the way a TD error changes as an agent learns what to expect.
The hypothesis was first explicitly stated by Montague, Dayan, and Sejnowski in 1996, following the earlier experimental work.
Avoiding Overstatements
Common mistake Treating the hypothesis as the claim that dopamine simply signals pleasure.
What it looks likeDopamine activity appears, so the hypothesis must be saying that the system is experiencing pleasure.
Why it failsThe hypothesis focuses on an error between old and new expected-reward estimates, not simply on pleasure.
Do this insteadDescribe phasic dopamine activity as a proposed signal related to reward prediction error and learning.
Common mistake Treating the hypothesis as the claim that dopamine signals every reward.
What it looks likeA reward exists, so dopamine must signal it in the same way regardless of what was expected.
Why it failsThe important change is the difference between what was expected and the new estimate formed after additional information.
Do this insteadAsk how the new information changes the expected-reward estimate.
Common mistake Describing TD error as an isolated description of a reward.
What it looks likeThe TD error is just the reward amount.
Why it failsThe source presents TD error as a reinforcement-learning quantity in a process unfolding over time.
Do this insteadConnect TD error to changing expectations and learning from consequences.
Common mistake Saying that the experiments proved that dopamine is identical to a TD error.
What it looks likeThe experiments found TD errors inside dopamine neurons.
Why it failsThe hypothesis is based on parallels between a computational description and experimental observations.
Do this insteadSay that the experiments revealed neural-computational parallels that helped support the hypothesis.
| Overstatement | More precise claim |
|---|---|
| Dopamine equals pleasure. | Phasic dopamine activity is proposed to deliver a reward prediction error. |
| Dopamine signals the presence of a reward. | The hypothesis tracks a change between old and new expected-reward estimates. |
| TD error is just the reward. | TD error is a reinforcement-learning quantity involved in learning over time. |
| The experiments established an identity between neurons and an equation. | The experiments revealed parallels between TD-error behavior and phasic dopamine activity. |
Use the right level of precision when describing the hypothesis.
Apply the Idea
What do you think happens?
An old expected-reward estimate is 5, and a new estimate is 8. What prediction error results from comparing them?
Reveal answer
+3
The new estimate is 8 and the old estimate is 5. Their difference is 8 minus 5, or +3. The positive result indicates that the new estimate is higher than the old estimate.
Explain in two or three sentences why a researcher would compare TD-error behavior with phasic dopamine neuron activity rather than simply recording whether a reward occurred.
Hints
- Begin with what a TD error represents in a learning process.
- Then explain what the reward prediction error hypothesis proposes about phasic dopamine activity.
Summarize the contribution of Wolfram Schultz's experiments without using the words pleasure or reward presence.
Hints
- Mention the parallels between dopamine neuron activity and TD errors.
- Explain that these parallels helped lead to the reward prediction error hypothesis.
Key Takeaways
- 1The reward prediction error hypothesis concerns the difference between an old estimate and a new estimate of expected future reward.
- 2A TD error is a reinforcement-learning quantity associated with learning over time from consequences.
- 3The hypothesis proposes that phasic dopamine-producing neuron activity delivers a reward prediction error to target areas throughout the brain.
- 4The relevance of dopamine activity comes from its parallel with TD-error behavior, not from the name dopamine alone or from a simple claim about pleasure.
- 5Wolfram Schultz's experiments in the late 1980s and 1990s revealed influential neural-computational parallels that helped lead to the hypothesis.
At a glance
- A reward prediction error is the difference between an old expected-reward estimate and a new estimate.
- TD errors describe a learning-related signal in reinforcement learning, and their behavior shows parallels with phasic dopamine activity.
- The hypothesis proposes that phasic dopamine-producing neuron activity delivers this error to target areas and supports learning.
- Schultz's experiments were influential because they revealed parallels between dopamine neuron activity and TD errors.
- The hypothesis should not be reduced to the claims that dopamine simply signals pleasure, reward presence, or reward value.