Dopamine and Reward-Based Learning

A reward is not informative only because it has a particular size. It can be informative because it differs from what was expected. Receiving more than expected, exactly what was expected, or less than expected creates three different learning situations. The reward prediction error hypothesis applies this idea to dopamine neurons: their activity is proposed to correspond to the gap between the expected reward and the actual reward.

Temporal-difference learning, usually shortened to TD learning, turns this comparison into a computational signal called TD error. At one time step, the learner has an expectation. An outcome then occurs, and the learner also considers the expected value associated with the next time step. TD error connects these pieces: it reflects how the new information differs from what the learner's expectations implied.

The hypothesis is also used to describe how prediction errors change across learning. Before a reward is predicted, the reward itself can be the important event for the prediction error. As the reward becomes learned and expected, the informative event can move earlier to a predictive cue. The cue then carries information about the upcoming outcome, while the reward no longer produces the same kind of surprise when it arrives as expected.

The important point is the change in where the mismatch is located in time. When the reward is not predicted, the reward can carry the unexpected information. Once an earlier cue predicts it, the cue becomes the informative event. This temporal pattern is one reason the choice of representation matters: the representation determines which parts of the situation TD learning treats as distinct states and when it assigns predictive information.

TD error is not produced independently of the input representation. TD learning must receive information in some representation of the situation. That representation affects which situations are treated as similar, which events are treated as different, and how prediction unfolds over time. As a result, two representations of the same broad situation can produce different TD-error patterns.

The source gives the CSC representation as an important example of a representation producing close parallels with observed dopamine neuron activity. Timing is especially important. A representation can affect when TD learning predicts a signal, so representation is not a minor implementation detail added after the theory is complete. It is part of what determines whether the model matches experimental observations well.

The reward prediction error hypothesis is influential because it provides a computational interpretation that can be compared with dopamine neuron activity. TD error gives researchers a concrete signal to examine rather than treating reward-related activity as an unexplained response to reward size alone. The correspondence is strong for most monitored dopamine neurons.

However, the correspondence is not perfect. Exceptions and mismatches exist, and some experimental situations do not fit the hypothesis exactly. A careful conclusion is therefore that TD error captures an important pattern in the data, not that every dopamine observation must be identical to a TD-error prediction. Representation choices can improve or weaken the match, especially in the timing of predicted activity.