TD Errors

Reinforcement learning theory describes signals that help an agent learn from the consequences of its behavior. Neuroscientists have searched for brain activity that could play a comparable role. The reward prediction error hypothesis proposes that dopamine neuron activity represents one such signal: a reward prediction error. The hypothesis became influential because the behavior of TD errors in reinforcement learning theory shows striking parallels with the phasic activity of dopamine neurons.

A TD error is not simply a description of whether a reward occurred. It is a learning quantity that compares an existing estimate with a better estimate formed after new information arrives. This makes TD errors relevant to dopamine research: both are studied as signals that may indicate how learning should respond to what happens over time.

The reward prediction error hypothesis links dopamine neuron activity with reward prediction errors. Its central evidence is the parallel between TD-error behavior and phasic dopamine activity. Phasic dopamine activity functions as a reinforcement signal that reaches multiple brain areas and supports learning.

Wolfram Schultz and colleagues conducted influential experiments in the late 1980s and 1990s. Those experiments revealed parallels between dopamine neuron activity and TD errors, helping lead scientists to the reward prediction error hypothesis. The important contribution was the comparison between a computational description and experimental observations, not the name of dopamine by itself.

The expression has two sides to compare. V(S_t) is the current estimate for the state at time t. R_t+1 + γV(S_t+1) is a one-step-better estimate because it uses information from the next transition: the reward received and the estimated value of the next state. The TD error records the difference between that newer one-step estimate and the old estimate.

The label δ_t refers to the error in the estimate V(S_t), but calculating it requires R_t+1 and S_t+1. Those are next-step observations. Therefore, the TD error associated with time t becomes available at time t+1.

After the transition from S_t to S_t+1, two pieces of new information are available: the next reward R_t+1 and the next state S_t+1. The reward is included directly. The estimate of the next state is multiplied by γ, the discount factor, and then included. Together they form the sample backup target R_t+1 + γV(S_t+1).

This target is called a sample backup target because it uses the particular reward and next state observed in one transition. The TD error then compares this target with the estimate that existed for S_t.

The sign of the TD error describes how the one-step estimate compares with the current estimate. A positive error means the one-step estimate is greater than V(S_t); a negative error means it is lower; and a zero error means the two estimates match. As a reinforcement signal, this difference can indicate whether the current estimate should move upward, downward, or not change in response to the new transition.

This is why phasic dopamine activity is discussed as a possible reinforcement signal. The hypothesis connects changes in dopamine neuron activity with a computational difference that tells a learning system how new consequences compare with its prior prediction.

TD error and Monte Carlo error use different references. TD error uses a one-step sample backup: the next reward plus the discounted estimate of the next state. Monte Carlo error is based on the actual return. The source gives an important relationship between them: the Monte Carlo error can be written as a sum of TD errors.