Reward Signal

The reinforcement signal is the major factor directing a learning update in response to the current situation. It acts as a multiplicative factor in parameter updates.

Programming

A Number That Changes Learning

In reinforcement learning, an agent does not change its policy or value function simply because a situation occurred. A numerical signal guides the size and direction of that change. This number is the reinforcement signal at time t. It may be positive, negative, or zero, and it is the number that most directly guides changes in a policy or value function.

Key point

The reinforcement signal is the major factor directing a learning update in response to the current situation. It acts as a multiplicative factor in parameter updates.

How does the reinforcement signal flow into a change in a policy or value function?

The TD Error Equation

The simplest case is an algorithm in which the reinforcement signal equals the reward signal Rₜ. In that case, the reward is the critical multiplier in the parameter-update equation. Other algorithms use a signal that contains more than the immediate reward. A TD error is one important example because it combines the reward with a change in predicted value.

Formula, step by step

1
Rₜ is the primary reinforcement contribution.
2
γV(Sₜ) − V(Sₜ₋₁) is the secondary reinforcement contribution.
3
δₜ is the complete TD error used as the reinforcement signal in TD state-value learning.
4
The complete signal can therefore reflect the reward contribution, the value-difference contribution, or both.

How do the immediate reward and the change in predicted future value combine to produce a TD error?

Primary and Secondary Reinforcement

The reward term Rₜ is primary reinforcement because it is the direct reward contribution. The expression γV(Sₜ) − V(Sₜ₋₁) is secondary reinforcement because it represents the value-difference contribution. The distinction matters because a TD error can be present even when the reward term is zero: the value-difference contribution can still provide the secondary part of the signal.

Which part comes directly from the reward, and which part comes from a change in predicted value?

Worked example

Classifying a TD Error

Consider a TD error in which Rₜ is zero but γV(Sₜ) − V(Sₜ₋₁) is nonzero.

1
Inspect the reward term
Rₜ equals zero, so there is no primary reinforcement contribution.
2
Inspect the value-difference term
The value-difference term is nonzero, so the TD error still contains a secondary reinforcement contribution.
3
Classify the signal
Only the secondary contribution remains in the TD error.

Result

The TD error represents pure secondary reinforcement.

The same expression can represent pure primary reinforcement, pure secondary reinforcement, or a mixture. These are not three different kinds of TD error. They are three ways to analyze which terms are contributing in a particular case.

Case	Primary contribution	Secondary contribution	Interpretation
Pure primary	Rₜ is nonzero	γV(Sₜ) − V(Sₜ₋₁) is zero	The signal comes from the reward term alone
Pure secondary	Rₜ is zero	γV(Sₜ) − V(Sₜ₋₁) is nonzero	The signal comes from the value-difference term alone
Mixture	Rₜ contributes	γV(Sₜ) − V(Sₜ₋₁) contributes	Both parts are present

Classifying the contents of a TD-error reinforcement signal

Reward as a Formal Goal

Reward signals are the reinforcement learning mechanism for formalizing goals. Because agents learn to maximize reward, the reward design determines what behavior is encouraged. A reward signal should communicate what to achieve rather than prescribe how to achieve it.

What does the reward signal actually encourage, and how does rewarding a subgoal differ from rewarding the desired final goal?

In the real world

A chess-playing agent might be rewarded for taking opposing pieces or controlling the center of the board. Those actions can look useful, but they are not the same as winning the game. If the agent learns to maximize those rewards, it might pursue them even when doing so causes it to lose.

Best practice

When evaluating a reward signal, ask whether maximizing it would reliably represent the outcome you actually want. The signal should reflect the actual desired outcome, not merely a convenient subgoal or a preferred method. It should communicate the goal itself and allow the agent to learn how to pursue it.

Mistakes in Reward Design

Common mistake
Treating a situation as if it automatically changes the policy or value function.
What it looks like
Assuming that merely encountering a situation is enough to produce learning.
Why it fails
A numerical reinforcement signal guides the size and direction of the change.
Do this instead
Identify the reinforcement signal that directs the update.
Common mistake
Ignoring the secondary contribution when the immediate reward is zero.
What it looks like
Concluding that the TD error must be zero whenever Rₜ equals zero.
Why it fails
The value-difference contribution can still be nonzero.
Do this instead
```
Check both Rₜ and γV(Sₜ) − V(Sₜ₋₁).
```
Common mistake
Rewarding a method or subgoal instead of the actual goal.
What it looks like
Rewarding a chess-playing agent for taking pieces rather than for winning.
Why it fails
The agent learns to maximize the reward it receives, even if that behavior can lead away from the desired outcome.
Do this instead
Design the signal to communicate what should be achieved, without prescribing how to achieve it.
Common mistake
Treating primary and secondary reinforcement as separate learning signals.
What it looks like
Describing the reward contribution and value-difference contribution as two independent updates.
Why it fails
They are conceptual parts of one TD-error reinforcement signal.
Do this instead
Analyze the terms separately, then interpret their combination as δₜ.

Check Your Reasoning

EASY

A TD error contains a nonzero reward term and a zero value-difference term. Is the signal pure primary reinforcement, pure secondary reinforcement, or a mixture? Explain which contribution remains.

Hints

Inspect Rₜ first.
Then inspect γV(Sₜ) − V(Sₜ₋₁).
Classify the signal according to which terms contribute.

Your attempt (not saved)

MEDIUM

Suppose a designer rewards an agent for performing an action that often helps achieve a goal, but does not reward the goal itself. What behavior might the agent learn to maximize, and what question should the designer ask about the reward signal?

Hints

Agents learn to maximize the reward they receive.
Compare the rewarded action with the actual desired outcome.
Ask whether maximizing the reward reliably represents success.

Your attempt (not saved)

The Essential Picture

1
A reinforcement signal is the number that most directly guides changes in a policy or value function.
2
In a TD error, Rₜ is the primary reinforcement contribution.
3
The value-difference term γV(Sₜ) − V(Sₜ₋₁) is the secondary reinforcement contribution.
4
A TD error can contain pure primary reinforcement, pure secondary reinforcement, or a mixture of both.
5
Because agents learn to maximize received reward, reward design should represent the actual desired outcome rather than a convenient subgoal or preferred method.

Key Takeaways

The reinforcement signal directs the size and direction of learning updates.
A TD error combines immediate reward with a temporal-difference value contribution.
Primary reinforcement is Rₜ; secondary reinforcement is γV(Sₜ) − V(Sₜ₋₁).
The same TD-error expression can represent pure primary reinforcement, pure secondary reinforcement, or a mixture.
A reward signal should formalize the actual goal, because the agent learns to maximize the reward it receives.

More concepts starting with R