Value Function Updates
The reinforcement signal is the number that most directly guides changes in a policy or value function.
The Number That Directs Learning
An agent does not change its policy or value function merely because a situation occurred. A numerical reinforcement signal guides the size and direction of that change. It may be positive, negative, or zero, and it acts as a multiplicative factor in a parameter update.
The simplest reinforcement signal is the reward itself, Rₜ. Temporal-difference learning uses a richer signal: the TD error. This combines the immediate reward with a change in predicted value, so learning can be directed by both what happened immediately and how the value estimate changed.
Breaking Down a TD Error
- 1Rₜ is the primary reinforcement contribution: the reward received at time t.
- 2γV(Sₜ) − V(Sₜ₋₁) is the secondary reinforcement contribution: the temporal-difference contribution from comparing the successor value with the previous value.
- 3δₜ is the complete reinforcement signal used for the TD state-value update.
The two terms should be understood as contributions to one signal, not as two independent learning signals. If both terms are present, the TD error is a mixture. If one term is zero, the remaining term determines the signal.
Reading the Bellman Equation
- 1Start with the state-action pair (s, a).
- 2Include the reward received when action a is taken in state s.
- 3Include the action values of the successor state-action pairs, discounted by γ.
- 4The policy π determines how successor actions contribute after the transition.
An action value is backed up from what can happen after a particular state-action pair. Taking action a in state s can lead to possible successor states s′ and rewards r. The action value qπ(s, a) therefore depends on the transition reward and on the action values available from the successor state. The policy π determines how the next action a′ is selected at s′.
Checking One State
Bellman verification is performed state by state. To check a proposed state-value equation at one state, substitute that state, list its neighboring outcomes, combine the relevant rewards with the values of the states reached from it, and compare the result with the proposed value. A successful check establishes consistency at that state; it does not establish the equation everywhere.
A local Bellman check
A proposed value function assigns a value to state s. The state has possible neighboring outcomes with known rewards and state values. Verify the equation at s.
- 1Identify the current state
Select the particular state whose proposed value is being checked.
- 2List neighboring outcomes
Record the rewards and successor-state values associated with the outcomes reachable from that state.
- 3Combine the outcomes
Use the Bellman equation to combine the relevant rewards and successor values.
- 4Compare
If the computed result equals the proposed value for s, the equation holds for this state.
The verification is local: it tests the Bellman relationship at the selected state, not automatically at every state.
Shifting Every Reward
Suppose the same constant c is added to every reward. The value of every state then increases by one common amount, written vᶜ. The values shift, but their relative ordering under a policy does not change. A state that was valued above another remains valued above it after the common shift.
Common Reasoning Errors
Common mistake Treating the reward as the only possible reinforcement signal.
What it looks likeAssuming δₜ must equal Rₜ.
Why it failsA TD error also contains the secondary contribution γV(Sₜ) − V(Sₜ₋₁).
Do this insteadWrite the complete TD-error expression before deciding which contributions are present.
Common mistake Calling primary and secondary reinforcement two separate learning signals.
What it looks likeUpdating once with Rₜ and again with γV(Sₜ) − V(Sₜ₋₁) as if they were independent signals.
Why it failsThey are conceptual parts of one TD-error reinforcement signal.
Do this insteadAdd the two contributions to obtain δₜ.
Common mistake Calling a TD error pure primary when the reward is nonzero but the value-difference term is also nonzero.
What it looks likeIgnoring γV(Sₜ) − V(Sₜ₋₁) because Rₜ is present.Why it failsBoth nonzero terms make the signal a mixture.
Do this insteadClassify the signal by checking whether each term is zero or nonzero.
Common mistake Assuming one successful Bellman check proves the value function everywhere.
What it looks likeVerifying state s and declaring all states verified.
Why it failsBellman verification is performed state by state.
Do this insteadRepeat the check for every state that must satisfy the equation.
Common mistake Assuming a common reward shift preserves the numerical values.
What it looks likeSaying that adding c to every reward leaves each state value unchanged.
Why it failsEvery state value increases by a common amount vᶜ.
Do this insteadDistinguish the changed absolute values from the unchanged relative ordering.
Apply the Checks
For each situation, identify the reinforcement pattern or value-function consequence. First, Rₜ is zero while γV(Sₜ) − V(Sₜ₋₁) is nonzero. Second, both terms in δₜ are nonzero. Third, every reward is increased by the same constant c. Fourth, a proposed Bellman equation matches at one state but has not been checked elsewhere.
Hints
- A zero reward removes the primary contribution but does not remove a nonzero secondary contribution.
- Two nonzero TD-error terms form a mixture.
- A common reward shift changes every state value by the same amount and preserves relative ordering.
- A check at one state establishes only a local verification.
- 1A reinforcement signal is the number that most directly guides a policy or value-function update. In TD learning, δₜ combines primary reinforcement Rₜ with secondary reinforcement γV(Sₜ) − V(Sₜ₋₁). Action values are backed up from transition rewards and successor action values, with the policy determining successor actions. Bellman equations must be checked state by state. Adding the same constant to every reward shifts every state value by a common amount while preserving the relative ordering of states.
Key Takeaways
- The reinforcement signal controls the size and direction of a policy or value-function update.
- A TD error combines primary reinforcement from the reward with secondary reinforcement from the temporal-difference value change.
- The action value qπ(s, a) is backed up from the reward and successor action values selected according to policy π.
- Bellman verification is local unless the equation is checked for every state.
- Adding a common constant to all rewards shifts all state values equally without changing their relative ordering.