Cumulative Reward

Return turns a sequence of rewards into one quantity by applying a specified function to that sequence.

Programming

Why One Reward Is Not Enough

A reinforcement-learning agent receives rewards over time rather than usually receiving one single score. To describe what the agent is trying to optimize, we combine those rewards into a longer-term quantity called the return. The agent seeks to maximize expected return, not merely one isolated reward.

Key point

Return turns a sequence of rewards into one quantity by applying a specified function to that sequence.

Choosing the Starting Time

Begin by choosing a time step called t. The rewards received after that point form an ordered sequence: R t+1, R t+2, R t+3, and so on. The first reward in the sequence is the reward at t+1, not the reward at t. The return G t is defined from this sequence by a specific function.

Which rewards belong to the return when starting at time step t, and where does the sequence begin and end?

From Sequence to Return

Once the rewards after t have been identified, a specified function is applied to that ordered sequence. The result is one quantity: the return G t. There is more than one possible way to define this function. In the simplest case, the function adds the rewards together.

How does a sequence of individual rewards become one return through a specified function?

Definition

Return: the quantity produced by applying a specified function to the sequence of rewards received after a chosen time step.

Adding the Rewards

Worked example

A Three-Reward Sequence

Use the simplest return definition for the reward sequence 2, 5, 1.

1
Identify the sequence
The three rewards are the rewards being considered after the chosen time step.
2
Start with the first reward
The running total begins at 2.
3
Add the second reward
Adding 5 to the running total gives 7.
4
Add the third reward
Adding 1 to 7 gives 8.

Result

Under the sum definition, the actual return for the sequence 2, 5, 1 is 8.

How do the rewards accumulate step by step to produce the total return?

Key point

With the simplest return definition, the order identifies the reward sequence, while addition combines that sequence into its total.

Actual and Expected Return

Quantity	Meaning	Role
Actual return	The return produced from one particular observed reward sequence	Describes what that sequence produced
Expected return	The expected value of the return	The quantity the agent seeks to maximize

Both quantities concern return, but they differ in whether the focus is one particular sequence or the expected value of return.

For one particular reward sequence, applying the return definition produces one actual return. In the sequence 2, 5, 1, the actual return under the sum definition is 8. Expected return is different in emphasis: it is the expected value of the return, and this expected quantity is what the agent seeks to maximize.

What is the difference between the return from one observed reward sequence and the expected return the agent seeks to maximize?

Discounting Future Rewards

Added by the IKSHA team to go beyond the source course material.

The plain sum treats a reward received far in the future exactly like one received now. Many tasks need the opposite: a reward soon should count for more than the same reward later. Discounting does this by multiplying each later reward by a factor between 0 and 1 that is applied once more for every extra step of waiting. That factor is the discount rate, written γ (the Greek letter gamma).

Formula

The first reward is weighted by 1, the next by γ, the one after that by γ times γ, and so on: one more factor of γ for every extra step of waiting. With γ = 0 only the immediate reward counts. With γ = 1 the discounted return is the plain sum from the main article, which is only well defined when the sequence of rewards ends.

Worked example

A Discounted Return

An agent receives the rewards 5, 10 and 20 on three steps in a row. Find the return with γ = 0.9.

1
Weight each reward
The weights are 0.9^{^0} = 1, 0.9^{^1} = 0.9 and 0.9^{^2} = 0.81.
2
Multiply
5 * 1 = 5, 10 * 0.9 = 9 and 20 * 0.81 = 16.2.
3
Add
5 + 9 + 16.2 = 30.2.

Result

The discounted return is 30.2. The plain sum is 35, so waiting for the later rewards costs 4.8.

Computing the Return Backwards

Added by the IKSHA team to go beyond the source course material.

A discounted return can be written in terms of the next one: the return from time t is the next reward plus γ times the return from time t+1. This is why returns are usually computed from the end of an episode backwards, one addition and one multiplication per step.

Formula

python

rewards = [5, 10, 20]
gamma = 0.9

G = 0.0
returns = []
for r in reversed(rewards):
    G = r + gamma * G
    returns.append(G)
returns.reverse()
print([round(x, 2) for x in returns])

rewards = [5, 10, 20]
gamma = 0.9

G = 0.0
returns = []
for r in reversed(rewards):
    G = r + gamma * G
    returns.append(G)
returns.reverse()
print([round(x, 2) for x in returns])

Output (expected)

[30.2, 28.0, 20.0]

Notes on the code

Start from 0 after the last step, because nothing comes after it
Each pass applies G = r + gamma * G, so later rewards are discounted once more at every step back
The first value, 30.2, is the return from the very start: the same answer as the worked example

Common Reasoning Errors

Common mistake
Including the reward at time step t in the sequence that starts after t.
What it looks like
Treating R t as the first reward when defining the sequence after time step t.
Why it fails
The rewards after t are written as R t+1, R t+2, R t+3, and so on.
Do this instead
Begin the sequence with R t+1.
Common mistake
Treating the return as one isolated reward.
What it looks like
Using only the first reward after t instead of the complete reward sequence.
Why it fails
Return is defined from a sequence of rewards, not merely one isolated reward.
Do this instead
First identify the ordered sequence after t, then apply the specified function.
Common mistake
Assuming that every return must be calculated by addition.
What it looks like
Claiming that summing is the only possible return definition.
Why it fails
A return is based on a specified function, and more than one way to define that function is possible.
Do this instead
Use addition for the simplest case described here, while remembering that the function must be specified.
Common mistake
Confusing an actual return with expected return.
What it looks like
Saying that the agent's objective is to maximize the return from one particular observed sequence.
Why it fails
The agent seeks to maximize expected return, not merely one isolated actual return.
Do this instead
Use actual return for one particular sequence and expected return for the expected value of return.

Check Your Understanding

EASY

A chosen time step is t, and the rewards after that point are 3, 4, and 2. Under the simplest return definition, identify the reward sequence and calculate the actual return. Then state whether that result is an actual return or an expected return.

Hints

The sequence begins with the first reward after t.
For the simplest definition, add the rewards together.
A result from one particular sequence is an actual return.

Your attempt (not saved)

What do you think happens?

For the reward sequence 3, 4, 2, what is the actual return under the sum definition?

Reveal answer

Answer:

The simplest return is the sum of the rewards in the sequence: 3 + 4 + 2 = 9. Because this comes from one particular sequence, it is an actual return.

Key Takeaways

1
Return turns an ordered sequence of rewards into one quantity using a specified function.
2
When starting at time step t, the sequence begins with R t+1, followed by R t+2, R t+3, and so on.
3
The simplest return is the sum of the rewards in the sequence.
4
An actual return comes from one particular reward sequence.
5
The agent seeks to maximize expected return.

At a glance

Return combines a sequence of rewards into one quantity through a specified function.
The rewards after time step t are R t+1, R t+2, R t+3, and so on.
Under the simplest definition, add the rewards to obtain the return.
A particular sequence produces an actual return, while the agent seeks to maximize expected return.

More concepts starting with C