Average Reward Methods

The distribution is not an unrelated extra input. It is policy-dependent: start with a policy, determine the state distribution under that policy, and evaluate average reward over that distribution.

Programming

Start with the Policy

Average reward methods begin with a policy. That policy determines the distribution with which states occur under it. The average reward objective then evaluates the average reward over that policy-dependent state distribution.

How do rewards earned in different states combine according to the distribution of states visited under a policy?

Tracing the State Distribution

Worked example

Comparing Two Policies through Their Distributions

Consider two policies whose states occur with different policy-dependent distributions. Explain what the average reward objective evaluates.

1
Identify the policy
Treat each policy as the starting point of the evaluation.
2
Identify its state distribution
Each policy determines the distribution with which states occur under that policy.
3
Evaluate the rewards over that distribution
The average reward objective evaluates average reward over the distribution associated with the policy being considered.
4
Compare the policies
The comparison concerns how the objective ranks the policies, rather than a calculation of particular numerical rewards.

Result

The average reward objective evaluates each policy using the average reward over that policy's own state distribution.

Key point

The distribution is not an unrelated extra input. It is policy-dependent: start with a policy, determine the state distribution under that policy, and evaluate average reward over that distribution.

Adding Discounted Values

The proposed discounted objective follows the same policy-dependent state distribution, but it does not combine rewards in exactly the same way as the average reward description. It sums discounted values over the distribution with which states occur under the policy.

How do a policy's state values and its state distribution combine to produce the proposed discounted objective?

Worked example

Following the Proposed Construction

Describe the evaluation sequence for one policy under the proposed discounted objective.

1
Begin with the policy
The policy determines the distribution with which states occur.
2
Keep that distribution
The proposed objective uses this same policy-dependent distribution rather than replacing it with an unrelated distribution.
3
Use state values
Associate values with the states represented in the distribution.
4
Apply discounting
Combine those values using discounting.
5
Form the objective
The proposed discounted objective sums the discounted values over the policy-dependent distribution.

Result

The proposed objective combines two ingredients: the values associated with states and the distribution with which those states occur under the policy.

Comparing Policy Rankings

The central result is about ordering policies. Although the proposed discounted objective is written using discounted values summed over a policy-dependent state distribution, it orders policies identically to the undiscounted, or average reward, objective.

When do the average reward and proposed discounted objectives rank policies in the same order?

Why Discounting Is Not Rescued

The proposed construction changes how the objective is presented: it combines discounted values with the state distribution induced by the policy. However, this change does not produce a different policy ordering from the undiscounted average reward objective. Therefore, the proposal does not rescue discounting.

Which effects of discounting remain after values are combined with the policy's state distribution?

Key point

Using the policy's state distribution does not by itself make the proposed discounted objective rank policies differently. The source result is that its policy ordering remains identical to the average reward objective.

Common Reasoning Mistakes

Common mistake
Assuming that a new-looking objective must produce a new policy ranking.
What it looks like
The proposed discounted objective uses discounted values and a policy-dependent state distribution, so one might expect it to prefer different policies.
Why it fails
The source result states that it orders policies identically to the undiscounted average reward objective.
Do this instead
Evaluate an objective by the policy ordering it produces, not only by the way its expression is written.
Common mistake
Treating the state distribution as independent of the policy.
What it looks like
Describing one fixed distribution as though it automatically applies to every policy.
Why it fails
The policy determines the distribution with which states occur under it.
Do this instead
For each policy, first identify its policy-dependent state distribution.
Common mistake
Describing the proposed objective as ordinary average reward without mentioning values or discounting.
What it looks like
Saying only that it averages rewards over states.
Why it fails
The proposed objective sums discounted values over the policy-dependent distribution.
Do this instead
Mention both ingredients: the state distribution under the policy and the discounted values associated with those states.
Common mistake
Claiming that the proposal rescues discounting because it incorporates the state distribution.
What it looks like
Concluding that adding the distribution must remove the relevant problem with discounting.
Why it fails
The proposal does not produce a different policy ordering from the average reward objective.
Do this instead
State the result precisely: the proposed discounted objective does not rescue discounting.

Check Your Understanding

MEDIUM

A learner says: “The proposed discounted objective must rank policies differently because it uses discounted values.” Explain why this conclusion does not follow from the objective's construction.

Hints

Identify the distribution used by the proposed objective.
Ask what the source says about the ordering of policies.
Distinguish an objective's written form from the policy ranking it produces.

Your attempt (not saved)

What do you think happens?

If the proposed discounted objective and the average reward objective are compared by the policies they prefer, should their policy rankings agree or differ?

They should agree.
They should differ.

Reveal answer

Answer:

They should agree.

The proposed discounted objective orders policies identically to the undiscounted average reward objective.

Key Takeaways

The average reward objective evaluates average reward over the state distribution determined by a policy.
The proposed discounted objective sums discounted values over that same policy-dependent state distribution.
The important comparison is the policy ranking produced by each objective.
The proposed discounted and undiscounted average reward objectives order policies identically.
Because the policy ordering does not change, the proposed discounted objective does not rescue discounting.

More concepts starting with A