Average Reward Formulation and Semi-gradient Methods

Episodic n-step semi-gradient Sarsa is based on the forward Sarsa(λ) algorithm of van Seijen (2017).

Programming

From Episodic Learning to Continuing Control

This topic connects several ideas rather than presenting a complete implementation. The source identifies episodic n-step semi-gradient Sarsa as being based on the forward Sarsa(λ) algorithm of van Seijen (2017). It then turns to continuing control, where the objective must be reformulated around average reward per time step. The key skill is to follow these relationships without adding update equations or pseudocode that the source does not provide.

Two Control Objectives

The source distinguishes episodic control from continuing control. In the episodic case, parameterized function approximation and semi-gradient descent extend naturally to control. Continuing control is different because the agent continues acting rather than working toward the end of an episode. For that setting, the formulation is changed so that the objective is to maximize average reward per time step.

How does the objective differ between episodic control and control that continues indefinitely?

Key point

The important change is not merely a different name for the same objective. The source presents average reward per time step as the formulation needed for continuing control, whereas the episodic discussion concerns the discounted formulation and its extension with function approximation.

Why Approximation Changes Control

The source identifies a limitation of carrying the discounted formulation into approximate control: most policies cannot be represented by a value function in the approximate case. Control still requires policies to be compared and ranked, but a value-function representation is unavailable for most of those policies. This is why the source introduces a different role for the average-reward formulation.

What becomes unusable when the discounted formulation is carried into approximate control?

The Scalar η(π)

Definition

η(π) is the scalar average reward associated with policy π. In the source's formulation, this scalar provides a common quantity for ranking arbitrary policies.

The distinction between representation and ranking is central. A value function is a representation of values, and the source says that most policies cannot be represented in that way when approximations are present. η(π) serves a different purpose: it supplies one scalar by which policies can be ordered. The policy-ranking role does not require every policy to have its own usable approximate value-function representation.

How does the scalar η(π) rank policies without requiring a value-function representation for most policies?

Worked example

Ranking Two Policies with η(π)

Suppose two hypothetical continuing policies, Policy A and Policy B, are evaluated by their scalar average rewards. Explain how the policies can be ranked without constructing a usable approximate value-function representation for both.

1
Assign the policy quantities
Associate η(πA) with Policy A and η(πB) with Policy B. This is a conceptual illustration, not a numerical calculation from the source.
2
Compare the scalars
Compare η(πA) and η(πB). The policy with the larger scalar average reward receives the higher ranking under the average-reward formulation.
3
Separate the roles
The comparison ranks the policies through their scalar average rewards. It does not claim that each policy has a separately available approximate value function.

Result

η(π) changes the comparison task from requiring value-function representation for most policies to ordering policies by a common scalar average reward.

Algorithmic Relationships

Which algorithmic and historical relationships does the source explicitly establish?

The source states that episodic n-step semi-gradient Sarsa is based on forward Sarsa(λ), associated here with van Seijen (2017). Separately, it characterizes the algorithm described in the section as the on-policy analog of R-learning, which was introduced by Schwartz in 1993. These are relationships and historical attributions, not a complete derivation of the algorithms.

The average-reward formulation is not presented as belonging exclusively to one area. The source says it has been discussed in both dynamic programming and reinforcement learning. It also mentions an access-control queuing example associated with Carlström and Nordström (1997), but does not provide the queue's states, actions, rewards, or calculations.

Evidence and Implementation Boundaries

Established by the source	Not provided by the source
Episodic n-step semi-gradient Sarsa is based on forward Sarsa(λ).	A pseudocode listing for the algorithm.
The described algorithm is identified as the on-policy analog of R-learning.	Specific update equations.
Average-reward methods are discussed in dynamic programming and reinforcement learning.	A numerical execution trace.
η(π) can rank arbitrary policies.	Numerical policy data or a calculated average reward.
Continuing control is formulated around average reward per time step.	Details of a particular implementation.

Use this boundary to avoid turning bibliographical relationships into unsupported implementation claims.

Common mistake
Treating the source's algorithm relationships as a complete algorithm specification.
What it looks like
Claiming that the section supplies the exact update sequence for semi-gradient Sarsa.
Why it fails
The source pack explicitly says that pseudocode, update equations, and a step-by-step execution trace are not provided.
Do this instead
State the relationship that is given, then label any implementation detail as unspecified.
Common mistake
Confusing value representation with policy ranking.
What it looks like
Assuming that a policy cannot be compared unless it has an approximate value-function representation.
Why it fails
The source says most policies cannot be represented by a value function in the approximate case, while η(π) can still provide a scalar for ranking.
Do this instead
Explain that a value function represents values, whereas η(π) supplies a common scalar for ordering policies.
Common mistake
Using the access-control queuing example as if its numerical details were supplied.
What it looks like
Inventing queue states, actions, or reward calculations for the cited example.
Why it fails
The source only identifies the example's bibliographical association with Carlström and Nordström (1997).
Do this instead
Treat it as a historical reference unless independent details are supplied.
Common mistake
Saying that the average-reward formulation is only a reinforcement-learning idea.
What it looks like
Leaving dynamic programming out of the formulation's history.
Why it fails
The source says the formulation has been discussed in both dynamic programming and reinforcement learning.
Do this instead
Mention both settings when describing the formulation's scope.

Check Your Understanding

MEDIUM

Explain, in your own words, why the average-reward formulation is useful for approximate continuing control. Your answer should mention the limitation involving value-function representation and the role of η(π) in ranking policies.

Hints

Start by stating what changes when control is continuing.
Explain why most policies create a problem for approximate value-function representation.
Finish by describing what the scalar η(π) makes possible.

Your attempt (not saved)

EASY

Classify each statement as established by the source or unspecified by the source: the algorithm is an on-policy analog of R-learning; the exact update equation; the average-reward formulation appears in dynamic programming and reinforcement learning; the numerical average reward of the access-control queuing example.