Sarsa and n-step Sarsa

When reinforcement learning is studied with an average-reward objective, the familiar value-function vocabulary needs a parallel version. The source describes differential value functions as the average-reward counterparts of familiar value functions. This is not a cosmetic change: the average-reward setting introduces corresponding differential value functions, differential Bellman equations, and differential temporal-difference errors.

The same parallel idea applies to Sarsa. Semi-gradient Sarsa is useful when the action-value function is represented approximately rather than stored as a fully explicit table. With linear semi-gradient Sarsa and ε-greedy action selection, the reported behavior is especially important: the parameters enter a bounded region near the best solution instead of converging in the usual sense to one fixed point.

Average-reward reinforcement learning does not simply reuse every object from the usual formulation unchanged. Instead, it develops corresponding differential objects. Differential value functions are the value-function counterpart for the average-reward setting, and differential Bellman equations and temporal-difference errors form a parallel set of components. The organizing principle is continuity: the conceptual changes are small enough to form a parallel family, but the average-reward setting still requires its own definitions and algorithms.

Semi-gradient Sarsa becomes relevant when the action-value function is represented approximately rather than treated as a fully explicit table. A sampled transition is used to change the approximated action-value estimate, and the learning process changes the parameter vector that represents that approximation. The word semi-gradient identifies the algorithmic family; the important connection here is that Sarsa is being used with function approximation instead of only with a fully explicit table.

Sarsa and n-step Sarsa belong to the same broader family, but their backup targets differ in how far into the future the learning information reaches. The one-step view uses the immediate sampled transition and an estimate associated with the next step. An n-step view extends the backup across multiple steps, using rewards and estimates from farther into the future. In the average-reward setting, this parallel family includes differential versions of semi-gradient n-step Sarsa.

Under ε-greedy action selection, learning chooses between an action currently estimated as best and exploratory actions. Therefore, the sequence of sampled transitions is shaped by both the current estimates and the exploratory part of the policy. The source reports the resulting convergence behavior for linear semi-gradient Sarsa with ε-greedy action selection: the parameters enter a bounded region near the best solution rather than converging in the usual sense.

Entering a bounded region near the best solution means that the parameter values remain within a neighborhood of that solution. Converging in the usual sense would mean approaching one limiting parameter value. These descriptions are not interchangeable: the first permits continued movement inside a bounded neighborhood, while the second describes approach to a single limit.