Semi-gradient prediction methods

Action-value prediction changes the training input from a state to a state-action pair.

Programming

The Input Changes

Semi-gradient prediction can be extended from predicting a value for a state to predicting an action value for a state-action pair. The central change is the training input. A state-value example uses a state alone, while an action-value example uses the state together with the action taken in that state.

How does the training input change when prediction moves from a state value to an action value?

Action-Value Inputs

A state can be associated with an estimated value, but action-value prediction must distinguish among actions available in that state. For this reason, the method receives both St and At as its example input. The action is not an optional label added after prediction; it is part of the input used to estimate the action value.

Worked example

Changing One Training Example

Represent the training input for state-only prediction and then represent the corresponding input for action-value prediction.

1
State-only example
The example is represented by the state St. The predictor is asked to estimate a value associated with that state.
2
Action-value example
The example is represented by the pair St, At. The predictor is asked to estimate the value associated with taking At in St.
3
What changed
The state remains part of the example, but the action is now included in the input because the prediction concerns an action value.

Result

The extension changes the example input from St to the pair St, At.

How do a state-action pair and the weight vector combine to produce the estimated action value?

The Approximate Predictor

Definition

The approximate action-value function, written as q-hat, is represented in parameterized form with a weight vector theta. The state-action pair is supplied to this parameterized predictor, which produces an estimated action value.

The weight vector is the adjustable part of the predictor. The general gradient-descent update uses the state-action example and its target to adjust this parameterized action-value predictor. The supplied source identifies this update structure but does not provide its full equation, so the essential information to track is the example, the target, the predictor, and the weight vector being adjusted.

What Ut Represents

In action-value prediction, Ut is the target associated with the training example St, At. It is a backed-up approximation of the action value at that state-action pair. The target may be the full Monte Carlo return, an n-step Sarsa return, or another approximation of q-pi at the state-action pair.

What information does Ut represent, and how does it connect the observed state-action pair to the desired prediction?

Target option	Role in the training example
Full Monte Carlo return	A possible target Ut for the state-action pair
n-step Sarsa return	A possible target Ut for the state-action pair
Another approximation of q-pi	A possible target Ut for the state-action pair

The source identifies these as possible forms of the action-value prediction target.

Following the Update

A useful way to follow the general update is to track the information it receives. First, identify the state St and the action At that form the training input. Next, identify Ut, the target associated with that pair. The parameterized approximate action-value predictor uses this example and target in a gradient-descent update that adjusts its weight vector theta.

Worked example

Tracing One Action-Value Example

A training example contains a state St, an action At, and a target Ut. Trace the role of each item without supplying a numerical update.

1
Identify the input
Combine St and At into the state-action example used by the action-value predictor.
2
Identify the target
Treat Ut as the backed-up approximation of the action value for that state-action pair.
3
Apply the general update
Use the state-action example and Ut to adjust the parameterized approximate action-value predictor.
4
Track the adjustable object
The adjustment is made to the predictor's weight vector theta.

Result

The update is organized around the pair St, At, its target Ut, the approximate action-value function, and its weight vector theta.

How does the action-value training information flow into the parameter adjustment?

Common Misreadings

Common mistake
Using only St as the input for an action-value example.
What it looks like
Treating an action-value training example as if it contained the state but not the action.
Why it fails
Action-value prediction must distinguish the action taken in the state, so the training input is the pair St, At.
Do this instead
Record both the state and the action whenever the prediction concerns an action value.
Common mistake
Treating Ut as the state-action input.
What it looks like
Describing Ut as the pair St, At.
Why it fails
The pair is the training input, while Ut is the target associated with that pair.
Do this instead
Keep the roles separate: St, At is the example input and Ut is its backed-up target.
Common mistake
Assuming Ut must always be one specific kind of return.
What it looks like
Allowing only the full Monte Carlo return as the target.
Why it fails
The source identifies the full Monte Carlo return, an n-step Sarsa return, and another approximation of q-pi as possible targets.
Do this instead
Recognize Ut as a target that may take one of the source-identified forms.
Common mistake
Confusing the weight vector with the training example.
What it looks like
Calling theta the state-action pair supplied to the predictor.
Why it fails
Theta represents the weights of the parameterized approximate action-value function.
Do this instead
Identify St, At as the input and theta as the adjustable weight vector.
Common mistake
Writing a full update equation that is not supplied by the source.
What it looks like
Presenting a specific numerical or symbolic gradient-descent equation as though it were given here.
Why it fails
The supplied source names the general update structure but does not include its full equation.
Do this instead
Explain the information flow: state-action example and target are used to adjust the parameterized predictor and its weights.

Check Your Understanding

EASY

A learner says: An action-value training example is St mapped to Ut. Correct the statement by naming the complete input and explaining the role of Ut.

Hints

Ask what must be added when prediction moves from a state value to an action value.
Separate the example input from the target.
Mention the parameterized approximate action-value predictor and its weight vector.

Your attempt (not saved)

What do you think happens?

If a prediction method changes from state values to action values, what is the main change in one training example?

The input changes from St to St, At
The target must always become a full Monte Carlo return
The weight vector becomes the training input
The state is removed from the example

Reveal answer

Answer:

The input changes from St to St, At.

Action-value prediction uses the state together with the action taken in that state. The target remains a backed-up approximation of the action value, and the parameterized predictor continues to use its weight vector.

What does each training example contain in state-value prediction versus action-value prediction?

Key Takeaways

Action-value prediction extends state-based prediction by changing the input from St to the state-action pair St, At.
The approximate action-value function q-hat is parameterized by a weight vector theta.
Ut is the target associated with St, At and may be a full Monte Carlo return, an n-step Sarsa return, or another approximation of q-pi.
The general gradient-descent update uses the state-action example and its target to adjust the parameterized action-value predictor.
The state-action pair, the target, the predictor, and the weight vector have separate roles and should not be conflated.

Related concepts

More concepts starting with S