Policy Gradient

The central search object is the policy defined by θ. A value estimate may help estimate a better direction, but value functions are not required by every policy-gradient method.

Programming

Following the Learning Signal

A policy-gradient method searches through policies defined by numerical parameters. Instead of treating a value estimate as the primary object being searched, it estimates a direction for changing the policy parameters so that the policy can perform better. The direction comes from the agent's interaction with its environment.

Key point

The central search object is the policy defined by θ. A value estimate may help estimate a better direction, but value functions are not required by every policy-gradient method.

The Actor-Critic Learning Loop

The policy-gradient actor-critic method combines two learning processes. The actor is a differentiable policy written as π(a|s,θ). It selects actions and changes the policy parameters θ. The critic is a differentiable state-value function written as v̂(s,w). It estimates the value of the current state and changes the value-function parameters w. The actor and critic therefore have separate parameter sets, traces, and step sizes.

How do the environment, policy-based actor, value-based critic, and parameter updates connect in one learning loop?

1
Begin with the current state S.
2
Select an action according to the current policy, written as A ∼ π(·|S,θ).
3
Execute the action and observe the next state S′ and reward R.
4
Use the reward, the next-state value estimate, and the current-state value estimate to compute the temporal-difference error δ.
5
Update the critic trace ew and the actor trace eθ.
6
Use the separate traces to change w and θ.

One Transition Through the System

Worked example

A single actor-critic transition

Trace what happens when the agent starts in state S, selects an action using its current policy, and then observes S′ and R.

1
Policy selection
The actor uses π(a|s,θ) to select an action from the current state. The selected action is generated by the policy currently defined by θ.
2
Environment response
The environment returns the next state S′ and reward R. These observations are the experience used by the learning process.
3
Shared error
The method compares the reward plus the discounted value estimate of S′ with the current state's value estimate. The resulting temporal-difference error is δ.
4
Separate traces
The critic updates ew using the current value-function gradient, while the actor updates eθ using the gradient of the log policy for the selected action. Each trace also retains a decayed contribution from earlier experience.
5
Separate parameter changes
The critic uses δ and ew to change w. The actor uses δ and eθ to change θ. The same δ reaches both branches, but the branches change different parameter sets.

Result

One interaction produces a shared learning signal but two distinct parameter updates: one for the value estimate and one for the policy.

The Shared Role of δ

The temporal-difference error δ connects the actor and critic learning processes. It compares the reward plus the discounted value estimate of the next state with the current state's value estimate. Once the traces have been formed, δ scales their influence. The critic uses that scaled influence to change w, while the actor uses it to change θ.

How does the temporal-difference error δ produced by the critic influence both the policy-parameter update and the value-function update?

Debugging

Suppose a value estimate looks wrong. The first branch to inspect is the critic path involving v̂, ew, β, and w. If the policy parameters look wrong, inspect the actor path involving π, eθ, α, and θ. If both paths are affected, inspect δ because both updates share it.

Eligibility Traces and Recent History

Eligibility traces prevent the update from depending only on the immediately current state or action. Each trace combines a decayed version of its previous value with a current gradient contribution. The critic trace uses the current value-function gradient. The actor trace uses the gradient of the log policy for the selected action. The λ terms control how much of the previous trace remains, while the current gradients add the present state's contribution.

How do eligibility traces preserve information about recently visited states or actions, and how does that information affect updates over subsequent steps?

Worked example

Why an earlier experience can still matter

Imagine that the agent visits one state, then moves to another state before an update is evaluated.

1
First visit
The first state's current gradient contributes to the relevant actor or critic trace.
2
Later visit
At the next step, the previous trace is not simply discarded. Its contribution is decayed according to λ, and the new state's gradient is added.
3
Current error
When δ is applied, the retained contribution from the earlier step can influence the current update along with the newer contribution.

Result

Eligibility traces carry a decayed record of recent states or actions, allowing current updates to reflect more than the most recent experience.

Two Updates, Two Parameter Sets

Learning branch	Parameters changed	Trace used	Step size	Gradient source
Actor	θ	eθ	α	Gradient of the log policy for the selected action
Critic	w	ew	β	Current value-function gradient

The actor and critic updates have a similar structure, but they act on different parameter sets and use different traces.

What changes in the actor's policy parameters θ versus the critic's value-function parameters w, and how do their update rules differ?

What the Method Searches

A policy-gradient method searches over parameter-defined policies. The numerical parameters θ define the policy being followed. Experience from interaction with the environment provides information for estimating a direction in which those parameters should move. Repeating this adjustment produces a sequence of parameter-defined policies rather than a search whose primary object is a value estimate.

What does the policy-gradient method search over, and how are the policy parameters θ related to the value-function parameters w?

Interaction matters because the improvement direction is estimated from what happens when the current policy acts in the environment. Without the resulting states and rewards, the method would not have the experience used to form the temporal-difference error and update its parameters.

Mistakes in Tracing Updates

Common mistake
Treating θ and w as the same parameter set
What it looks like
Assuming that improving the critic directly replaces the actor's policy parameters
Why it fails
The actor changes θ, while the critic changes w. They are separate learning processes.
Do this instead
Track the actor and critic branches separately before applying the shared δ signal.
Common mistake
Using the critic trace for the actor update
What it looks like
Applying ew when tracing the change to θ
Why it fails
The actor uses eθ, while the critic uses ew.
Do this instead
Match θ with eθ and α; match w with ew and β.
Common mistake
Ignoring the environment interaction
What it looks like
Describing the parameter update without mentioning S, A, S′, or R
Why it fails
The direction is estimated from the agent's interaction with the environment.
Do this instead
Start the trace with the current state, selected action, next state, and reward.
Common mistake
Assuming eligibility traces store only the current experience
What it looks like
Discarding the previous trace before adding the current gradient
Why it fails
Eligibility traces combine a decayed previous contribution with the current gradient.
Do this instead
Follow the λ-controlled decay and then add the present state's or action's contribution.
Common mistake
Assuming every policy-gradient method requires a value function
What it looks like
Treating v̂(s,w) as mandatory for the entire policy-gradient family
Why it fails
Value estimates can improve gradient estimates in some methods, but they are not required in all of them.
Do this instead
Describe the value function as part of the actor-critic combination, not as a universal requirement.

Check the Mechanism

MEDIUM

A learner says: “The actor and critic both use δ, so they must update the same parameters.” Explain why this is incorrect. Name the parameter set, trace, and step size used by each branch.

Hints

Begin with the actor's parameter set and trace.
Then identify the critic's parameter set and trace.
Remember that δ is shared, but the destinations of the updates are different.

Your attempt (not saved)

HARD

Trace a two-step interaction in words. Include the first state and action, the later state and reward, the decayed previous eligibility trace, the current gradient contribution, and the way δ affects the final actor and critic updates.