Value Functions

The primary search object is a policy described by numerical parameters, not a value estimate. The central estimate in a policy gradient method is a direction for changing those parameters.

Machine Learning

From Values to Policies

Many reinforcement learning discussions begin by asking how good a situation or action is. Policy gradient methods begin from a different angle: they search directly through possible policies. Each policy is described by a collection of numerical parameters. The method uses the agent's interaction with its environment to estimate how those parameters should change if the policy is to perform better.

How do numerical parameters define the policies being searched?

Key point

The primary search object is a policy described by numerical parameters, not a value estimate. The central estimate in a policy gradient method is a direction for changing those parameters.

The Policy-Environment Loop

To trace a policy gradient method, follow information from the policy into the environment and back again. Numerical parameters first define the policy being used. The agent then interacts with the environment. Those behavioral interactions provide information that can be used to estimate a direction for improving the policy. The parameters are adjusted in that direction, producing another parameter-defined policy, and the search can repeat.

How does experience from the environment lead to a new policy?

The environment interaction matters because the improvement direction is not described as an arbitrary change to the parameters. It is estimated from the agent's behavioral interaction with the environment. Without that interaction, the method would not have the experience-based information described in the source.

What Gets Adjusted

Worked example

Tracing One Policy Update

Suppose an agent currently uses one policy defined by numerical parameters. Trace what changes when the method estimates that a different parameter setting would improve performance.

1
Start with parameters
The agent has numerical parameters that define the policy it currently follows.
2
Interact with the environment
The agent uses that policy while interacting with the environment. The resulting behavioral interactions provide information about how the policy is performing.
3
Estimate a direction
The method estimates a direction for changing the numerical parameters.
4
Adjust the parameters
The parameters are moved in the estimated direction. The meaningful change is in the numerical parameters, not a switch between two named algorithms.
5
Define another policy
After adjustment, the parameters define another policy. The method can repeat this search process.

Result

A policy gradient update changes the numerical parameters that define the policy, using information from interaction with the environment.

What is the meaningful state change during a policy gradient search?

Value Estimates and Gradients

Value function estimates can appear inside some policy gradient methods, but they are not required by every method in the family. Their role is supportive: they can improve the estimated gradient, meaning they can make the direction for changing policy parameters more useful. A value estimate and a policy gradient estimate are therefore related but not identical.

Where can a value-function estimate enter the policy-gradient process?

Common mistake
Treating the value function as the primary search object in every policy gradient method.
What it looks like
Saying that the method searches over value estimates rather than policies defined by numerical parameters.
Why it fails
The source identifies the policy, not the value estimate, as the object being searched through. It also states that value functions are not required by every method in this family.
Do this instead
Say that the method searches through parameter-defined policies. Add that some methods use value estimates to improve gradient estimates.
Common mistake
Equating a value estimate with the policy gradient estimate.
What it looks like
Describing a value estimate itself as the direction used to change policy parameters.
Why it fails
The value estimate can improve the gradient estimate, but the gradient estimate is the direction for changing the policy parameters.
Do this instead
Keep the roles separate: the value estimate may support the gradient estimate, while the gradient estimate supplies the parameter-change direction.
Common mistake
Ignoring environment interaction.
What it looks like
Describing parameter adjustment as if it happens without experience from the agent's behavior.
Why it fails
The source states that the direction is estimated from the agent's interaction with the environment.
Do this instead
Include the flow from policy to environment interaction, then from interaction to an estimated improvement direction.

State Value and Action Value

In a Markov decision process, value functions help describe how valuable a situation or decision is under a policy. A state-value function evaluates a state s under a policy π. It asks about the value of being in that state when the policy is the policy being considered.

The action-value function qπ(s, a) evaluates a more specific situation. It represents the expected return from starting at state s, taking action a, and thereafter following policy π. Its description fixes three parts: the starting state, the action taken first, and the policy followed afterward.

What changes when the evaluation includes one specific initial action?

Function	What is fixed or specified?	What is evaluated?
State-value function	State s and policy π	The state under that policy
Action-value function qπ(s, a)	Starting state s, first action a, and later policy π	The expected return for that specific initial action followed by the policy

The essential difference is whether a specific initial action is included.

Reading qπ(s, a)

Worked example

Unpacking the Action-Value Description

Interpret qπ(s, a) without losing track of what is fixed.

1
Identify s
The evaluation starts from the specified state s.
2
Identify a
The agent takes the specified action a first. This is the detail that makes action value more specific than state value.
3
Identify π
After that initial action, the agent follows policy π.
4
Read the result
The function evaluates the expected return for that complete description: start at s, take a, and thereafter follow π.

Result

qπ(s, a) is the expected return starting from state s, taking action a, and thereafter following policy π.

What is fixed when evaluating qπ(s, a), and how does the selected action lead into following policy π afterward?

Practice Check

MEDIUM

Explain the difference between a state-value function and qπ(s, a) in two or three sentences. Your answer should identify the starting state, the initial action when one is specified, and the policy followed afterward.

Hints

Begin with what a state-value function evaluates.
Then explain what extra detail appears in qπ(s, a).
Use the sequence: start at s, take a, then follow π.

Your attempt (not saved)

MEDIUM

A policy gradient method has a current policy defined by numerical parameters. Describe the information flow that can lead to a new policy. Include the environment interaction and explain where a value estimate may help.