Optimal Policies in Reinforcement Learning
The optimal value belongs to the state-level question. It does not commit the agent to one named first action.
The Best Possible Continuation
An optimal value function answers a focused question: starting in a particular state, what is the maximum value that can be obtained? It evaluates the best achievable outcome from that state after considering the available choices. The word optimal means that the later actions are selected to achieve the best possible result.
The optimal value belongs to the state-level question. It does not commit the agent to one named first action.
Reading State Values
Suppose an agent is in a particular state. Its optimal value describes the best result available from that point onward when the agent makes choices that achieve the best possible outcome. This value therefore summarizes the quality of the state under optimal decision-making. It is not merely the result of one predetermined action; it accounts for the choices available from the state.
The golf example makes this distinction concrete. An optimal value for a location answers how well the hole can be completed from that location when the available strokes are used in the best way. The value changes when the number of strokes available changes: a one-stroke contour covers only a small area near the hole, while two- and three-stroke contours reach progressively farther.
Interpreting a Golf State Value
Interpret an optimal value assigned to a golf state without fixing the first stroke.
- 1Start at the state
Treat the ball's location as the state from which the agent must plan.
- 2Consider available choices
The agent considers the available stroke choices rather than committing to one named first stroke.
- 3Continue optimally
After each choice, later strokes can be selected to achieve the best possible result.
- 4Interpret the value
The optimal value summarizes the best achievable outcome from that location under this optimal continuation.
An optimal state value evaluates the best result from the location as a whole; it is not the value of one specifically named first stroke.
State Value versus Action Value
| Function | Question it answers | First action |
|---|---|---|
| Optimal value function | What is the maximum value that can be obtained from this state? | Not fixed to one named action |
| Optimal action-value function | What is the value of this state after a particular first action has been chosen? | Fixed for the first action, with later choices made optimally |
The distinction is whether the first action is left open or already chosen.
An optimal action-value function evaluates a state after a particular first action has been chosen. The first action is therefore part of the question. Later actions can still be selected optimally, but the initial commitment remains fixed.
In the golf example, q∗(s, driver) describes the value of state s under the commitment to use a driver first. The driver must be used for that first stroke, while later strokes may be selected optimally from the available driver or putter choices. By contrast, v∗(s) asks for the best achievable value from s without naming the first stroke in advance.
Bellman Mutual Checks
Bellman optimality equations characterize optimal values by relating the value of each state to the choices available from that state. The equations express a state-by-state consistency requirement: an assigned optimal value must agree with the best choices and the resulting values of successor states.
The recycling robot example has two states: high and low. Because there are two states, the Bellman optimality description contains two equations, one for the high state and one for the low state. Each equation checks the value assigned to its state against the choices available there. Solving the equations gives the optimal values for both states.
Checking the Recycling Robot States
Explain how the Bellman optimality equations organize the two-state recycling robot example.
- 1List the states
The state space contains high and low.
- 2Write one relationship per state
The high state receives its own Bellman optimality equation, and the low state receives its own equation.
- 3Account for available choices
Each equation relates its state's value to the choices available there and the resulting state values.
- 4Solve the system
The two equations act as mutual checks. Solving them gives values that are consistent with optimal choices in both states.
Bellman optimality equations characterize the optimal values by requiring every state's value to agree with the best available choices.
From Values to Policy
Once the optimal value function has been obtained, it can be used to identify an optimal policy. For each state, the policy selects actions that achieve the maximum value. The policy is therefore the action-selection side of the value calculation: the values tell us how good the best attainable outcomes are, and the policy identifies actions that attain those outcomes.
In the gridworld example, solving the Bellman equation produces v∗, the optimal value function for the gridworld states. The corresponding optimal policy identifies actions that achieve the maximum value from each state. If several actions tie for the maximum, all of them can be optimal. This is why multiple arrows in one gridworld cell represent multiple optimal actions rather than an error.
Common Interpretation Errors
Common mistake Treating the optimal value function as the value of one fixed first action.
What it looks likeReading v∗(s) as though the agent had already committed to driver.
Why it failsThe optimal value function concerns the best achievable value from the state without a named first-action commitment.
Do this insteadUse q∗(s, driver) when the first action is fixed as driver; use v∗(s) for the unrestricted best value from the state.
Common mistake Assuming an optimal policy must select exactly one action in every state.
What it looks likeRejecting a gridworld cell because it has multiple optimal arrows.
Why it failsMore than one action may achieve the maximum value.
Do this insteadTreat all maximizing actions as optimal when they tie.
Common mistake Viewing Bellman equations as unrelated calculations for separate states.
What it looks likeSolving the high-state relationship without checking the low-state relationship in the recycling robot.
Why it failsThe equations form a system of mutual checks, with every state's value required to be consistent with the available choices.
Do this insteadConsider one Bellman optimality equation for each state and solve them as a connected system.
Check Your Understanding
A state has several available actions, and two of them achieve the same maximum value. Explain what the optimal value function says about the state and what the optimal policy may contain.
Hints
- Separate the state-level question from the action-selection question.
- Remember that an optimal policy may contain more than one action in a state when actions tie for the maximum.
In the golf example, explain why q∗(s, driver) is not the same question as v∗(s).
Hints
- Identify whether the first stroke is fixed.
- Later strokes may still be selected optimally in both interpretations.
Key Takeaways
- 1An optimal value function evaluates the best achievable outcome from a state.
- 2An optimal action-value function evaluates a state after a particular first action has been chosen.
- 3Bellman optimality equations relate each state's value to the available choices and characterize optimal values state by state.
- 4An optimal policy selects actions that achieve the maximum value.
- 5Several actions can be optimal in the same state when they tie for the maximum.
At a glance
- The optimal value function answers what maximum value can be obtained from a particular state.
- The optimal action-value function answers what value results when a particular first action has already been chosen.
- Bellman optimality equations provide mutually consistent, state-by-state relationships for optimal values.
- An optimal policy is obtained by selecting actions that achieve the maximum value, including multiple tied actions when applicable.