One-Step-Ahead Search
With q*, the one-step-ahead results are already available as state-action values.
Choosing Without Repeating the Search
Whenever an agent is in a state, it must choose an action. One possible method is to examine an action, consider what may happen next, and use those possible outcomes to judge the action. This is a one-step-ahead search. The optimal action-value function q* changes when that work happens: instead of repeating the look-ahead process at decision time, the agent can compare the q* values already available for the current state's actions.
The central operation is simple: in the current state, select any available action that maximizes q*(s, a).
Reading a State-Action Value
The expression q*(s, a) associates a value with taking action a from state s. The function stores the results of one-step-ahead searches as immediately available state-action values. Therefore, when the agent reaches s, it can inspect the values associated with the available actions instead of carrying out the look-ahead process again.
The diagram represents q* as a collection of values attached to the actions available from one state. The agent's selection task is to compare those values. The value does not require the agent to discover the successor state during this selection step, because the result needed for comparison is already stored in q*.
From Look-Ahead to Lookup
In the search approach, the agent considers an action, examines what may happen next, and uses the possible outcomes to judge that action. With q*, the one-step-ahead results are already available as state-action values. Selection becomes a lookup-and-compare process: identify the current state, inspect q* for its available actions, and choose an action with the largest value.
Selecting from Equal Maximum Values
Two Actions Share the Best Value
In a state, action a1 has q* value 6 and action a2 also has q* value 6. Which action can the agent select as an optimal action?
- 1Compare the values
The available actions have equal q* values: both a1 and a2 have value 6.
- 2Find the maximum
The maximum value among the available actions is 6.
- 3Apply the maximizing choice
Any action that maximizes q*(s, a) can be selected as an optimal action.
The agent can select either a1 or a2. No additional one-step-ahead search is needed because q* already supplies the values used for comparison.
When several actions share the maximum q* value, each of those actions is an optimal choice according to the maximizing rule.
Common Selection Mistakes
Common mistake Choosing an action without comparing the available q* values.
What it looks likeThe agent selects the first available action even though another action has a larger q*(s, a) value.
Why it failsThe optimal action is obtained by choosing an action that maximizes q*(s, a).
Do this insteadCompare the q* values for all available actions before selecting one.
Common mistake Assuming that equal maximum values make the choice invalid.
What it looks likeThe agent rejects a1 and a2 because both have q* value 6.
Why it failsBoth actions maximize the available q* values.
Do this insteadSelect either action that has the maximum value.
Common mistake Repeating a one-step-ahead search even though q* already provides the comparison values.
What it looks likeThe agent tries to enumerate successor states before deciding between actions whose q* values are already available.
Why it failsq* stores the results of one-step-ahead searches as immediately available state-action values.
Do this insteadUse q* to compare the current state's available actions directly.
Apply the Maximizing Rule
An agent is in state s. Three actions are available: a1 has q*(s, a1) equal to 3, a2 has q*(s, a2) equal to 8, and a3 has q*(s, a3) equal to 5. Which action should the agent select as an optimal action?
Hints
- Compare the three q* values.
- Select an action with the largest value.
For the practice values, a2 is the optimal selection because its q* value is the largest of the three. The example illustrates the same maximizing operation described by the source concept.
The Lookup-and-Compare Pattern
- 1q*(s, a) provides a value for taking action a from state s.
- 2q* stores the results of one-step-ahead searches as immediately available state-action values.
- 3At a state, the agent can select any available action that maximizes q*(s, a).
- 4Because q* already supplies the values for comparison, the agent does not need successor-state information or environment dynamics during this selection.
- 5If multiple actions share the maximum value, any one of them can be selected as an optimal action.
Key Takeaways
- q*(s, a) associates a stored value with an action taken from a particular state.
- One-step-ahead search evaluates what may happen next; q* makes the resulting values immediately available for selection.
- Optimal action selection using q* is a lookup-and-compare process.
- The agent chooses any available action with the largest q* value and does not need successor-state information for that selection.
- Ties are allowed: if multiple actions share the maximum value, any of them is optimal.