Action-Value Methods for Decision Making
Greedy action selection chooses an action with the highest estimated action value.
Choosing by Estimated Value
When a decision-making method has estimated action values, it needs a rule for choosing an action. Greedy action selection uses the simplest rule: compare the estimated values and choose an action with the highest estimated value. The decision depends on the ordering of the estimates: the action selected is one that reaches the maximum.
Tracing the Maximum
Selecting One Greedy Action
Suppose three available actions have these estimated values: Action A has 4, Action B has 7, and Action C has 5. Which action does greedy selection choose?
- 1Compare
Place the estimated values in order from lowest to highest: 4, 5, and 7.
- 2Find the maximum
The highest estimated value is 7.
- 3Map the maximum to an action
The value 7 belongs to Action B, so Action B reaches the maximum.
- 4Select
The greedy rule selects Action B because it has the highest estimated action value.
Action B is selected.
The Greedy Rule
Greedy action selection chooses an action with the highest estimated action value.
The rule can be written as Qₜ(A*ₜ) = maxₐ Qₜ(a). Here, the estimated value associated with the selected action equals the maximum estimated value among the available actions. To apply the rule, compare the estimates, find the maximum, and select an action that reaches it.
- List the available actions and their estimated action values.
- Compare the estimated values.
- Find the maximum estimated value.
- Select an action whose estimated value reaches that maximum.
Resolving Equal Maxima
A maximum does not always belong to only one action. If two or more actions share the highest estimated value, they are all greedy actions. The greedy rule allows selecting one of the actions tied at the maximum.
A Tie at the Maximum
Suppose Action A has an estimated value of 6, Action B has 6, and Action C has 3. Which actions can greedy selection choose?
- 1Compare
The estimates are 6, 6, and 3.
- 2Find the maximum
The highest estimated value is 6.
- 3Identify every action reaching it
Both Action A and Action B have the maximum value of 6.
- 4Select
Greedy selection can choose Action A or Action B because both are tied at the maximum.
The possible greedy actions are Action A and Action B.
Common Selection Mistakes
Common mistake Selecting the action with the lowest estimated value.
What it looks likeAction A has 4 and Action B has 7, but Action A is selected.
Why it failsGreedy selection chooses an action with the highest estimated action value.
Do this insteadCompare the estimates and select an action that reaches the maximum.
Common mistake Stopping after finding one maximum without checking for ties.
What it looks likeAction A and Action B both have an estimated value of 6, but only Action A is treated as eligible.
Why it failsA tie at the maximum creates multiple greedy actions.
Do this insteadIdentify every action whose estimated value equals the maximum.
Common mistake Treating the action name as the selection criterion.
What it looks likeChoosing Action A because it appears first, even though another action has a higher estimate.
Why it failsThe selected action is determined by the ordering of the estimated values.
Do this insteadUse the numerical estimates to locate the maximum.
Common mistake Assigning an unspecified behavior to another action-selection method.
What it looks likeClaiming that a different method must select the lowest-valued action.
Why it failsThe greedy rule is defined here, but the source does not specify the details of other action-selection methods.
Do this insteadDescribe only the greedy criterion unless another method has been defined separately.
Apply the Selection Rule
Three actions have estimated values: Action A = 8, Action B = 5, and Action C = 8. Identify the maximum estimated value and list every action that greedy selection can choose.
Hints
- First compare 8, 5, and 8.
- Then identify every action associated with the maximum.
What do you think happens?
Which actions are eligible under the greedy rule when Action A = 8, Action B = 5, and Action C = 8?
Reveal answer
Action A or Action C
The maximum estimated value is 8, and both Action A and Action C reach it. A tie at the maximum creates multiple greedy actions.
Selection Checklist
- 1Greedy action selection chooses an action with the highest estimated action value.
- 2The maximum estimated value determines which actions are eligible.
- 3With one maximum, the action associated with it is the greedy selection.
- 4With a tie, every action at the maximum is a greedy action, and one of them may be selected.
- 5The reliable procedure is compare, find the maximum, and identify all actions that reach it.
Key Takeaways
- Greedy selection uses the highest estimated action value as its decision criterion.
- The selected action is found by mapping the maximum estimate back to its action.
- When multiple actions share the maximum, each is a possible greedy action.
- Always compare all estimates and check for ties before identifying the selection.