Exploration vs Exploitation
Exploitation can be the best choice for the next step because it uses the action with the greatest current estimated value.
One Choice, Two Purposes
When an agent must choose an action, it has two different ways to use its current situation. It can rely on what it currently believes about the actions and seek the best expected reward for the next step. Or it can choose an action to learn more about it. These purposes create the exploration-versus-exploitation problem.
A single action selection cannot both explore and exploit. Selecting the currently highest-valued action pursues immediate reward; selecting a different action creates an opportunity to improve knowledge about that action.
Finding the Greedy Action
The agent has an estimated value for each available action. The greedy action is the action with the greatest current estimated value. “Greedy” describes the choice at the present decision, not a guarantee that the action will always produce the greatest reward.
Comparing three estimated action values
Suppose an agent currently estimates the values of three actions as A = 4, B = 7, and C = 5. Which action is greedy?
- 1List the estimates
The current estimates are 4 for A, 7 for B, and 5 for C.
- 2Locate the greatest estimate
The greatest current estimated value is 7.
- 3Identify the action
Action B has the greatest estimated value, so B is the greedy action in this situation.
Action B is greedy because its current estimated value is greatest.
Exploitation and Exploration
Exploitation means choosing the greedy action: the agent uses its current knowledge to pursue the best expected one-step reward. Exploration means choosing a nongreedy action: the agent gathers information that may improve its estimates later.
| Choice | Action selected | Primary purpose |
|---|---|---|
| Exploitation | Greedy action | Pursue the best expected one-step reward |
| Exploration | Nongreedy action | Gather information that may improve later estimates |
The distinction is based on both the selected action and the purpose of the choice.
Short-Term and Long-Term Outcomes
Exploitation can be the best choice for the next step because it uses the action with the greatest current estimated value. Exploration can be worthwhile when many future selections remain: trying a nongreedy action may reveal that it is better than currently estimated, allowing improved choices later.
A short-term sacrifice for later knowledge
An agent estimates that Action A is better than Action B. It can choose A now for the best expected one-step reward, or choose B to learn more about B. How do the two choices differ?
- 1Exploit now
Choosing A follows the current estimate and aims at the best expected immediate reward.
- 2Explore now
Choosing B is nongreedy under the current estimates, but it creates an opportunity to improve the estimate for B.
- 3Consider future selections
If many future selections remain, information from B may support better choices later and may produce greater total reward.
Exploitation favors the immediate decision; exploration may improve the quality of later decisions.
Common Reasoning Errors
Common mistake Treating the greedy action as the action that is guaranteed to be best.
What it looks likeCalling an action greedy because it will certainly produce the greatest reward.
Why it failsA greedy action is defined by having the greatest current estimated value, not by a guarantee about its actual reward.
Do this insteadDescribe it as the action currently estimated to have the greatest value.
Common mistake Calling every nongreedy choice irrational.
What it looks likeAssuming that choosing an action other than the current maximum is always a bad decision.
Why it failsA nongreedy choice can be exploration, which may reveal a better action and improve later estimates.
Do this insteadAsk whether the choice is intended to gather information for future decisions.
Common mistake Assuming exploration is always better than exploitation.
What it looks likeChoosing exploration without considering the immediate reward or the number of future selections.
Why it failsExploration and exploitation serve different purposes and horizons.
Do this insteadSeparate the immediate-reward goal from the possible long-term value of improved knowledge.
Common mistake Claiming that one action selection both explores and exploits.
What it looks likeDescribing a single selected action as simultaneously greedy and nongreedy.
Why it failsSelecting the greedy action uses the current estimate for immediate reward, whereas selecting a nongreedy action creates an opportunity to learn about that action.
Do this insteadClassify the selected action as exploitation if it is greedy and exploration if it is nongreedy.
Practice the Decision
An agent has current estimated values of 3 for Action A, 6 for Action B, and 4 for Action C. Identify the greedy action. Then describe what it means to exploit in this situation and what it means to explore instead.
Hints
- Start by finding the greatest current estimated value.
- Exploitation selects the action with that greatest estimate.
- Exploration selects a nongreedy action to gather information that may improve later estimates.
A complete explanation should name the greedy action, connect it to exploitation, and explain that choosing another action would be exploration because it prioritizes information rather than the best expected one-step reward.
Key Takeaways
- A greedy action has the greatest current estimated value.
- Exploitation chooses the greedy action to pursue the best expected one-step reward.
- Exploration chooses a nongreedy action to gather information that may improve later estimates.
- Exploitation can favor immediate reward, while exploration can improve total reward when many future selections remain.
- One action selection cannot simultaneously serve both exploration and exploitation.