Concepts / Estimated Action Values

Estimated Action Values

Exploration chooses a nongreedy action to improve knowledge about its value.

  • Machine Learning

The Choice Behind Every Action

When several actions are available, a learner must decide whether to choose the action that currently appears best or choose another action to learn more about it. The first choice uses current knowledge to pursue reward now. The second may give up some immediate reward so that later decisions can be better informed. This is the exploration-exploitation trade-off.

Reading Current Action Estimates

An estimated action value represents the learner's current view of how rewarding an action is expected to be. When one action has the highest current estimate, that action is called greedy. A nongreedy action is any available action that does not currently have the highest estimate.

Action Ahighest current estimatehighest estimategreedy choiceAction Blower current estimatelower estimatenongreedy choiceAction Cuncertain estimateuncertain estimatenongreedy choicemaps tomaps tomaps to
How are actions mapped to their current estimated expected rewards, and which action has the highest estimate?
ChoiceWhat it doesTime scale
Greedy actionUses the action with the highest current estimateCurrent step
Nongreedy actionChooses an action that does not currently have the highest estimateCan improve knowledge for later steps

Greedy and nongreedy actions are distinguished by their position among the current estimated action values.

Immediate Reward Versus Future Knowledge

Worked example

Choosing between a known option and an uncertain option

A learner has two available actions. Action A currently has the highest estimated value. Action B is nongreedy, but its value is uncertain and it might be better than the current estimate suggests. Which action should the learner choose?

  1. 1
    Evaluate exploitation

    Choosing Action A is exploitation. Because Action A is greedy, this choice focuses on maximizing the expected reward on the current step.

  2. 2
    Evaluate exploration

    Choosing Action B is exploration. This choice may sacrifice some immediate reward, but it can improve the learner's knowledge about Action B.

  3. 3
    Consider future decisions

    If Action B turns out to be better, improved knowledge about it can support higher reward on later steps, especially when many future decisions remain.

  4. 4
    Make the time-scale distinction

    The best choice depends on the objective being considered. Action A is appropriate when the goal is current-step expected reward. Action B can be valuable when learning may improve later decisions.

Result

Exploitation emphasizes immediate expected reward, while exploration considers whether information gained now can improve total reward later.

Greedy actionhighest current estimateImmediate expectedrewardprioritizedNongreedy actionuncertain valueImproved knowledgeabout the action's valueHigher later rewardpossible when the action isbetterpursuescan producecan support
How can choosing an uncertain action produce a lower immediate reward but a higher total reward later?

How Exploration Changes Later Decisions

Exploration is a decision to choose a nongreedy action in order to improve knowledge about its value. The important change is not only the reward received on the exploratory step. The learner also gains information that can affect the estimates used on later steps. If the explored action turns out to be better, that improved knowledge can support higher reward in the future.

Current estimatesone action appears bestNongreedy actionchosen to learnNew informationabout the action's valueLater estimatesknowledge has improvedLater decisionuses improved knowledgeguides a choicecan revealimprovessupports
What changes in the decision process when an exploratory action reveals information about its true reward?

What do you think happens?

An action is currently nongreedy but uncertain, and many future decisions remain. What is the main possible benefit of choosing it now?

  • It guarantees the highest immediate reward

  • It improves knowledge that may support better later decisions

  • It makes exploration and exploitation happen simultaneously

Reveal answer
Answer:

It improves knowledge that may support better later decisions.

Exploration can sacrifice some immediate reward to improve knowledge about a nongreedy action. If that action turns out to be better, the improved knowledge can support higher reward on later steps.

Knowledge Before and After Exploration

Consider a generated scenario with two actions. Action A currently has the highest estimated value, so it is greedy. Action B has a lower current estimate, so it is nongreedy, but the learner is less certain about its value. If the learner chooses B to explore, the immediate result may be less rewarding than the result expected from A. However, the observation provides information about B. The learner can then make later decisions with better knowledge than before.

Action Buncertain valueLimited knowledgebefore explorationAction Bobserved outcomeImproved knowledgeafter explorationhasprovides information for
How does taking a nongreedy action update what the learner knows about that action's value?
Best practice

When analyzing an action choice, ask two separate questions: Which action has the highest current estimated value, and how valuable could it be to learn about an uncertain action before making many future decisions? The first question identifies the greedy choice. The second evaluates whether exploration may improve total reward.

Mistakes in Action Selection

  • Common mistake

    Treating the action with the highest estimate as the best choice for every time scale.

    What it looks like

    Choosing the greedy action whenever an uncertain nongreedy action might be better and many future decisions remain.

    Why it fails

    The greedy action is best according to current knowledge for immediate expected reward, but exploration can improve total reward over later steps.

    Do this instead

    Separate the goal of maximizing the current step from the goal of improving future decisions through learning.

  • Common mistake

    Calling every nongreedy choice irrational.

    What it looks like

    Assuming that choosing an action with a lower current estimate can never be useful.

    Why it fails

    Exploration deliberately chooses a nongreedy action to improve knowledge about its value.

    Do this instead

    Ask whether the possible value of learning justifies sacrificing some immediate reward.

  • Common mistake

    Assuming exploration guarantees a higher total reward.

    What it looks like

    Expecting every exploratory choice to lead to a better later action.

    Why it fails

    Exploration can produce greater total reward when an uncertain action may be better, but the source describes this as a possibility rather than a guarantee.

    Do this instead

    Describe exploration as a way to improve knowledge and create the possibility of higher later reward.

  • Common mistake

    Confusing immediate expected reward with total reward over later decisions.

    What it looks like

    Comparing an exploratory choice only with the reward received on that one step.

    Why it fails

    Exploration may sacrifice immediate reward while improving knowledge that affects future choices.

    Do this instead

    State clearly whether the analysis concerns the current step or a longer sequence of decisions.

Practice the Trade-Off

MEDIUM

A learner has several available actions. One action currently has the highest estimated value. Another action has a lower estimate but is uncertain, and many future decisions remain. Explain the case for exploitation and the case for exploration. Then state what additional consideration determines which time scale should guide the decision.

Hints
  • Identify the greedy action from the current estimates.
  • Explain what immediate reward exploitation prioritizes.
  • Explain how exploration could improve knowledge about the uncertain action.
  • Distinguish current-step reward from possible total reward over later steps.
Worked example

A concise analysis

Action A has the highest current estimate. Action B is nongreedy and uncertain, and many future decisions remain.

  1. 1
    Identify the greedy action

    Action A is greedy because it currently has the highest estimated value.

  2. 2
    Describe exploitation

    Selecting Action A exploits current knowledge and focuses on immediate expected reward.

  3. 3
    Describe exploration

    Selecting Action B explores because it chooses a nongreedy action to improve knowledge about B's value.

  4. 4
    Compare time scales

    Because many future decisions remain, information about B may support higher reward later if B turns out to be better.

Result

Action A is the exploitation choice for current-step expected reward; Action B is the exploration choice when the value of learning may improve later decisions.

Key Takeaways

  1. 1
    A greedy action has the highest current estimated action value.
  2. 2
    Exploitation chooses a greedy action based on current knowledge and focuses on immediate expected reward.
  3. 3
    Exploration chooses a nongreedy action to improve knowledge about its value.
  4. 4
    Exploration can improve total reward when an uncertain action may be better and many future decisions remain.
  5. 5
    A sound analysis considers both the reward available now and the value of information for later decisions.

At a glance

  • Estimated action values express the learner's current expectations about available actions.
  • The greedy action has the highest current estimate; choosing it is exploitation.
  • A nongreedy action can be selected for exploration when learning about its uncertain value may improve future decisions.
  • Exploitation and exploration optimize different time scales: immediate expected reward versus possible total reward over later steps.
  • The exploration-exploitation trade-off requires weighing current reward against the value of improved knowledge.

Related Topics

Related concepts

More concepts starting with E