Sample-Average Action-Value Methods
Initial estimates affect action-value methods because they provide the starting values used before experience is available.
Why Starting Values Matter
An action-value method must have an estimate for an action before that action has produced any experience. That starting value is called an initial action-value estimate. It is not necessarily neutral: before enough experience has been collected, it can influence which actions the method selects and therefore influence later behavior.
Initial-estimate bias is the influence that starting action-value estimates have on an action-value method's behavior and later estimates before experience has removed or reduced that influence.
The useful question is not whether initial estimates matter at all. They do matter at the beginning. The important question is how long their influence remains under the update method being used.
Sample Averages Forget the Start
A sample-average method updates an action's estimate from the rewards observed for that action. Each new experience contributes to the estimate, and the estimate is based on the observed sample rather than continuing to preserve the initial value as a separate influence.
Tracing One Action
Suppose one action begins with an initial estimate, then produces experience. How does a sample-average method's dependence on that initial estimate change?
- 1Before selection
The action has no experience yet, so the method needs its initial action-value estimate.
- 2After the first selection
The action now has experience. The observed reward contributes to its sample-based estimate.
- 3After further selections
More observed rewards determine the estimate, while the initial estimate no longer remains as a continuing source of bias in the sample-average method.
- 4After every action has been selected
The condition stated by the source for losing the initial-estimate bias has been met: all actions have been selected at least once.
Sample-average methods lose their initial-estimate bias after all actions have been selected at least once.
Constant-α Remembers Gradually
A method using a constant α also allows experience to weaken the effect of the initial estimate. However, the source distinguishes this behavior from the sample-average case: the influence decreases but is not eventually erased. Later estimates continue to retain some dependence on the starting values.
Repeated Constant-α Updates
An action begins with an initial estimate and is updated repeatedly with a constant α. What happens to the initial estimate's influence?
- 1First update
The new experience changes the current estimate, but the initial estimate still contributes to the result.
- 2Later updates
Additional experience reduces the relative influence of the starting value.
- 3Longer run
The influence becomes smaller, but the source describes it as permanent rather than erased.
Constant-α methods retain a decreasing influence from their initial estimates: reduced does not mean gone.
Two Ways to Weight Experience
The central difference is how each method treats the starting value as experience accumulates. A sample-average method eventually loses its initial-estimate bias once every action has been selected at least once. A constant-α method lets later experience reduce the influence of the starting value, but continues to retain some dependence on it.
| Method | Early behavior | Longer-term influence of initial estimates | Key condition |
|---|---|---|---|
| Sample-average | Uses initial estimates before experience is available | Initial-estimate bias is lost | All actions have been selected at least once |
| Constant-α | Uses initial estimates before experience is available | Influence decreases but remains | Repeated updates reduce, but do not erase, dependence |
Qualitative comparison of initial-estimate influence
Prior Knowledge or Extra Parameter
Choosing initial estimates is a user decision. Setting every initial estimate to zero is still a choice, and it can introduce dependence on a starting assumption. In that sense, initial estimates are an extra parameter to choose.
The same choice can also be useful. If there is prior knowledge about expected rewards before experience is available, initial estimates can represent that knowledge. Initial estimates are therefore both a possible source of bias and a tool for expressing beliefs about expected rewards.
Treat an initial estimate deliberately. Use it to express prior knowledge when that knowledge is meaningful. Otherwise, remember that the selected starting value is an additional assumption that may affect early behavior.
Mistakes About Bias
Common mistake Assuming an initial estimate is neutral simply because it is used for every action.
What it looks likeGiving all actions the same starting estimate still determines the values used before experience is available.
Why it failsThe source states that choosing initial estimates is a user decision and that starting values can influence later behavior.
Do this insteadTreat even a uniform starting choice as an assumption that can affect early action selection.
Common mistake Claiming that sample-average methods lose the bias immediately after one action is selected.
What it looks likeOne action has experience, but another action has never been selected.
Why it failsThe source gives a condition involving all actions, not just one action.
Do this insteadSay that the bias is lost after all actions have been selected at least once.
Common mistake Saying that constant-α updates eventually erase the initial estimate.
What it looks likeLater experience has reduced the starting value's influence.
Why it failsThe source describes the influence under constant α as permanent but decreasing.
Do this insteadDescribe the influence as reduced, not gone.
Common mistake Treating initial estimates only as a problem.
What it looks likeA starting estimate is chosen using knowledge about expected rewards.
Why it failsThe source says initial estimates can deliberately represent prior knowledge.
Do this insteadRecognize both roles: initial estimates can cause bias and can encode useful prior knowledge.
Check Your Understanding
A method has selected some, but not all, available actions. First, decide whether a sample-average method has definitely lost its initial-estimate bias. Then decide whether a constant-α method's initial-estimate influence is gone or merely reduced. Finally, explain whether choosing initial estimates can ever be useful.
Hints
- Use the condition involving every available action.
- Separate decreasing influence from disappearing influence.
- Think about prior knowledge of expected rewards.
What do you think happens?
If one available action has never been selected, has a sample-average method already lost all initial-estimate bias?
Reveal answer
No, because all actions must have been selected at least once.
The source states that sample-average methods lose this bias after all actions have been selected at least once.
What do you think happens?
After repeated constant-α updates, what happens to the influence of the initial estimate?
Reveal answer
It decreases but remains.
The source describes constant-α bias as permanent but decreasing. Later experience weakens the dependence without erasing it.
Key Takeaways
- 1Initial action-value estimates provide the values used before an action has produced experience, so they can influence early behavior.
- 2This influence is called bias from the initial estimates.
- 3Sample-average methods lose this bias after all actions have been selected at least once.
- 4Constant-α methods reduce the influence of initial estimates over time but retain some dependence on them.
- 5Initial estimates are an extra user choice, but they can also encode useful prior knowledge about expected rewards.
At a glance
- Initial estimates matter because action-value methods need starting values before experience is available.
- Sample-average methods eventually lose their initial-estimate bias once every action has been selected at least once.
- Constant-α methods retain a decreasing, permanent influence from their initial estimates.
- Choosing initial estimates can add a parameter to the method or deliberately express prior knowledge.