Action-value update rule
The n-step target is a quantity used by an action-value update rule.
Two quantities to track
The key distinction is between the n-step target and the action-value update rule. The n-step target is a quantity used by an action-value update rule. It supplies a learning quantity, while the update rule determines how that quantity is used to update an action-value.
Target-to-update flow
First, an n-step target is obtained. That target is then supplied to an action-value update rule. The action-value estimate is the quantity that is updated as a result. In other words, the target is an input to the update process, not the name of the process itself.
Separating the two roles
Suppose an n-step target has already been obtained for an action-value estimate.
- 1Identify the target
The n-step target is the learning quantity that has been computed.
- 2Identify the rule
The action-value update rule is the mechanism that uses the target.
- 3Identify the changed quantity
The action-value estimate is the quantity updated by the rule. The target itself is not renamed as the update rule.
The target is an input to the action-value update rule, and the action-value estimate is what the rule updates.
What changes after the target exists
Once the n-step target has been obtained, the important question is what happens next. The target does not by itself describe the complete learning operation. It is used with the usual action-value update rule from n-step Sarsa. This is why a description of a target must be followed by a description of the update rule that consumes it.
When reading or explaining this formulation, name the target and the update separately. A clear explanation should say both what quantity was obtained and that the quantity is then used by the action-value update rule.
Tree Backup and n-step Sarsa
At the level stated in the source, the Tree Backup Algorithm uses its target with the usual action-value update rule from n-step Sarsa. The shared point is the action-value update rule. The target is the quantity supplied to that rule. The source does not provide the internal construction details needed to describe the target paths more specifically, so the safe distinction is that target construction and target use are separate parts of the explanation.
The zero-factor convention
A product of zero factors is defined as 1. This convention applies when a product in the target formulation contains no factors. The empty product therefore contributes 1 rather than 0.
An empty product
A boundary case calls for a product, but there are zero factors to multiply.
- 1Count the factors
There are no factors in the product.
- 2Apply the convention
A product of zero factors is defined as 1.
- 3Use the result
The boundary case contributes 1 wherever that empty product appears in the target or update formulation.
The empty product has value 1.
Common interpretation mistakes
Common mistake Calling the n-step target the update rule.
What it looks likeThe target has been computed, so the update is complete.
Why it failsThe source distinguishes the target from the action-value update rule. The target is a quantity used by the rule.
Do this insteadSay that the target is supplied to the action-value update rule, which then updates an action-value estimate.
Common mistake Describing only the target and omitting what happens afterward.
What it looks likeAn explanation stops immediately after defining the n-step target.
Why it failsThe important question includes what happens after the target has been obtained.
Do this insteadContinue by identifying the usual action-value update rule that uses the target.
Common mistake Treating an empty product as zero.
What it looks likeThere are no factors, so the product contributes 0.
Why it failsThe stated convention defines a product of zero factors as 1.
Do this insteadUse 1 for the empty product.
Common mistake Claiming more detail about Tree Backup and n-step Sarsa than the source states.
What it looks likeAn explanation invents specific target-construction paths that are not provided.
Why it failsThe source establishes the use of the usual action-value update rule but does not supply those internal details.
Do this insteadState only that the Tree Backup target is used with the usual action-value update rule from n-step Sarsa.
Check your understanding
Explain the sequence in your own words: an n-step target is obtained, and then an action-value update rule is applied. Identify which quantity is the input and which quantity is updated.
Hints
- The target supplies a learning quantity.
- The action-value estimate is the quantity changed by the rule.
- Do not use target and update rule as synonyms.
In a boundary case with a product containing zero factors, what value should the product contribute? Then state how the Tree Backup target is related to the usual action-value update rule from n-step Sarsa.
Hints
- Use the stated empty-product convention.
- The relationship concerns how the target is used after it is obtained.
Essential distinctions
- 1The n-step target is a quantity used by an action-value update rule.
- 2The target is not the same thing as the update rule.
- 3The update rule uses the target to update an action-value estimate.
- 4In the stated Tree Backup formulation, the target is used with the usual action-value update rule from n-step Sarsa.
- 5A product of zero factors is defined as 1.
Key Takeaways
- Separate target construction from action-value updating.
- Treat the n-step target as an input to the update rule, not as the update itself.
- The stated Tree Backup relationship is that its target is used with the usual action-value update rule from n-step Sarsa.
- Use 1 for a product containing zero factors.