Advantages of Temporal Difference Prediction Methods
The supporting estimate and the target estimate have different roles. The target estimate is the one being learned or improved. The supporting estimate supplies part of the basis for that learning.
The Central Learning Pattern
Temporal-difference methods are characterized in the supplied material by bootstrapping. Bootstrapping means learning a guess from another guess. Instead of relying only on a final, independently established answer, the method allows one estimate to contribute to the learning of another estimate.
Bootstrapping is the process of learning a guess from another guess.
Following the Estimates
To detect bootstrapping, trace the direction of learning. First identify the estimate that the method is trying to improve. This is the target estimate: the estimate being updated. Then identify the information that contributes to that update. If another estimate is part of that basis, the method is learning one guess from another guess.
The supporting estimate and the target estimate have different roles. The target estimate is the one being learned or improved. The supporting estimate supplies part of the basis for that learning.
A Trace of Bootstrapping
Tracing Two Estimates
A method is improving Estimate A. The update uses Estimate B, which is already available to the method. Does this describe bootstrapping?
- 1Find the target
Estimate A is the target because it is the estimate the method is trying to improve.
- 2Find the support
Estimate B is the supporting estimate because information from it contributes to the update of Estimate A.
- 3Trace the dependency
The learning direction runs from Estimate B to Estimate A. One estimate is therefore contributing to the learning of another estimate.
- 4Name the pattern
Because the method learns one guess from another guess, the pattern is bootstrapping.
Yes. Estimate A is the target estimate, Estimate B is the supporting estimate, and the dependency between them is bootstrapping.
The example does not require Estimate B to be correct. Bootstrapping is identified by the role Estimate B plays in the learning process: it is another estimate that contributes to the update of Estimate A.
What the Comparison Establishes
| Method named in the material | What the supplied material establishes |
|---|---|
| Temporal-difference methods | Bootstrapping is identified as a key characteristic. |
| Monte Carlo methods | The material says TD methods have advantages over Monte Carlo methods, but does not list those advantages. |
| Dynamic programming methods | The material says TD methods have advantages over dynamic programming methods, but does not list those advantages. |
This table records only the comparison explicitly established by the supplied material.
Mistakes in Tracing Learning
Common mistake Looking only at whether an estimate is correct
What it looks likeA learner asks whether the supporting estimate is a correct answer before deciding whether bootstrapping is present.
Why it failsThe defining issue is the dependency between the supporting estimate and the target estimate, not whether the supporting estimate is correct.
Do this insteadIdentify the estimate being updated and then ask whether another estimate contributes to that update.
Common mistake Confusing the target estimate with the supporting estimate
What it looks likeA learner calls Estimate B the target even though Estimate A is the estimate being improved.
Why it failsThe target is the estimate being updated. The supporting estimate supplies part of the basis for that update.
Do this insteadTrace the direction of learning from the supporting estimate to the target estimate.
Common mistake Adding unprovided advantages to the comparison with other methods
What it looks likeA learner gives specific reasons that TD methods are better than Monte Carlo or dynamic programming methods even though those reasons are not in the supplied material.
Why it failsThe material states that TD methods have advantages over both, but postpones the details.
Do this insteadState only that the advantages are identified as existing; leave their detailed explanation for later study.
Practice the Dependency Test
A method updates Estimate X. The information used in that update includes Estimate Y, which is another estimate already available to the method. Identify the target estimate, the supporting estimate, and the learning pattern.
Hints
- Start with the estimate that is being updated.
- Then identify the other estimate that contributes to the update.
- Use the definition of learning a guess from another guess.
Practice Solution
A method updates Estimate X using information that includes Estimate Y.
- 1Target estimate
Estimate X is the target because it is the estimate being updated.
- 2Supporting estimate
Estimate Y is the supporting estimate because it contributes to the update.
- 3Learning pattern
The method is learning one guess from another guess, which is bootstrapping.
Estimate X is the target, Estimate Y is the support, and the dependency describes bootstrapping.
Key Takeaways
- 1Bootstrapping is learning a guess from another guess.
- 2In TD methods, one estimate can contribute to the learning of another estimate.
- 3The target estimate is the estimate being updated; the supporting estimate provides part of the basis for that update.
- 4The clearest way to identify bootstrapping is to trace the dependency between the supporting estimate and the target estimate.
- 5The supplied material says that TD methods have advantages over Monte Carlo and dynamic programming methods, but it does not provide the details of those advantages.
At a glance
- Bootstrapping means learning a guess from another guess.
- A TD method can use one estimate to support the learning of another estimate.
- To recognize the pattern, identify the estimate being updated and trace which other estimate contributes to that update.
- The supplied material establishes that TD methods have advantages over Monte Carlo and dynamic programming methods, while leaving the specific advantages for later study.