Temporal-Difference Prediction Methods
Bootstrapping means that an estimate already available to the method contributes to the learning of another estimate.
The Learning Pattern
Temporal-difference prediction methods are defined in this material through bootstrapping: learning one guess from another guess. The central question is not simply whether an estimate is correct. Instead, ask how one estimate contributes to improving another estimate.
Bootstrapping means that an estimate already available to the method contributes to the learning of another estimate.
Following the Estimates
To detect bootstrapping, trace the dependency between two estimates. First identify the estimate that the method is trying to update. This is the estimate being learned. Then identify the information that supports that update. If another estimate is part of that support, the method is learning one guess from another guess.
A Worked Dependency Trace
Identifying the Two Guesses
Suppose a method is trying to improve Estimate A, and the information used for that improvement includes Estimate B, which is already available to the method. Which estimate is being learned, and which estimate supports its learning?
- 1Identify the target
Estimate A is the estimate being updated, so it is the estimate being learned.
- 2Identify the support
Estimate B contributes to the basis for the update, so it is the supporting estimate.
- 3Name the pattern
Because one estimate contributes to learning another estimate, this is the bootstrapping pattern.
Estimate A is the target estimate, Estimate B is the supporting estimate, and learning A from B is bootstrapping.
This example does not claim that every temporal-difference method uses only two estimates or that the supplied material specifies a particular numerical update. It illustrates the dependency that the material tells you to trace: one estimate is updated, and another estimate contributes to that learning.
Target and Support Roles
The two estimates have different roles. The target estimate is the one the method is trying to improve. The supporting estimate is the other available guess that contributes to the target estimate's learning. Calling both of them simply an estimate can hide the direction of the dependency, so always name their roles separately.
When reading a method description, write down two labels: estimate being updated and estimate providing support. This prevents you from confusing the object of learning with the information that helps produce that learning.
Avoiding Misidentification
Common mistake Treating bootstrapping as merely making an estimate.
What it looks likeA method has an estimate, so bootstrapping is assumed automatically.
Why it failsThe defining feature is not the existence of an estimate. It is that one estimate contributes to learning another estimate.
Do this insteadTrace the dependency and identify both the target estimate and the supporting estimate.
Common mistake Checking only whether an estimate is correct.
What it looks likeThe analysis asks whether the supporting estimate is right before identifying its role.
Why it failsThe supplied material says to trace the direction of learning rather than asking whether an estimate is correct.
Do this insteadFirst ask which estimate is being updated, then ask what information contributes to that update.
Common mistake Assuming the supplied comparison gives the detailed advantages of TD methods.
What it looks likeA summary claims that TD methods are better than Monte Carlo and dynamic programming methods for a particular reason.
Why it failsThe supplied material states that TD methods have advantages over both, but it postpones the details and does not list them.
Do this insteadState only that the material establishes the existence of advantages; leave the specific reasons for later study.
Check Your Understanding
A method updates Estimate X. The information contributing to that update includes Estimate Y, which is already available to the method. Identify the target estimate, the supporting estimate, and the bootstrapping relationship.
Hints
- The target is the estimate being updated.
- The supporting estimate is the other guess contributing to that update.
- Describe the relationship as learning one guess from another guess.
What do you think happens?
If a method updates Estimate X using information that includes already available Estimate Y, is this the bootstrapping pattern?
Reveal answer
Yes, because one estimate contributes to learning another estimate.
Bootstrapping is identified by the dependency between the supporting estimate and the target estimate, not by particular numerical values.
What the Passage Establishes
The supplied material identifies bootstrapping as a key characteristic of temporal-difference methods. It also states that TD methods have advantages over Monte Carlo methods and dynamic programming methods. However, this passage does not list those advantages; their detailed explanation belongs to later study.
Key Takeaways
- Bootstrapping is the process of learning a guess from another guess.
- The target estimate is the estimate being updated.
- The supporting estimate is the available estimate that contributes to the target estimate's learning.
- The clearest way to recognize bootstrapping is to trace the dependency from the supporting estimate to the target estimate.
- The supplied material says TD methods have advantages over Monte Carlo and dynamic programming methods, but it does not specify what those advantages are.