Introduction to λ-return
The central change is not the creation of a new individual return. It is the combination of several existing n-step returns into one target.
From One Horizon to Many
An n-step return uses information from a particular backup horizon. Instead of selecting only one horizon, an update can use an average of several n-step returns. The resulting target is called a composite return. A λ-return is introduced as this kind of combined target: one target assembled from multiple simpler n-step targets.
The central change is not the creation of a new individual return. It is the combination of several existing n-step returns into one target.
Weighted Combination
A valid composite return uses positive weights for the component n-step returns. Those weights must add up to 1. Each return therefore contributes part of the final target, while the complete combination remains an average rather than an unscaled sum.
Checking a Composite Target
Suppose a learner combines a short-horizon return, a medium-horizon return, and a long-horizon return.
- 1Assign contributions
Give each of the three n-step returns a positive weight. For illustration, the contributions can be 0.2, 0.3, and 0.5.
- 2Check the total
The three weights add to 1.0, so they form a valid average.
- 3Form the target
Combine the three weighted returns into one composite target. The target contains information from all three backup horizons.
The combination is a valid composite return because every component has a positive weight and the weights add up to 1.
Preserved Error Reduction
The source describes an error reduction property for individual n-step returns and states that a composite return retains a similar property. The reason to care about the weighting rule is that the composite target is an average of the component returns: it does not simply combine them without scale. The averaged target can therefore preserve the useful error-reduction behavior while drawing information from multiple backup horizons.
Individual n-step target One backup horizon
A single n-step return has an error reduction property.
Composite λ-return Several weighted backup horizons
The averaged composite return retains a similar error reduction property.
A useful way to interpret the combination is to treat backup horizon as a choice of how much information to use. A short horizon contributes a nearer-term view, while longer horizons contribute information from farther along the backup. The composite return avoids committing the update to only one of these choices.
Backup Availability
A compound backup cannot be performed as soon as its shortest component is available. It becomes available only when the longest component included in the combination is complete. This timing condition follows from the fact that the composite target depends on every selected n-step return.
If a combination includes a long-horizon return, the shorter returns being complete is not enough to perform the compound backup. The longest selected component controls when the combined target is available.
Connections to Backup Algorithms
Averaging has a broader role than smoothing two targets together. It provides a way to create additional backup algorithms from existing n-step components. The source discusses averaging a one-step return and an infinite-step return as a way to relate temporal-difference and Monte Carlo methods.
The λ-return perspective treats backup methods as points that can be combined through weighted n-step returns, rather than requiring a choice of only one backup horizon.
Common Mistakes
Common mistake Treating the composite return as just one selected n-step return
What it looks likeChoosing the 2-step return and calling it the composite target
Why it failsA composite return is assembled from multiple n-step returns rather than selecting only one horizon.
Do this insteadIdentify all included n-step returns and describe how their weighted contributions form one target.
Common mistake Using weights that do not add up to 1
What it looks likeCombining component returns with weights whose total is greater than or less than 1
Why it failsThe source requires positive weights that add up to 1 for the combination to be a valid average.
Do this insteadCheck both conditions: every weight is positive and the complete set of weights sums to 1.
Common mistake Performing the backup when only the shortest return is complete
What it looks likeUpdating immediately after the 1-step component is available even though a longer selected component is unfinished
Why it failsThe compound backup is unavailable until its longest component is complete.
Do this insteadWait until the longest n-step return included in the composite target is complete.
Common mistake Assuming averaging removes the error-reduction property
What it looks likeConcluding that only an individual n-step return can retain error reduction
Why it failsThe source states that a composite return retains an error reduction property similar to individual n-step returns.
Do this insteadRecognize the composite return as an averaged target that preserves a similar property.
Check Your Understanding
A proposed composite target uses three n-step returns. The weights are positive, but their total is not 1. The longest return is also incomplete. Which two conditions prevent the compound backup from being treated as a valid, currently available composite backup?
Hints
- Check the rule for a valid average.
- Check which component determines backup availability.
- 1The proposed target fails both checks. Its weights do not form a valid average because they do not add up to 1, and the backup cannot yet be performed because the longest selected n-step return is incomplete.
Key Takeaways
- A λ-return combines multiple n-step returns into one composite target.
- The component returns must receive positive weights that add up to 1.
- The composite return retains an error reduction property similar to that of individual n-step returns.
- A compound backup is unavailable until the longest included n-step return is complete.
- Weighted combinations can create additional backup algorithms, including connections between one-step and infinite-step methods.