Value Estimation Across Episode Horizons

The h-truncated λ-return method does not process an episode once and wait until the end before making all updates. Instead, it expands its available information one horizon at a time. At horizon 1, it makes the updates possible from the earliest available data. At horizon 2, it revisits the episode with more data, revises the earlier target, and updates the next state as well. The same pattern continues as the horizon advances.

Before learning begins for a new episode, the algorithm starts with θ₀, the weight vector inherited from the end of the previous episode. When the data horizon reaches time step 1, the algorithm has observed R₁ and reached S₁. For the estimate at step 0, the available target is therefore the one-step return: it uses R₁ and then bootstraps from the estimate of S₁ generated with θ₀.

When the horizon advances, additional episode data allows targets for more earlier time steps. At horizon 2, the first update uses the improved target G λ|2 0 for the state at step 0, and the next update uses G λ|2 1 for the state at step 1. At horizon 3, the pass can use the targets for steps 0, 1, and 2. The essential change is not just the number of updates: the target available for an earlier state can also improve because the horizon contains more episode information.

A single symbol such as θ₁ becomes ambiguous when the algorithm makes several passes. The notation θʰₜ resolves this ambiguity. The superscript h identifies the horizon of the pass, while the subscript t identifies the time position within that pass. Thus, θʰₜ records both where the vector occurs inside a pass and which horizon-specific pass is being considered.

Every horizon-specific sequence begins with the inherited θ₀. The vector at the end of the pass is written θʰ, meaning θʰₕ. It is the final vector produced after all updates available at horizon h have been performed.

This trace illustrates the central bookkeeping rule. The algorithm revisits the beginning because later data can construct targets for more earlier states. It also restarts from θ₀ so that the horizon-specific pass is defined consistently. The final vector of a previous horizon is not used as the starting vector of the next horizon within the same episode.

Within one episode, the method repeatedly expands its horizon, revisiting earlier states as more information becomes available. Each pass starts from the inherited θ₀, uses the targets available at its own horizon, and ends with a horizon-specific final vector. At the complete horizon T, the final weights are θᵀ. Those weights become the θ₀ inherited by the next episode, connecting repeated passes within an episode to learning across episodes.