Truncated Importance Sampling Ratio

A common first impression is that an entire return must always be multiplied by an importance sampling ratio covering the whole episode. That approach treats the return as one indivisible object. Discounting-aware importance sampling starts with a more precise question: which actions could have influenced the part of the return being estimated? If the relevant return becomes determined at an earlier horizon, policy-probability factors from later actions are not needed for that partial return. Including them can add variability without adding useful information.

Think of a discounted return as being assembled from several partial views of the future. One component may stop early, while another extends farther. Discounting can be viewed as partial termination at several possible horizons: later portions contribute less because the return structure gives them less weight. This interpretation makes it natural to ask what would happen if each component were estimated separately.

This does not mean that the physical episode literally ends at every possible horizon. It is an interpretation of the return's structure. Each horizon identifies a portion that can be considered on its own, and the discount rate determines how strongly different horizons matter. When the discount rate is smaller than 1, recognizing these different horizons becomes especially important.

A flat partial return is a nondiscounted sum that stops at a selected horizon. For a starting time t and horizon h, write its structure as G_h,t = reward at t + reward at t+1 + ... + reward at t+h. The important features are that the rewards are not discounted within this component and that the sum ends at the chosen cutoff. The full discounted return can then be understood through components with different horizons rather than as one indivisible object.

Once a return is decomposed into flat partial returns, each component should use only the policy-probability information relevant to its own horizon. For a flat partial return ending at h, the corresponding truncated ratio is formed from the target-policy-to-behavior-policy probability ratios beginning at time t and continuing through time t+h. In symbolic form, ρ_h,t is the product of π(A_k|S_k) divided by µ(A_k|S_k) for the relevant k values from t through t+h. The product stops where the partial return stops.

Suppose a particular discounted-return component is already determined by an early portion of a trajectory. A full-episode ratio still multiplies in action-probability factors from later steps. Those factors are unrelated to that component's horizon. They can increase variance without changing the expected update in the relevant setting. Truncating the ratio removes these unnecessary later factors while preserving the policy information needed to scale the partial return.

This is the practical reason to avoid treating every return as indivisible. A full-episode ratio may be mathematically associated with the entire trajectory, but that does not make every factor relevant to every return component. Discounting-aware estimation respects the different horizons created by the discounted-return structure.