Concepts / TD(λ) Algorithm and Eligibility Traces

TD(λ) Algorithm and Eligibility Traces

The central operational pattern is repeated: obtain the scalar TD error and the vector eligibility trace for the current step, use both to form a proportional weight-vector update, and apply that update to the weights.

  • Programming

Why Every Step Matters

TD(λ) changes the weight vector at every learning step. It does not wait until the end of a sequence or until a final batch of changes has been collected. Each step uses the information available at that moment to form and apply a new change to the weights.

Key point

The central operational pattern is repeated: obtain the scalar TD error and the vector eligibility trace for the current step, use both to form a proportional weight-vector update, and apply that update to the weights.

Two Quantities in One Update

A TD(λ) learning step brings together two different kinds of information. The scalar TD error supplies one overall error signal for the current step. The vector eligibility trace supplies the pattern used to distribute that signal across the weight vector. The resulting change to the weights is proportional to both quantities.

TD errorone scalarWeight updateproportional changeWeight vectorupdated each stepEligibility traceone vectorerror quantityeligibility patternapply
How do one scalar TD error and one vector eligibility trace combine to determine the direction and size of the weight-vector update?
QuantityShapeRole in the step
TD errorScalarProvides one overall error signal for the current step
Eligibility traceVectorProvides the pattern used to distribute the signal across the weights
Weight updateVectorCombines the scalar error with the vector eligibility pattern before changing the weights

The TD error and eligibility trace participate in the same update, but they carry different kinds of information.

Reading the Eligibility Trace

Read the eligibility trace alongside the weight vector. The trace is not a replacement for the weights. Instead, it participates in forming the weight update, and that update is then applied to the weight vector. Because the trace is a vector, it supplies a feature-indexed pattern for the change rather than a single undifferentiated quantity.

Feature 1Trace position 1eligibility valueFeature 2Trace position 2eligibility valueFeature 3Trace position 3eligibility valueindexed byindexed byindexed by
How are eligibility values associated with positions in the eligibility vector?

At the next learning step, the same relationship is used again with the quantities available for that step. The trace therefore belongs to the repeated-step process: it helps determine the current weight change, and the updated weights are ready for the next step.

The Step-by-Step Control Flow

  1. 1
    Observe the information for the current learning step.
  2. 2
    Use the current step to obtain the scalar TD error and the vector eligibility trace used by the update.
  3. 3
    Combine the scalar error signal with the vector eligibility pattern to form a proportional weight-vector update.
  4. 4
    Apply that update to the weight vector.
  5. 5
    Move to the next learning step and repeat the relationship with the quantities available then.
Current stepobserve informationTD errorscalar signalWeight updateproportional to bothWeight vectorapply changeNext steprepeatEligibility tracevector patterncomputeupdate or usescalar inputvector inputapplycontinue
What happens next in a TD(λ) step, from observing a transition to computing the error, updating the eligibility trace, and changing the weights?

A Worked Mental Model

Worked example

Separating Signal from Pattern

Imagine a TD(λ) step with one scalar TD error and an eligibility trace containing one value for each weight position. Explain what each quantity contributes before the weight vector is changed.

  1. 1
    Read the TD error

    Treat the TD error as one overall signal for the current step. It tells the update how much error information is available at this step.

  2. 2
    Read the eligibility trace

    Treat the eligibility trace as a vector pattern aligned with the weight vector. Its positions indicate how the scalar signal is distributed across the weight positions.

  3. 3
    Form the update

    Combine the scalar TD error with the vector eligibility trace to form a weight-vector change proportional to both quantities.

  4. 4
    Apply the change

    Apply the resulting vector update to the weight vector. This is the weight change for the current step, and the process is repeated on the next step.

Result

The scalar TD error supplies the overall error quantity, while the vector eligibility trace supplies the pattern across weights. Together they determine the weight-vector update.

Weight vectorbefore updateWeight vectorafter updateone TD(λ) step
What changes in the weight vector during one TD(λ) update, and how does the eligibility pattern determine the vector pattern of that change?

This mental model intentionally separates the jobs of the two inputs. The TD error is not itself the complete weight update because it is only a scalar. The eligibility trace is not itself the complete weight update because it provides the pattern but not the current scalar error signal. The update requires both.

Mistakes to Avoid

  • Common mistake

    Treating the TD error and eligibility trace as interchangeable.

    What it looks like

    Describing both as if they were the same kind of quantity.

    Why it fails

    The TD error is scalar, while the eligibility trace is a vector. They carry different shapes of information and have different roles.

    Do this instead

    Describe the TD error as the overall error signal and the eligibility trace as the vector pattern used to distribute that signal.

  • Common mistake

    Calling the eligibility trace the weight vector.

    What it looks like

    Saying that the trace itself is the set of weights being learned.

    Why it fails

    The trace participates in forming the update, but the update is applied to the separate weight vector.

    Do this instead

    Keep the trace and weight vector as separate objects in your explanation.

  • Common mistake

    Assuming TD(λ) waits until the end to change the weights.

    What it looks like

    Describing the algorithm as collecting information first and applying one final change later.

    Why it fails

    TD(λ) changes the weight vector at every learning step.

    Do this instead

    Look for the repeated update relationship: form the proportional update and apply it on each step.

  • Common mistake

    Mentioning both quantities without explaining proportionality.

    What it looks like

    Listing the TD error and eligibility trace but not stating how they affect the update.

    Why it fails

    Both quantities determine the weight-vector update; they are not background labels.

    Do this instead

    State that the scalar error and vector trace jointly form a proportional weight-vector change.

Check Your Understanding

MEDIUM

In your own words, describe one TD(λ) learning step. Identify which quantity supplies the overall error signal, which quantity supplies the vector pattern, what object is changed, and when the change is applied.

Hints
  • Start by naming the shape of the TD error.
  • Then name the shape and role of the eligibility trace.
  • Finish by explaining the repeated application to the weight vector.

What do you think happens?

If a description of a TD(λ) step mentions the scalar TD error but omits the eligibility trace, does it fully specify the weight-vector update?

  • Yes, because the scalar error is the complete update

  • No, because the vector eligibility pattern is also required

  • Yes, because the trace is only descriptive

  • No, because TD(λ) changes weights only after a final batch

Reveal answer
Answer:

No, because the vector eligibility pattern is also required.

The TD error supplies the overall error signal, while the eligibility trace supplies the vector pattern used to distribute that signal across the weight vector. The update is proportional to both.

Key Takeaways

  1. 1
    TD(λ) updates the weight vector at every learning step.
  2. 2
    The TD error is one scalar overall error signal for the current step.
  3. 3
    The eligibility trace is a vector that supplies the pattern for distributing the signal across the weights.
  4. 4
    The weight-vector update is proportional to both the TD error and the eligibility trace.
  5. 5
    The algorithm repeats this relationship on the next learning step.

At a glance

  • TD(λ) changes the weight vector on every learning step.
  • The scalar TD error and vector eligibility trace have different shapes and different roles.
  • The TD error provides the overall signal, while the eligibility trace provides the vector pattern.
  • Their proportional combination determines the weight-vector update.
  • The same update relationship is repeated at the next step.

Related Topics

Related concepts

More concepts starting with T