Actor and Critic Learning Rules Explained
The actor's eligibility trace is contingent: it depends on both presynaptic and postsynaptic factors.
The Delayed-Credit Problem
A common first guess is that a synapse should change as soon as an action is produced. The difficulty is that the reinforcement signal evaluating that action may arrive later. At that later point, the learning system still needs a record of which synapses helped produce the action. An eligibility trace provides that temporary synaptic record.
A Trace That Waits
An eligibility trace is a temporary synaptic record. Earlier activity creates or increments the record, and the record remains available long enough for a later reinforcement signal to use it. If the trace is still non-zero when reinforcement arrives, the synapse is eligible to receive credit for the reinforced action.
Credit for an Earlier Action
An actor synapse participates in producing an action. The reinforcement signal evaluating that action arrives later, while the synapse's eligibility trace is still non-zero. How is credit assigned?
- 1Earlier activity
Activity in the actor helps produce the action, and the relevant interaction at the synapse creates or increments its eligibility trace.
- 2Temporary record
The trace preserves information about the synapse's earlier participation instead of requiring the reinforcement signal to arrive at the exact same moment.
- 3Later reinforcement
The reinforcement signal arrives while the trace is still non-zero.
- 4Credit assignment
Because the trace is available, the synapse is eligible for modification and can receive credit for the reinforced action.
The later reinforcement signal can modify the earlier participating synapse through its eligibility trace.
Actor Traces and Hebbian Change
The actor's eligibility trace is contingent: it depends on both presynaptic and postsynaptic factors. This connects the trace to the activity relationship emphasized by Hebb's proposal about synaptic change. In the actor rule, a synapse becomes eligible through the relevant interaction of activity before reinforcement is applied. The reinforcement signal then supplies the additional learning factor that determines whether that eligible synapse receives credit.
When explaining an actor trace, always name both activity factors. Calling it merely a record of presynaptic activity loses the defining feature of the actor learning rule: the trace is contingent on presynaptic and postsynaptic factors together.
Critic and Actor Compared
| Learning rule | Eligibility information | Learning-factor description |
|---|---|---|
| Critic | Depends only on presynaptic signals | Two-factor learning rule: reinforcement interacts with the presynaptic trace |
| Actor | Depends on presynaptic and postsynaptic activity | Three-factor learning rule: reinforcement combines with a trace reflecting two kinds of neural activity |
The main distinction is what information is used to form the eligibility trace.
Timing and Causal Order
The order of activity matters because the presynaptic factor must help cause the postsynaptic factor for credit assignment to be meaningful. Neural activation is also not instantaneous in realistic neural models. Therefore, forming a contingent actor trace requires attention to when the relevant activity becomes active, not merely whether both factors appear somewhere in the episode.
Common Reasoning Errors
Common mistake Assuming the synapse must change immediately when the action is produced.
What it looks likeThe action occurs first, while the reinforcement signal that evaluates it arrives later.
Why it failsThis ignores the delayed-credit problem in actor-critic learning.
Do this insteadUse the eligibility trace as a temporary record that remains available for the later reinforcement signal.
Common mistake Describing the actor trace as presynaptic-only.
What it looks likeTreating the actor rule as if presynaptic activity alone formed its eligibility record.
Why it failsThe actor trace depends on both presynaptic and postsynaptic factors.
Do this insteadIdentify the actor trace as contingent on both kinds of activity.
Common mistake Treating the critic and actor traces as identical.
What it looks likeSaying that both rules use the same eligibility information.
Why it failsThe critic trace depends only on presynaptic signals, whereas the actor trace depends on presynaptic and postsynaptic activity.
Do this insteadUse the presynaptic-only versus contingent distinction to compare the rules.
Common mistake Ignoring activation time and causal order.
What it looks likeConcluding that credit is meaningful whenever presynaptic and postsynaptic activity both occur.
Why it failsThe presynaptic factor must help cause the postsynaptic factor, and realistic neural activation is not instantaneous.
Do this insteadConsider the order and activation time of the factors when interpreting a contingent trace.
Check Your Understanding
A synapse in the actor helps produce an action. Presynaptic activity contributes to postsynaptic activity, and the actor's contingent trace is then available. Reinforcement arrives later. Explain why this synapse can receive credit, and state which two activity factors made its actor trace different from a critic trace.
Hints
- Start with the definition of an eligibility trace.
- Identify what happens before reinforcement arrives.
- Compare the information used by the actor and critic traces.
What do you think happens?
Which statement best describes the actor's eligibility trace?
Reveal answer
It depends on presynaptic and postsynaptic activity, with reinforcement arriving later.
The actor trace is contingent on both presynaptic and postsynaptic factors. It preserves eligibility so later reinforcement can assign credit to earlier activity.
Key Takeaways
- 1An eligibility trace is a temporary synaptic record that connects earlier activity to later reinforcement.
- 2The actor trace is contingent on both presynaptic and postsynaptic factors.
- 3The critic trace depends only on presynaptic signals, while the actor trace reflects two kinds of neural activity.
- 4For actor credit assignment to be meaningful, the presynaptic factor must help cause the postsynaptic factor.
- 5Because neural activation is not instantaneous in realistic models, the timing and causal order of activity must be considered.
At a glance
- Eligibility traces preserve a temporary record of earlier synaptic activity until a later reinforcement signal arrives.
- The actor uses a contingent trace formed from presynaptic and postsynaptic factors.
- The critic uses a presynaptic-only trace, making its learning interaction different from the actor's.
- Causal order and activation time determine whether activity provides meaningful credit for a later reinforced action.