Synaptic Plasticity and Reinforcement Signals

The important idea is not that reinforcement must arrive at the same moment as synaptic activity. Instead, a local eligibility trace keeps a recent event available for later evaluation.

Programming

The Delayed-Credit Problem

A central problem in reinforcement learning is temporal credit assignment: how can a system connect a past synaptic event with a reinforcement signal that arrives later? Miller's 1981 proposal addressed this problem with a Law-of-Effect-like learning rule containing synaptically local contingent eligibility traces. The trace preserves the relevance of a recent synaptic event until reinforcement can determine whether learning should occur.

How does a locally stored eligibility trace preserve a synapse's recent activity until a later reward determines whether the connection changes?

Key point

The important idea is not that reinforcement must arrive at the same moment as synaptic activity. Instead, a local eligibility trace keeps a recent event available for later evaluation.

Miller's Learning Rule

Miller's rule was Law-of-Effect-like: synaptic learning was related to a later reinforcement signal, rather than being determined only by the local synaptic event itself. Its distinctive mechanism was the contingent eligibility trace. The trace was synaptically local, meaning that the relevant record of recent activity was associated with the synapse where the event occurred. When reinforcement arrived, that stored eligibility could participate in determining whether the connection changed.

Worked example

A Delayed Reinforcement Episode

A synaptic event occurs first, and a reinforcement signal arrives later. How does Miller's proposal connect them?

1
Event
The synapse undergoes a relevant recent activity event.
2
Local trace
The synapse stores a contingent eligibility trace associated with that recent event.
3
Later reinforcement
A reinforcement signal arrives after the synaptic event rather than simultaneously with it.
4
Learning decision
The later signal can use the stored eligibility to determine whether the earlier synaptic event should contribute to synaptic change.

Result

The eligibility trace supplies a bridge between past local synaptic activity and later reinforcement.

The Critic-Like Sensory Analyzer

Miller's proposal also included a sensory analyzer unit with a critic-like role. The source characterizes this unit as a mechanism that provided reinforcement signals to neurons. This adds an evaluating component to the learning system: synaptic activity is not the only relevant process, because another mechanism supplies information that can reinforce or evaluate what happened.

How does sensory information reach the critic-like unit, and how does its evaluative signal influence synaptic learning?

Definition

Critic-like mechanism: a mechanism that evaluates or supplies reinforcement information to neurons, rather than being described merely as the synapse that undergoes change.

The Hedonistic Synapse

Seung's hedonistic synapse focuses on an individual synapse and its neurotransmitter release probability. The synapse changes that probability according to whether reward follows release or follows failure to release. In this view, the synapse is sensitive not only to its own release event but also to the reward relationship that follows it.

What changes at the synapse when reward follows release or follows failure to release?

Connections to Later Frameworks

The historical proposals can be understood as addressing the same broad coordination problem later seen in reward-modulated STDP and actor-critic methods: how should a system connect a past synaptic event with a later reinforcement signal? The source connects Miller's proposal with general features of reward-modulated STDP and connects the later actor-critic architecture with TD error. These are historical and conceptual relationships, not a claim that all of the mechanisms are identical.

Which components correspond across the historical proposal, reward-modulated STDP, and actor-critic methods?

Idea	Role in the source pack
Miller's proposal	Uses a Law-of-Effect-like rule with synaptically local contingent eligibility traces.
Reward-modulated STDP	Is connected by the source to general features of Miller's proposal.
Actor-critic methods	Later use a critic and TD error as part of the historical refinement.
Central question	How can a past synaptic event be connected with a later reinforcement signal?

These relationships identify shared learning roles without treating the frameworks as interchangeable.

Metaphor and Mechanism

Bacterial chemotaxis belongs in this topic as a metaphor or source of inspiration, not as the synaptic learning mechanism itself. The learning mechanisms discussed here are the eligibility trace, the critic-like reinforcement signal, and the reward-dependent change in neurotransmitter release probability. Keeping these levels separate prevents a biological analogy from being mistaken for a literal description of Miller's rule, the hedonistic synapse, reward-modulated STDP, or actor-critic learning.

Which features belong to the biological metaphor, and which belong to the abstracted synaptic learning mechanisms?

Best practice

When explaining the analogy, name the abstract learning operation explicitly. Say that chemotaxis is a metaphor or inspiration, then separately identify the trace, reinforcement signal, or release-probability update being discussed.

Mistakes in Reasoning

Common mistake
Treating the eligibility trace as the reinforcement signal.
What it looks like
The trace is described as the reward itself.
Why it fails
Miller's proposal separates the local record of recent synaptic activity from the later reinforcement signal.
Do this instead
Describe the trace as preserved eligibility and the critic-like mechanism as a source of reinforcement information.
Common mistake
Assuming that all three frameworks are identical.
What it looks like
Miller's 1981 rule, reward-modulated STDP, and actor-critic methods are presented as the same algorithm.
Why it fails
The source connects them historically and conceptually, but describes them as related ideas and later refinements.
Do this instead
Use the shared credit-assignment question to relate them while preserving their distinct names and roles.
Common mistake
Ignoring whether reward followed release or failure to release.
What it looks like
The hedonistic synapse is described as changing release probability based on reward alone.
Why it fails
The source emphasizes the relationship between reward and the preceding release outcome.
Do this instead
State whether the relevant preceding event was neurotransmitter release or failure to release.
Common mistake
Turning the chemotaxis metaphor into a literal synaptic mechanism.
What it looks like
A description of bacterial chemotaxis is used as though it directly specified the eligibility-trace update.
Why it fails
The metaphor and the inspired learning mechanism are different explanatory levels.
Do this instead
Separate the metaphor from the mechanisms explicitly named in the source.

Practice the Credit Assignment

EASY

A synaptic event occurs, a local eligibility trace is retained, and reinforcement arrives later. Identify which part records the past event, which part supplies later evaluation, and which part is ultimately influenced by the relationship between the two.

Hints

Look for the mechanism described as synaptically local.
Look for the critic-like mechanism that provides reinforcement signals.
The final learning target is the connection or synapse.

Your attempt (not saved)

MEDIUM

Compare these two descriptions: one says that reward follows neurotransmitter release; the other says that reward follows failure to release. Explain why the hedonistic synapse treats them as different cases, and identify the synaptic quantity that changes.