Reward-Modulated Synaptic Plasticity

A learning signal often arrives after the neural activity that should receive credit. A synapse may therefore need to retain information about an earlier activity event until a later reward or other learning consequence appears. An eligibility trace provides this bridge. It marks activity as relevant for a possible later change; it does not mean that the synapse must change immediately.

The history of actor-critic learning includes two forms of an actor-unit trace. An early form was written as A × φ(s). A later form was written as (A - π(A|S,θ)) × φ(s). These expressions show a historical change in what the actor trace records: the later form includes a policy-related term rather than only the product of the actor quantity and the state representation.

Eligibility can be described at several levels. Synaptically local eligibility is associated with an individual synapse. Neuronal eligibility is associated with activity at the level of an entire neuron. Computational actor-unit traces describe a related idea in an actor-critic learning rule. These are not merely different names for exactly the same object: they identify different levels at which information about activity and possible consequences is retained.

This distinction also appears in the historical development of the ideas. Klopf's ideas about local synaptic eligibility inspired the actor-critic algorithm implemented by Barto, Sutton, and Anderson as an artificial neural network with a single neuron-like actor unit. Crow proposed a related but different form in 1968, in which contingent eligibility was associated with entire neurons rather than individual synapses.

In an actor-critic learning rule, neural activity contributes to an actor-unit trace. The critic supplies a prediction-related part of the learning process, while a reward or other later learning signal supplies the consequence that makes earlier activity relevant. The trace allows the earlier activity to remain available for the learning update instead of requiring the consequence to arrive at the same instant.

The history of eligibility traces overlaps with research on reward-modulated synaptic plasticity. Reynolds and Wickens proposed a three-factor rule for plasticity in the corticostriatal pathway: dopamine modulates changes in corticostriatal synaptic efficacy. This comparison adds a modulatory factor to the activity-related factors involved in synaptic change.

STDP, or spike-timing-dependent plasticity, concerns synaptic plasticity that depends on the relative timing of pre- and postsynaptic spikes. Earlier evidence showed that the timing relationship between spikes matters for changing synaptic efficacy, and the definitive demonstration is attributed to Markram, Lübke, Frotscher, and Sakmann in 1997.

The relationship between STDP, eligibility, and temporal-difference ideas has not been treated as one settled equivalence. Rao and Sejnowski suggested that STDP could arise from a TD-like mechanism at synapses with non-contingent eligibility traces lasting about 10 milliseconds. Dayan later commented that this proposal would require an error like the one in Sutton and Barto's early model of classical conditioning rather than a true TD error. These remarks show that the connections have been actively debated.

Dopamine also appears directly in later work connecting reward modulation and spike timing. Pawlak and Kerr reported that dopamine is necessary to induce STDP at corticostriatal synapses of medium spiny neurons. Together with work on reward-modulated STDP, this places dopamine and spike timing within the broader discussion of how synaptic changes may depend on both activity and later signals.