The Hedonistic Neuron Hypothesis

The hedonistic neuron hypothesis applies reinforcement learning at the level of an individual neuron. Its central proposal is that a neuron can change the efficacies of its synapses according to what follows its own action potentials. Inputs associated with rewarding consequences should become more influential, while inputs associated with punishing consequences should become less influential. Modern research does not establish every detail of this proposal, but spike-timing-dependent plasticity and reward-modulated STDP provide a biologically plausible route toward this kind of outcome-sensitive change.

Spike-timing-dependent plasticity, or STDP, is a Hebbian form of plasticity with an additional temporal condition. Consider a synapse carrying an incoming presynaptic spike to a postsynaptic neuron. When the presynaptic spike arrives shortly before the postsynaptic neuron fires, the most studied form of STDP increases the synapse's strength. When the order is reversed and the presynaptic spike arrives shortly after the postsynaptic neuron fires, the synapse decreases in strength. STDP therefore distinguishes two relationships that would be treated alike by a rule that merely asked whether both neurons were active.

STDP supplies a timing-dependent candidate change, but it does not by itself provide the whole actor-like learning mechanism. Reward-modulated STDP adds a third factor: neuromodulatory input arriving after suitable presynaptic and postsynaptic timing. The three factors are presynaptic activity, postsynaptic activity, and a later reward or punishment-related neuromodulatory signal. The first two establish a potential synaptic modification; the third is needed for that potential change to become lasting.

A difficulty in reinforcement learning is that an outcome can occur after the neural activity that contributed to it. Reward-modulated STDP addresses this timing problem with synaptically local molecular traces. When closely timed presynaptic and postsynaptic spikes occur, a trace can leave that synapse eligible for a later modification. If neuromodulatory input arrives within the relevant time window, the earlier timing relationship can be converted into a lasting change. The trace therefore links an earlier local event with later reinforcement without requiring the reinforcement signal to arrive at exactly the same moment as the spikes.

The proposal starts with a single neuron rather than an entire animal or a centralized instructor. In the original formulation described in the source, rewarding and reinforcing information reached the neuron through synaptic input. That input could be part of the same general synaptic influence that excites or inhibits the neuron's spike-generating activity. Klopf also wanted to avoid depending on a centralized source of training information. The source notes a possible historical update: if Klopf had known what is now known about neuromodulatory systems, he might have assigned the reinforcing role to neuromodulatory input instead.

The actor in an actor-critic system changes the tendencies that produce actions. STDP supplies one ingredient for actor-like change because it can select synapses according to their recent contribution to a postsynaptic firing event. Reward-modulated STDP adds outcome sensitivity: a later neuromodulatory signal can determine whether a timing-dependent potential change becomes lasting. In this sense, synaptic plasticity can provide a biological route for connecting action-related activity with later outcome-related modulation.

The critic-like side of the analogy is represented by the later evaluation or reinforcement signal that distinguishes useful from harmful consequences. The analogy should be handled carefully. A mechanism that changes when a neuron becomes active is not enough for an actor-critic system. The actor-like part must change connections according to what happened before an action and whether that action was later associated with a useful outcome. The available biological evidence makes this kind of learning plausible, but neuroscience does not establish that the brain implements an actor-critic algorithm in every detail.