Actor-Critic Brain Implementation

A useful first interpretation says that dopamine is the brain's reward number: one central signal that tells every learning system how much reward has occurred. The hypothetical actor-critic implementation makes a more careful proposal. The ventral striatum provides value information to the VTA and SNpc, while dopamine neurons combine value information with reward-related information. Their resulting activity corresponds to a temporal-difference error, or TD error. This is a teaching signal about how current reward-related information compares with value information, not a single master reward scalar stored in one neuron.

In the proposed arrangement, the ventral striatum sends value information to both the VTA and the SNpc. Dopamine neurons then combine value information with reward-related information. The source description is intentionally a hypothetical implementation: it identifies information flow and signal computation without reducing the whole system to one reward-producing neuron.

A TD error is best understood here as a mismatch signal. Value information supplies one part of the comparison, while reward-related information supplies another part. When those sources differ, dopamine-related activity corresponds to the resulting TD error. The important point is not that dopamine is identical to reward; it is that dopamine-related activity reflects the relationship between value information and reward-related information.

The phrase brain's reward number can create a misleading picture. That picture suggests one central scalar reward signal, located in one place, that is broadcast to every system needing reward information. The hypothetical actor-critic implementation rejects that simplification. Dopamine activity corresponds to a TD error, while the reward-related contribution is distributed across many brain areas, neurons, and input channels.

The actor-critic interpretation connects dopamine-related activity to learning at cortical-to-striatal synapses. In this view, the dopamine-related activity functions as a teaching-related influence on the updating or learning associated with those synaptic contacts. The proposal therefore links three levels: value information in the hypothetical arrangement, dopamine-related activity corresponding to a TD error, and changes in learning at cortical-to-striatal synapses.

The critic side of the hypothetical implementation is associated with value information represented through the ventral striatum. The actor side concerns learning related to cortical-to-striatal synapses, where dopamine-related activity is hypothesized to influence updating. The TD-error-related activity connects these roles: value information contributes to the signal, and the resulting dopamine-related activity provides a teaching influence for learning.