Model-Based Reinforcement Learning in the Brain

Imagine an animal that has learned an action leading to a valuable outcome. If that outcome becomes less valuable, a flexible animal should reconsider the action. This adjustment is characteristic of goal-directed behavior. Habitual behavior is different: an established action can continue even when its outcome is no longer as valuable. The evidence suggests that these two modes of control rely on different, though not necessarily exclusive, brain contributions.

Model-based and habitual control represent different kinds of guidance for behavior. In model-based behavior, the animal uses information about the environment and expected outcomes to support a flexible choice. In habitual behavior, a learned action can be expressed without the same degree of adjustment when outcome value changes. The contrast is functional rather than a claim that every decision follows a simple one-way sequence.

The dorsal striatum includes the dorsolateral striatum, or DLS, and the dorsomedial striatum, or DMS. The DLS is more closely associated with model-free learning and habits. The DMS is more closely associated with model-based learning and goal-directed processes.

Goal-directed choice depends on more than one brain contribution. The prefrontal cortex is implicated in executive functions such as planning and decision making. Within the prefrontal cortex, the orbitofrontal cortex, or OFC, is related both to the subjective reward value of biologically significant stimuli and to the reward expected as a consequence of actions. This makes the OFC a candidate contributor to the reward portion of an animal's environment model.

The hippocampus adds a different kind of support. It is critical for memory and spatial navigation. In rats, hippocampal function is important for navigating a maze in a goal-directed manner. This connects the hippocampus with the possibility that animals use models, or cognitive maps, when selecting actions. The hippocampus may also contribute to the human ability to imagine new experiences.

Temporary inactivation experiments provide a way to test how changing activity in a region affects behavioral control. When the DLS is inactivated, habit learning is impaired and the animal relies more on goal-directed processes. When the DMS is inactivated, goal-directed processes are impaired and the animal relies more on habit learning.

A common way to investigate these roles is to temporarily inactivate a brain area in a rat and then observe behavior in an outcome-devaluation experiment. The logic is comparative. If inactivating a region weakens habit learning while behavior becomes more goal directed, that region is implicated more strongly in habitual control. If inactivating another region weakens goal-directed processes while behavior becomes more habitual, that region is implicated more strongly in goal-directed control.