Concepts / Computational Foundations of Reinforcement Learning

Computational Foundations of Reinforcement Learning

Reinforcement learning algorithms are not uniformly modeled on neuroscience.

  • Programming

Two Design Routes

Reinforcement learning algorithms are not uniformly modeled on neuroscience. A feature can enter an algorithm because it helps solve a computational learning problem, or because a hypothesis about neural learning mechanisms suggests that the feature belongs there. These are different design routes, and a single algorithm can contain features that came from both.

Computational goalsolve a learning problemAlgorithm featurecomputationally motivatedNeural hypothesisdescribe a learningmechanismAlgorithm featureneuroscience-influencedmotivatesinfluences
Which parts of a reinforcement learning algorithm were introduced to solve computational problems, and which were proposed to model neural mechanisms?
Key point

The important question is not whether an algorithm has any connection to neuroscience. The more precise question is what originally motivated each feature and whether later evidence agrees with it.

Tracing a Feature's Origin

Suppose a designer chooses a feature because it helps an algorithm learn from a difficult problem. That is a computational motivation. The designer is reasoning about what the algorithm must accomplish, such as learning effectively from experience. In a different route, a designer begins with a hypothesis about how neural learning mechanisms work and uses that hypothesis to influence the algorithm's design.

These routes can coexist. Computational considerations account for the design of most algorithm features, while some features are influenced by hypotheses about neural learning mechanisms. Therefore, saying that an algorithm feature is compatible with neuroscience does not by itself show that neuroscience created or directly derived the feature.

Worked example

Separating Origin from Agreement

An algorithm contains a feature that was introduced because it helped solve a learning problem. Later neuroscience data show brain reward processes that are consistent with the same feature. What can be concluded?

  1. 1
    Identify the original reason

    The feature's design origin was computational: it was chosen because it helped the algorithm solve a learning problem.

  2. 2
    Interpret the later evidence

    The neuroscience data provide support by showing consistency with the feature.

  3. 3
    Avoid reversing the history

    The later evidence does not mean that neuroscience originally produced the feature.

Result

The feature can be computationally motivated and later supported by neuroscience. Design origin and later empirical agreement are separate claims.

Learning problemcomputational analysisAlgorithm designfeatures may have differentoriginsComputational featurehelps solve a problemNeural evidencehypothesis about mechanismsNeural featureinfluenced by a neuralhypothesisguidesinfluencescontainscontains
How can neuroscience influence one component of an algorithm while computational analysis determines other components?

Expected Future Reward

A central decision principle in reinforcement learning is expected future reward: the expected amount of reward an agent can accumulate in the future. The agent considers possible future outcomes, how likely those outcomes are, and the rewards associated with them. Maximizing expected future reward means choosing with the goal of obtaining the greatest average future reward across those possibilities.

Agent choiceselect an actionLikely outcomehigher probabilityFuture rewardreward amountExpected futurerewardprobability-weightedaverageLess likely outcomelower probabilityFuture rewardreward amountprobabilityprobabilityproducesproducescontributescontributes
How do possible future outcomes, their probabilities, and their rewards combine into an agent's expected return?
Worked example

A Probability-Weighted Choice

An agent is comparing two choices. Choice A usually produces a moderate future reward and occasionally produces no reward. Choice B produces a smaller reward in every possible outcome. If the probability-weighted average reward for Choice A is larger, which choice does expected-value optimization prefer?

  1. 1
    List the possibilities

    For Choice A, consider both its moderate-reward outcome and its no-reward outcome. For Choice B, consider its repeated smaller reward.

  2. 2
    Weight by likelihood

    Each possible reward contributes according to how likely that outcome is.

  3. 3
    Compare averages

    Expected-value optimization compares the resulting average future rewards.

Result

If Choice A has the larger expected future reward, expected-value optimization selects Choice A, even though Choice A has more than one possible outcome.

When Average Reward Hides Risk

Averages do not reveal variance. Two choices can have the same expected reward while differing substantially in how spread out or dangerous their possible outcomes are. This means that maximizing expected value alone can overlook risk.

Worked example

Same Average, Different Outcomes

Choice A always produces a moderate reward. Choice B sometimes produces a very large reward and sometimes produces a very poor outcome. Imagine that both choices have the same expected reward. What difference remains?

  1. 1
    Compare expected value

    The two choices are tied when judged only by their average future reward.

  2. 2
    Compare outcome spread

    Choice A has little variation, while Choice B has more widely varying outcomes.

  3. 3
    Introduce risk

    The wider variation in Choice B represents a risk-related distinction that expected value alone does not capture.

Result

Expected-value optimization treats the choices as equal in this illustration, while a risk-sensitive approach can distinguish them by considering risk or variance.

Choice AModerate rewardnarrow variationLarge rewardone possible outcomeSame expected rewardaverage alone cannot decideChoice BPoor outcomeanother possible outcomeproducesmay producemay producecontributescontributescontributes
How can two choices have the same expected reward but differ in the spread or danger of their possible outcomes, and why might an agent prefer one?

Risk-sensitive optimization extends the decision perspective by considering risk or variance in addition to expected reward. It is therefore different from optimization based only on expected value. Excessive risk can be ruinous, which is why risk-sensitive optimization has been developed especially in areas such as finance and optimal control. Risk also matters for reinforcement learning, although the treatment here does not develop that topic in detail.

From Computation to Evidence

Neuroscience data about brain reward processes has increasingly been consistent with many features that were originally motivated computationally. This creates an important interpretive sequence: an algorithmic feature may first be designed for computational reasons, and later evidence may show that the feature agrees with observed neural reward processes.

Algorithmic ideacomputational motivationAlgorithm featureused to address learningNeuroscience databrain reward processesConsistencylater supportmotivatesprecedessupports
What is the sequence from a computationally motivated idea to later neuroscience evidence that is consistent with, but did not directly produce, that idea?
Best practice

When evaluating a claimed connection between reinforcement learning and neuroscience, state two things separately: first, why the algorithmic feature was designed; second, whether neuroscience evidence is consistent with that feature. Do not treat consistency as proof of direct derivation.

  • Common mistake

    Assuming that every reinforcement learning feature was copied from neuroscience.

    What it looks like

    An explanation describes the whole algorithm as a model of the brain.

    Why it fails

    Most algorithm features are attributed to computational considerations, while only some have been influenced by hypotheses about neural learning mechanisms.

    Do this instead

    Check the design reason for each feature instead of assigning one origin to the entire algorithm.

  • Common mistake

    Treating later neuroscience agreement as the original source of an algorithmic idea.

    What it looks like

    A feature was developed for a computational goal, and later data were consistent with it.

    Why it fails

    Later consistency supports the feature but does not change its original design history.

    Do this instead

    Describe the feature as computationally motivated and later supported by neuroscience evidence.

  • Common mistake

    Equating expected reward with a complete decision description.

    What it looks like

    Two choices have the same average reward, so an explanation declares them identical in every relevant respect.

    Why it fails

    Averages do not reveal variance, and different outcome spreads can represent different levels of risk.

    Do this instead

    Separate the expected-reward question from the risk or variance question.

Utility Theory's Boundary

Utility theory connects to reinforcement learning when the subject is interpreted through human desires or economic behavior. This connection can add a perspective on what outcomes mean to people or economic agents rather than treating reward only as an abstract decision quantity. The treatment summarized here does not develop utility theory in detail, so it is best understood as a related topic rather than as part of the central explanation of computational foundations.

Practice and Review

MEDIUM

A proposed reinforcement learning feature was introduced because it helped an algorithm solve a learning problem. Years later, neuroscience data about brain reward processes were found to be consistent with the feature. Separately, two actions were found to have the same expected reward, but one had much more variable outcomes. Explain how you would describe both situations without confusing computational motivation, neuroscience support, expected value, and risk-sensitive optimization.

Hints
  • Name the original reason for the algorithmic feature before discussing the later evidence.
  • Use consistency or support for the neuroscience relationship, not direct derivation.
  • For the two actions, distinguish their average reward from the spread of their possible outcomes.
  1. 1
    Reinforcement learning algorithms combine different kinds of design reasoning. Most features are motivated by computational considerations, while some are influenced by hypotheses about neural learning mechanisms. Later neuroscience evidence can be consistent with a feature without being the reason that feature was originally designed. Expected future reward describes the probability-weighted average reward an agent may accumulate, but expected value alone does not reveal outcome variance or risk. Risk-sensitive optimization considers that additional risk dimension, and utility theory provides a related connection to human desires and economic behavior that lies outside this treatment.

Key Takeaways

  • Algorithm features can be motivated by computational goals, neural-learning hypotheses, or both, but computational considerations account for most features.
  • Neuroscience can later support an algorithmic feature without being the feature's original source.
  • Expected future reward focuses on the average reward an agent can accumulate across possible future outcomes.
  • Expected value does not reveal variance, so risk-sensitive optimization considers information that average-reward optimization can miss.
  • Utility theory relates reinforcement learning to human desires and economic behavior, but that connection is outside this treatment.

Related Topics

Related concepts

More concepts starting with C