Concepts / Reward Baselines in Reinforcement Learning

Reward Baselines in Reinforcement Learning

Stochastic gradient ascent updates preferences only after reward feedback arrives.

  • Programming

The Feedback Moment

A gradient bandit does not update an action merely because the action was selected. The update waits for feedback. Two events must occur: the agent selects an action, and the agent observes the reward produced after that selection. Only then can stochastic gradient ascent change the action preferences.

AgentEnvironmentReward feedbackR_tAction preferencesupdatedselects an actionproducestriggers update
What happens next after an agent selects an action and reward feedback arrives?

The Moving Baseline

The latest reward is not judged in isolation. The algorithm compares it with a running reward average. This average is the baseline for judging the latest outcome, and it includes rewards through the current time t. Because it is computed incrementally, the baseline changes as new rewards arrive. It is therefore a moving point of comparison rather than a fixed target selected once at the beginning.

R_tlatest rewardRunning rewardaveragebaselineReward comparisonabove or belowAbove baselineselected action favoredBelow baselineselected action discouragedcomparecompareR_t is higherR_t is lower
How does the comparison between R_t and the running baseline determine the direction of the preference change?

Selected-Action Probability

An above-baseline reward favors the action that was selected. Its preference changes so that its future probability of selection moves upward. A below-baseline reward discourages the selected action, so its future probability moves downward. This is a qualitative direction of change; the source material does not determine the exact numerical amount of the change.

Above baselineR_tHigher futureprobabilityselected actionBelow baselineR_tLower futureprobabilityselected actionfavorsdiscourages
When the reward is above or below the baseline, does the selected action become more or less likely in the future?

What do you think happens?

Suppose the selected action receives a reward above the running reward average. What happens to its future probability?

  • It increases

  • It decreases

  • The source material does not establish a direction

Reveal answer
Answer:

It increases.

Above-baseline feedback favors the selected action. The exact size of the increase requires the complete update rule, but the direction is established.

Opposite Movement

The selected action is not the only action whose preference changes. The non-selected actions move in the opposite direction. When feedback is above the baseline, the selected action is favored and the non-selected actions are discouraged. When feedback is below the baseline, the selected action is discouraged and the non-selected actions move oppositely. This is why the update changes the relative preference among actions rather than simply recording whether one action produced a win or loss.

Selected actionpreferencebefore feedbackNon-selectedpreferencesbefore feedbackSelected actionpreferencefavored afterabove-baseline feedbackNon-selectedpreferencesopposite movementmoves upwardmoves oppositely
Why do the preferences of non-selected actions move in the opposite direction from the selected action?
In the real world

Generated example: Imagine two actions, A and B. If A is selected and receives feedback above the running reward average, the update favors A while moving B in the opposite direction. If A instead receives below-baseline feedback, A is discouraged and B moves oppositely. The example illustrates direction only; it does not specify numerical preferences or probabilities.

The Step-Size Parameter

α is the positive step-size parameter. It controls the scale of a preference update. The baseline comparison determines the qualitative direction: above-baseline feedback favors the selected action, while below-baseline feedback discourages it. α is part of determining how much the preferences change, not whether the feedback is classified as above or below the baseline.

Smaller αpositive step sizeLarger αpositive step sizeSmaller updatesame directionLarger updatesame directionscalesscales
How does changing α alter the size of each preference update while keeping its direction the same?

What R_t Represents

Definition

R_t denotes the reward signal at time t. A reward signal helps define the reinforcement learning problem together with the environment. In this use, R_t is a theoretical abstraction used to describe the feedback available to an agent.

Reward signals and reinforcement signals are distinct concepts in reinforcement learning theory. The model can use the symbol R_t to represent a reward signal at a particular time, while the broader biological system may involve signals with different physical origins and functions. Neural signals are physiological events that may behave like theoretical signals in function, but the theoretical description and the physiological event are not automatically the same thing.

R_ttheoretical reward signalReinforcementlearning problemdefined with environmentReinforcementsignaldistinct conceptPhysiological neuralsignalsbiological eventsMany neural signalsand systemsbiological complexityhelps definemay relate functionally tomay involvedoes not identify one literal signal
How are R_t and reinforcement signals related to physiological signals in the brain, and why is R_t not a literal single master reward signal in an animal?

Reasoning Practice

Worked example

Classifying Two Feedback Outcomes

An agent selects an action and then receives feedback. In one case, the feedback is above the running reward average. In another case, it is below the running reward average. Predict the direction of the selected action's future probability and the movement of the non-selected actions.

  1. 1
    Above-baseline case

    The selected action is favored, so its future probability moves upward. The non-selected actions move in the opposite direction.

  2. 2
    Below-baseline case

    The selected action is discouraged, so its future probability moves downward. The non-selected actions move oppositely.

  3. 3
    Update timing

    Neither preference change occurs immediately after selection alone. The action must first be selected and its reward must then be observed.

  4. 4
    Update size

    The positive step-size parameter α contributes to the size of the update. The exact numerical amount cannot be determined without the complete update rule and the action-selection probabilities.

Result

Compare R_t with the moving reward average first. Use the comparison for direction, and treat α as a positive parameter involved in the update size.

EASY

An action has just been selected, but no reward has yet been observed. Should stochastic gradient ascent update the action preferences at this moment? Explain what event is still required and what comparison will be made once it arrives.

Hints
  • Recall the two events required before preferences change.
  • The latest reward is compared with a running average rather than a fixed initial target.
MEDIUM

A learner says, R_t must be one physical master reward signal in an animal's brain because the model uses one symbol. Correct the statement using the distinction between theoretical signals and physiological events.

Hints
  • R_t is a theoretical reward signal at time t.
  • Neural signals are physiological events that may behave like theoretical signals in function.
  • One model symbol does not establish one unchanged biological signal.

Common Mistakes

  • Common mistake

    Updating preferences as soon as an action is selected.

    What it looks like

    Generated example: The agent selects action A and immediately changes A's preference before any feedback arrives.

    Why it fails

    Stochastic gradient ascent updates preferences only after both action selection and reward observation.

    Do this instead

    Wait for the reward feedback, then compare it with the running reward average.

  • Common mistake

    Comparing the latest reward with a fixed target chosen at the beginning.

    What it looks like

    Generated example: Treating 5 as a permanent baseline even after many new rewards have arrived.

    Why it fails

    The baseline is the incrementally computed average of rewards through the current time t.

    Do this instead

    Use the moving reward average as the point of comparison.

  • Common mistake

    Assuming that a large reward always favors the selected action.

    What it looks like

    Generated example: Calling a reward of 10 favorable without checking the running average.

    Why it fails

    The relevant question is whether the reward is above or below the rewards seen so far, not whether its isolated numerical value sounds large.

    Do this instead

    Judge R_t relative to the baseline.

  • Common mistake

    Forgetting the non-selected actions.

    What it looks like

    Generated example: Increasing the selected action's preference while leaving every other action unchanged.

    Why it fails

    The non-selected actions move in the opposite direction.

    Do this instead

    Track both the selected action and the non-selected actions.

  • Common mistake

    Treating α as the factor that decides whether feedback is good or bad.

    What it looks like

    Generated example: Claiming that changing α turns below-baseline feedback into above-baseline feedback.

    Why it fails

    The baseline comparison determines the direction; α is the positive step-size parameter involved in the update size.

    Do this instead

    Separate direction from magnitude.

  • Common mistake

    Treating R_t as one literal master reward signal in an animal's brain.

    What it looks like

    Generated example: Identifying R_t with one unchanged physical neural event.

    Why it fails

    R_t is a theoretical abstraction, while neural signals are physiological events and biological systems may involve many signals and systems.

    Do this instead

    Keep the model-level signal distinct from claims about a particular physiological event.

Key Takeaways

  1. 1
    Preferences change only after an action has been selected and its reward has been observed.
  2. 2
    The current reward R_t is judged against an incrementally updated running reward average.
  3. 3
    Above-baseline feedback favors the selected action; below-baseline feedback discourages it.
  4. 4
    Non-selected actions move in the opposite direction, while α is the positive step-size parameter that contributes to update size.
  5. 5
    R_t is a theoretical reward signal, not automatically a single physical master reward signal in an animal's brain.

At a glance

  • Stochastic gradient ascent waits for reward feedback before changing action preferences.
  • The running reward average provides a moving baseline for evaluating R_t.
  • Above-baseline feedback increases the selected action's future probability, while below-baseline feedback decreases it.
  • Non-selected actions move oppositely, and α controls the scale of the preference update as a positive step-size parameter.
  • R_t is a theoretical model signal and should not be identified with one unitary physiological reward signal in an animal's brain.

Related Topics

Related concepts

More concepts starting with R