Concepts / Actor-Critic Algorithms

Actor-Critic Algorithms

The actor and critic are separate components with connected responsibilities.

  • Programming

Why Prediction Is Not Control

Actor-critic algorithms divide reinforcement learning into two connected responsibilities. The critic learns to predict the expected return, while the actor learns how actions should be selected. This distinction matters because a method can support prediction without providing enough information for control. In particular, non-contingent eligibility traces can be adequate for the critic's prediction task while failing to connect an individual action with a later change in the reward signal.

The Missing Action-to-Outcome Link

Imagine the learning process in two moments. First, an agent performs an action. Later, the common reinforcement signal changes. For control learning, the method must connect those two events so that the earlier action can be evaluated in light of its later consequence. Non-contingent eligibility traces do not provide this action-to-consequent-reward-change connection.

Agent actiona specific actionReward changelater consequenceAction-to-outcomelinknot provided by anon-contingent tracehappens latermust be related back
What connection is missing when an eligibility trace does not link a specific action to the later change in the reward signal?
Worked example

Checking whether control information exists

An agent performs an action, and the common reinforcement signal changes afterward. What must the learning process establish before that action can support control learning?

  1. 1
    Identify the earlier event

    The earlier event is the particular action performed by the agent.

  2. 2
    Identify the later event

    The later event is the change in the common reinforcement signal.

  3. 3
    Check the connection

    The learning process must relate the action to the later reward change. Non-contingent eligibility traces do not provide this connection.

  4. 4
    Determine the consequence

    Without that connection, the signal can support prediction for the critic but is not sufficient support for the actor's control learning.

Result

Control requires an action-to-consequent-reward-change connection; non-contingent eligibility traces do not supply it.

Actor and Critic Responsibilities

The actor and critic are separate components with connected responsibilities. The actor follows a policy and makes action choices. It learns policies, so its role is control. The critic studies the policy currently being followed by the actor. It learns to predict the expected return and evaluates that current policy, so its role is prediction and evaluation.

Criticpredicts expected returnActorlearns action policyCurrent policyfollowed by the actorAction choiceselected by the actorselectsfollowsevaluatessends TD error
What does the critic evaluate, what does the actor control, and how do their responsibilities differ within the same learning loop?
ComponentMain responsibilityWhat it learns or provides
CriticPrediction and evaluationPredicts the expected return and evaluates the actor's current policy
ActorControlLearns policies and selects actions

The two connected responsibilities in an actor-critic algorithm.

Following One Learning Cycle

A useful way to understand the algorithm is to trace one action choice. First, the actor follows its current policy and chooses an action. The critic examines the policy currently being followed and learns a state-value function for that policy using a temporal-difference method. The critic then sends a TD error, written as δ in the source material, to the actor. Finally, the actor uses that critique while continually updating its policy.

Actor policycurrent policyActionchosen by actorResultlater outcomeCriticevaluates current policyTD errorbetter or worse thanexpectedPolicy updateactor continues learningselectsleads tois evaluated byproducesguidescontinually changes
How does the actor choose an action, how does the environment produce a result, and how does the critic's evaluation flow back to update the actor's policy?
  1. 1
    The actor follows its current policy and makes an action choice.
  2. 2
    The critic evaluates the policy currently being followed by the actor.
  3. 3
    The critic uses a temporal-difference method to learn about that policy.
  4. 4
    The critic sends a TD error to the actor.
  5. 5
    The actor uses the TD error while continually updating its policy.

Reading the TD Error

The critic communicates its judgment through a TD error. At the beginner level, the most important feature is its sign. A positive TD error means that the action led to a state with a better-than-expected value. A negative TD error means that the action led to a state with a worse-than-expected value. The actor receives this feedback and uses it in its continuing policy updates.

Expected valuecritic's expectationBetter-than-expectedpositive TD errorWorse-than-expectednegative TD errorresult exceeds expectationresult falls below expectation
How does the TD error's sign communicate whether the resulting value was better or worse than expected?
Worked example

Interpreting two TD-error signs

The actor chooses an action. In one case, the resulting state has a better value than the critic expected. In another case, the resulting state has a worse value than expected. Interpret the two TD errors.

  1. 1
    Better result

    When the resulting state has a better-than-expected value, the TD error is positive.

  2. 2
    Worse result

    When the resulting state has a worse-than-expected value, the TD error is negative.

  3. 3
    Actor feedback

    The sign communicates the critic's judgment to the actor, which uses the feedback while continually updating its policy.

Result

Positive means better than expected; negative means worse than expected.

What do you think happens?

The critic observes that an action led to a state with a better-than-expected value. What sign should the TD error have?

  • Positive

  • Negative

Reveal answer
Answer:

Positive

The source defines a positive TD error as indicating a better-than-expected result.

Team Learning and Differentiated Behavior

The defined team learning setting contains a population of agents that receives a common reinforcement signal. The goal is for the agents to learn differentiated patterns of activity. This is more demanding than making every agent predict the same quantity. The learning process must provide information about how each agent's activity relates to changes in the shared reward signal.

Shared prediction does not necessarily produce differentiated behavior. If agents receive or learn the same shared prediction but the learning process does not correlate each agent's actions with later changes in the common reward, the signal does not identify how individual activity should contribute to distinct patterns. This is why non-contingent eligibility traces are inadequate for the team setting: they do not supply the required action-to-reward-change connection for each agent.

Agent populationmultiple agentsCommon reinforcementsignalshared by agentsShared predictionprediction aloneIndividual actionlinknot identified bynon-contingent tracesDifferentiatedactivityrequires individual controlinformationreceivecan supportneed links for individual actionssupportsdoes not guarantee
How can multiple agents receive a shared prediction while still failing to develop distinct action policies?

Mistakes in Role Assignment

  • Common mistake

    Treating prediction and control as the same job.

    What it looks like

    Assuming that because the system can predict the expected return, it can automatically learn which actions should be selected.

    Why it fails

    The source distinguishes the critic's prediction role from the actor's control role. Prediction alone does not provide the action-selection responsibility.

    Do this instead

    Assign prediction and evaluation to the critic, and policy learning and action selection to the actor.

  • Common mistake

    Assuming that a common reinforcement signal identifies each agent's contribution.

    What it looks like

    Believing that all agents will develop differentiated activity merely because they receive the same shared signal.

    Why it fails

    The team setting requires a connection between each agent's actions and later changes in the shared reward signal.

    Do this instead

    Check whether the learning method relates individual actions to their consequent reward changes.

  • Common mistake

    Reversing the meaning of the TD-error sign.

    What it looks like

    Interpreting a negative TD error as evidence of a better-than-expected result.

    Why it fails

    The source states that a positive TD error indicates a better-than-expected result, while a negative TD error indicates a worse-than-expected result.

    Do this instead

    Remember: positive means better than expected; negative means worse than expected.

  • Common mistake

    Describing the critic as the component that directly controls the policy.

    What it looks like

    Saying that the critic chooses the actions and the actor only observes them.

    Why it fails

    The actor follows a policy and makes action choices. The critic evaluates the policy currently followed by the actor.

    Do this instead

    Describe the critic as evaluator and the actor as policy learner and controller.

Check Your Understanding

MEDIUM

A population of agents receives one common reinforcement signal. Explain why a shared prediction alone may fail to produce differentiated patterns of activity. Then trace the actor-critic cycle: identify the actor's action, the critic's evaluation, the meaning of the TD error, and the actor's next learning responsibility.

Hints
  • Start with the missing connection between an individual action and a later reward change.
  • State separately what the critic predicts and what the actor controls.
  • Use the sign of the TD error to describe whether the result was better or worse than expected.
  • End by explaining that the actor uses the critic's feedback to continually update its policy.
  1. 1
    Name the component that selects an action according to a policy.
  2. 2
    Name the component that evaluates the policy currently being followed.
  3. 3
    Explain why non-contingent eligibility traces can support prediction but not sufficient control learning.
  4. 4
    Interpret a positive TD error.
  5. 5
    Interpret a negative TD error.
  6. 6
    Explain how the actor uses the TD error in its continuing policy updates.

Essential Takeaways

  1. 1
    Non-contingent eligibility traces do not correlate an action with a consequent change in the reward signal.
  2. 2
    The critic predicts the expected return and evaluates the policy currently followed by the actor.
  3. 3
    The actor controls action selection by learning and continually updating policies.
  4. 4
    A positive TD error indicates a better-than-expected result, while a negative TD error indicates a worse-than-expected result.
  5. 5
    In team learning, shared prediction alone does not guarantee differentiated agent behavior; individual action-to-reward-change information is required for control.

At a glance

  • Actor-critic algorithms separate prediction and control into a critic and an actor.
  • The critic evaluates the actor's current policy using a temporal-difference method.
  • The TD error tells the actor whether an action produced a better-than-expected or worse-than-expected result.
  • Non-contingent eligibility traces fail to connect individual actions with later reward changes.
  • Without that connection, a common reinforcement signal may support shared prediction without producing differentiated agent behavior.

Related Topics

Related concepts

More concepts starting with A