Concepts / Policies in Markov Decision Processes

Policies in Markov Decision Processes

In a Markov decision process, a value function describes how valuable a situation or decision is under a policy.

  • Programming

One State, Two Evaluations

In a Markov decision process, a value function describes how valuable a situation or decision is under a policy. The key question is whether we are evaluating only the current state or evaluating one particular action chosen in that state. That single difference changes what the value function is describing.

State sstarting stateAction aselected firstPolicy πfollowed afterwardExpected returnstart atthen followevaluate
What is held fixed when evaluating qπ(s, a), and what happens after the selected action?

State Value Under a Policy

A state-value function evaluates a state under a policy. In the expression for state value, the focus is the state s and the policy π. The function asks how valuable that state is when considered under that policy. It does not single out one particular initial action in the description.

State sevaluate herePolicy πunder this policyFuture actionsselected by πState valuevalue of s under πconsiderguidescontributes to
Starting from a state, how do the policy and later outcomes determine the state’s value?
Worked example

Reading a state-value description

A learner says, “The value of state s under policy π evaluates one particular action taken in s.” Is this an accurate description of state value?

  1. 1
    Identify the object being evaluated

    The object is state s, not a named action.

  2. 2
    Identify the policy

    The state is evaluated under policy π.

  3. 3
    Check for an initial action

    No specific initial action is included in the state-value description.

Result

The description is not accurate. State value evaluates state s under policy π; evaluating a specific initial action requires the action-value function.

Action Value After the First Move

Definition

The action-value function qπ(s, a) evaluates the expected return from starting at state s, taking action a, and then following policy π. Its description fixes the starting state, fixes the action taken first, and specifies that the policy is followed afterward.

The initial action and the later policy have different roles. Action a is selected explicitly at the beginning of the description. After that initial action, policy π governs what is followed afterward. Therefore, qπ(s, a) is more specific than a state-value description: it evaluates one particular action in one particular state while keeping the later policy fixed.

State sfixed startLater stateExpected returnAction afixed first actionPolicy actionselected by πtakethen reachπ selectscontinue toward
How does control move from a fixed initial action to actions selected by the policy in later states?
Worked example

Reading qπ(s, a)

Interpret the statement: qπ(s, a) evaluates the expected return from starting at s, taking a, and thereafter following π.

  1. 1
    Fix the starting state

    The evaluation begins at state s.

  2. 2
    Fix the first action

    The action taken first is specifically a.

  3. 3
    Apply the policy afterward

    After the selected first action, the policy π is followed.

  4. 4
    Identify the evaluated result

    The function evaluates the expected return for that complete description.

Result

qπ(s, a) evaluates one state-action starting situation under a specified policy for what follows.

State Versus Action Evaluation

QuestionState-value evaluationAction-value evaluation
What is evaluated?A stateA particular action taken in a state
Starting pointState sState s
Is an initial action named?No specific initial action is includedAction a is named and taken first
What policy is involved?State s is evaluated under policy πAfter action a, policy π is followed
Main distinctionFocuses on the state under the policyFocuses on one action in the state, then the policy afterward

The essential difference is whether a specific initial action is included.

State valuestate s under πState sevaluatedPolicy πunderlying policyAction valueqπ(s, a)State sstarting stateAction ataken firstPolicy πfollowed afterwardevaluatesunderstarts attakesthen follows
How does evaluating a state under a policy differ from evaluating one specific action chosen in that state?
Best practice

When answering a question about value functions, state the object being evaluated before discussing the policy. For state value, say that state s is evaluated under policy π. For action value, add the missing specificity: start at s, take a, and thereafter follow π.

Mistakes in Reading qπ(s, a)

  • Common mistake

    Treating qπ(s, a) as if it evaluated only state s.

    What it looks like

    Saying that qπ(s, a) is simply the value of state s under π.

    Why it fails

    The expression also fixes action a as the action taken first.

    Do this instead

    Describe qπ(s, a) as the expected return from starting at s, taking a, and then following π.

  • Common mistake

    Forgetting the policy after the first action.

    What it looks like

    Describing qπ(s, a) only as taking action a in state s.

    Why it fails

    The action-value description includes what happens afterward: policy π is followed.

    Do this instead

    Mention both the initial action and the policy followed afterward.

  • Common mistake

    Assuming the initial action is selected by the policy in the same way as every later action.

    What it looks like

    Saying that qπ(s, a) starts by letting π choose the action.

    Why it fails

    The action a is explicitly fixed in the action-value description.

    Do this instead

    Keep action a fixed as the first action, then identify π as the policy followed afterward.

  • Common mistake

    Leaving out the starting state.

    What it looks like

    Describing qπ(s, a) as evaluating action a under π without saying where the action is taken.

    Why it fails

    The description begins from a particular state s.

    Do this instead

    Name all three parts: starting state s, initial action a, and later policy π.

Check Your Interpretation

EASY

Explain the difference between these two descriptions: evaluating state s under policy π, and evaluating qπ(s, a). Your answer should identify what is evaluated, what is fixed at the start, and what happens after the first action.

Hints
  • State value does not name one specific initial action.
  • Action value names action a as the first action.
  • The policy π is followed after that initial action.
Worked example

A complete answer

Give a complete explanation of qπ(s, a).

  1. 1
    Name the start

    Begin from state s.

  2. 2
    Name the selected action

    Take the specific action a first.

  3. 3
    Name what follows

    After that action, follow policy π.

  4. 4
    Name the evaluation

    The function evaluates the expected return for this starting state, first action, and later policy.

Result

qπ(s, a) evaluates the expected return from starting at s, taking a, and thereafter following π.

Key Takeaways

  1. 1
    A state-value function evaluates a state under a policy.
  2. 2
    The action-value function qπ(s, a) evaluates the expected return from starting at s, taking a, and then following π.
  3. 3
    State-value evaluation focuses on a state; action-value evaluation focuses on a specific initial action in that state.
  4. 4
    A complete action-value description fixes the starting state, the first action, and the policy followed afterward.
  5. 5
    The essential distinction is whether a specific initial action is included.

At a glance

  • State value evaluates state s under policy π.
  • qπ(s, a) evaluates the expected return from starting at s, taking action a, and thereafter following π.
  • Action value is more specific because it names the initial action.
  • To interpret qπ(s, a), track the starting state, the selected first action, and the later policy.

Related Topics

Related concepts

More concepts starting with P