Concepts / Returns and State Values

Returns and State Values

A state contributes the outcome that follows its observation, not merely the state name itself. The first episode therefore supplies a return of 0 for A and also a return of 0 for its later observation of B.

  • Programming

From Episodes to Predictions

A completed episode tells us what happened after a state was observed. That later outcome is the return associated with the observation. By collecting the returns associated with a state, we can form an estimate of that state's value. In this exercise, the goal is to turn the episode record into predictions for V(A) and V(B).

Observed stateCompleted episodewhat happened afterwardReturn0 or 1State estimateV(state)appears inprovidescontributes to
After a state is observed in an episode, what outcome or return is associated with it, and how does that data move into the estimate?

Tracing the Recorded Episodes

There are eight episodes altogether. Only the first begins in A. That first episode moves from A to B with reward 0 and then terminates from B with reward 0. The other seven episodes begin in B and end immediately.

Afirst episodeBsame episode0return from A0return from B in firstepisodeBseven other episodes1 or 0immediate episode outcomesmoves tolater returnthen terminatesends immediately with
What happens after each state is visited, and how are the state and its later outcome connected?
Key point

A state contributes the outcome that follows its observation, not merely the state name itself. The first episode therefore supplies a return of 0 for A and also a return of 0 for its later observation of B.

Counting Returns by State

Worked example

Estimating V(A) and V(B)

Use the eight recorded episodes to determine the supported estimates for V(A) and V(B).

  1. 1
    Collect A's observations

    Only the first episode begins in A. Its return is 0, so A has one observed return and that return is 0.

  2. 2
    Estimate A's value

    Because the only observed episode beginning in A has return 0, the batch supports V(A) = 0.

  3. 3
    Collect B's observations

    B is observed in the first episode after the transition from A and also at the beginning of the other seven episodes. This gives eight observations of B altogether.

  4. 4
    Count B's returns

    Across those eight observations, B has six returns of 1 and two returns of 0.

  5. 5
    Estimate B's value

    The supported estimate for B is 3/4 because three quarters of B's eight observed returns are successful: six of the eight returns are 1.

Result

V(A) = 0 and V(B) = 3/4 for this batch of episodes.

Aone observation0one returnV(A) = 0Beight observations1, 1, 1, 1, 1, 1, 0,0six 1s and two 0sV(B) = 3/4providessupportsprovidessupports
Which episode outcomes belong to state A or state B, and how are they combined to produce each state estimate?

Why B Equals Three Quarters

The value 3/4 comes directly from the eight recorded observations of B. Six observations are followed by a return of 1, while two are followed by a return of 0. Thus, the successful outcomes account for six of the eight observations, which is 3/4.

B1return 13/4six successful returns outof eightB2return 1B3return 1B4return 1B5return 1B6return 1B7return 0B8return 0countscountscountscountscountscountsdoes not add a successful returndoes not add a successful return
Which four observations of state B are being counted, and which three contribute a successful outcome to make the estimate 3/4?
Key point

Do not count only the seven episodes that begin in B. The first episode also contains an observation of B after the move from A. Including that observation gives eight B observations in total.

Evidence Versus Guesswork

An observed state estimate must be tied to the returns recorded after that state. For A, the batch contains one observation and its return is 0, so V(A) = 0 is supported. For B, the batch contains eight observations with six returns of 1 and two returns of 0, so V(B) = 3/4 is supported. A different value with no corresponding observations in the episode record would be an unsupported guess.

Observed stateepisode record existsRecorded returnsA: 0; B: six 1s and two 0sSupported estimateV(A) = 0 or V(B) = 3/4Unobserved evidenceno corresponding returnsUnsupported guessno episode supporthassupportscannot support
What evidence from the recorded episodes supports an estimate for a state, and what would be an unsupported value with no corresponding observations?
  • Common mistake

    Counting only episodes that begin in B.

    What it looks like

    Treating B as having seven observations because seven episodes begin in B.

    Why it fails

    The first episode begins in A but later observes B before terminating. That observation also contributes a return for B.

    Do this instead

    Count every observation of B, including the B observation inside the first episode. There are eight B observations.

  • Common mistake

    Using the return for A as the return for every state in the episode.

    What it looks like

    Assigning only the first episode's return of 0 to A and then ignoring the later observation of B.

    Why it fails

    Each observed state is connected to what happens after that observation. The first episode supplies a return of 0 for A and a return of 0 for B.

    Do this instead

    Track the state observations separately and associate each with its later outcome.

  • Common mistake

    Choosing a value without evidence in the batch.

    What it looks like

    Assigning a value to a state even though the episode record contains no observation and no return for that state.

    Why it fails

    The exercise asks for estimates supported by completed episodes, not unsupported guesses.

    Do this instead

    Base each estimate on the returns recorded after observations of that state.

Check Your Estimate

MEDIUM

Explain in your own words why the batch supports V(A) = 0 and V(B) = 3/4. In your explanation, state how many observations belong to A, how many belong to B, and how many of B's observations have return 1.

Hints
  • Start with the only episode that begins in A.
  • Remember that the first episode later observes B.
  • For B, distinguish the six returns of 1 from the two returns of 0.

What do you think happens?

Before checking the episode record, how many observations of B should be included in the estimate for V(B)?

  • One

  • Seven

  • Eight

  • Only the observations with return 1

Reveal answer
Answer:

Eight

Seven episodes begin in B, and the first episode also observes B after moving from A. The first episode's B observation must be included.

Key Takeaways

  1. 1
    Episode data records what happened after states were observed.
  2. 2
    Only the first episode begins in A, and its return is 0, so the batch supports V(A) = 0.
  3. 3
    B is observed eight times: once in the first episode and once at the beginning of each of the other seven episodes.
  4. 4
    B has six returns of 1 and two returns of 0, giving the supported estimate V(B) = 3/4.
  5. 5
    An estimate is supported when it is based on recorded returns; a value with no corresponding observations is an unsupported guess.

At a glance

  • A return is the outcome associated with what happens after a state is observed in a completed episode.
  • The only observed episode beginning in A has return 0, so V(A) = 0 for this batch.
  • B has eight observations, with six returns of 1 and two returns of 0.
  • Because six of eight B observations have return 1, the supported estimate is V(B) = 3/4.
  • State estimates must be grounded in recorded episode evidence rather than unsupported guesses.

Related Topics

Related concepts

More concepts starting with R