Planning with Models

The central question is not only whether a model produces an outcome. You must also ask what kind of outcome it produces: one possible result, or every possible result together with how likely each result is.

Programming

Simulated Experience

In reinforcement learning, an agent does not always need to obtain every learning experience directly from the environment. A model can imitate the environment well enough to produce possible experience for inspection or planning. This changes the source of the experience: instead of relying only on a direct interaction, the system can ask the model what might happen.

How does information flow from a policy through a model to create simulated experience instead of using a direct environment interaction?

Key point

One Simulated Transition

A transition is one simulated step from a starting state after an action is supplied. To interpret this request, focus on two pieces of input: the state and the action. The model uses that state-action information to describe what can happen next.

What information enters the model, and how does it produce the next state and reward?

A sample model answers with a single possible way that the step could unfold. A distribution model answers more broadly by generating all possible transitions and attaching their probabilities of occurring. Both model types describe a transition, but they provide different amounts of information about its possible outcomes.

Worked example

Reading a Transition Request

A planning request supplies a state and an action. Determine the scale of the requested simulation and compare the two possible model outputs.

1
Identify the input
The input contains a starting state and an action, so the request concerns one transition rather than a complete episode.
2
Apply a sample model
The sample model returns one possible transition from that state after the action is supplied.
3
Apply a distribution model
The distribution model returns all possible transitions for that state and action together with their probabilities.

Result

A state plus an action calls for a transition-level interpretation. The model type determines whether the result is one possible transition or the full probability-weighted collection of possible transitions.

Sample and Distribution Outputs

Given a state and action, how does a sample model's single predicted outcome differ from a distribution model's possible outcomes and probabilities?

Model type	Input scale	Result
Sample model	State and action for a transition, or state and policy for an episode	One possible transition or episode
Distribution model	State and action for a transition, or state and policy for an episode	All possible transitions or episodes together with their probabilities

The model type determines whether the output is narrow and path-specific or broad and probability-weighted.

What do you think happens?

A model receives a starting state and an action. If it is a distribution model, will it return only one possible next result?

Yes, always one result
No, all possible transitions with their probabilities
It returns an entire episode only

Reveal answer

Answer:

No, all possible transitions with their probabilities

A starting state plus an action describes a transition. A distribution model gives the complete set of possible transitions and the probabilities of those transitions.

Generating an Episode

A simulation can continue beyond one transition. When the input consists of a starting state and a policy, the model can extend the simulation into an episode. The policy supplies the actions used while the model supplies possible experience resulting from those actions.

How do a starting state and a policy produce a sequence of states, actions, and rewards?

A sample model can produce an entire episode as one possible sequence of experience. In contrast, a distribution model can generate all possible episodes and the probabilities of those episodes. Therefore, a sample-model episode is narrow and path-specific, while a distribution-model result describes the broader set of possible episodes in probabilistic terms.

Worked example

Interpreting a Policy-Based Simulation

A simulation begins with a state and uses a policy. Determine what an episode-level sample model and an episode-level distribution model produce.

1
Identify the input
A starting state plus a policy describes an episode-level simulation rather than a single transition.
2
Use a sample model
The sample model follows one possible sequence of experience generated from the starting state while using the policy.
3
Use a distribution model
The distribution model describes all possible episodes and gives the probabilities of those episodes.

Result

The same starting state and policy can produce either one possible episode or the full probability-weighted collection of possible episodes, depending on the model type.

Choosing the Right Interpretation

When interpreting a model's output, use two questions in order. First, identify the requested input. A state and action describe a step, so interpret the result as a transition. A state and policy describe an extended simulation, so interpret the result as an episode. Second, identify the model type. A sample model gives one possible result; a distribution model gives every possible result together with its probabilities.

What kind of result does each model return: one sampled transition or a probability distribution over possible transitions?

1
Read the input: state and action means transition; state and policy means episode.
2
Read the model type: sample means one possible result; distribution means all possible results with probabilities.
3
State the output at the same scale as the input: one transition or one episode for a sample model, and all transitions or all episodes with probabilities for a distribution model.

EASY

A planning system receives a starting state and an action. It uses a sample model. Describe the kind of result the system should interpret as the model's output.

Hints

Use the input to identify whether this is a transition or an episode.
Use the model type to decide whether the output is one result or a probability-weighted collection.

Your attempt (not saved)

Common Interpretation Errors

Common mistake
Treating a sample model as if it lists every possible outcome.
What it looks like
A sample model is asked about a state and action, and its single returned transition is treated as the complete set of transitions.
Why it fails
A sample model returns one possible transition or episode, not all possible results.
Do this instead
Interpret the output as one possible result unless the model is identified as a distribution model.
Common mistake
Treating a distribution model as if it returns only one outcome.
What it looks like
A distribution model's output is described as one predicted next transition without mentioning other possibilities or their probabilities.
Why it fails
A distribution model returns all possible transitions or episodes together with their probabilities.
Do this instead
Look for the complete probability-weighted collection of possible results.
Common mistake
Using the wrong simulation scale.
What it looks like
A starting state and policy are interpreted as a request for only one transition.
Why it fails
A state and policy describe an episode-level simulation, while a state and action describe a transition-level simulation.
Do this instead
Identify the requested input before interpreting the output.
Common mistake
Ignoring the policy when describing an episode.
What it looks like
An episode is described as beginning from a state but without considering the policy used to generate actions.
Why it fails
The episode-level input consists of a starting state and a policy.
Do this instead
Include both the starting state and the policy when explaining how the episode is generated.

Key Takeaways

1
A model lets a reinforcement-learning system simulate possible experience instead of relying only on directly observed experience.
2
A state and action describe one simulated transition.
3
A state and policy describe an episode-level simulation.
4
A sample model returns one possible transition or episode.
5
A distribution model returns all possible transitions or episodes together with their probabilities.

At a glance

Models imitate an environment well enough to produce possible experience for inspection or planning.
The input determines the scale: state plus action means transition, while state plus policy means episode.
Sample models provide one possible result.
Distribution models provide all possible results together with their probabilities.
Correct interpretation requires checking both the input and the model type.

Related concepts

More concepts starting with P