Understanding Models and Planning in Reinforcement Learning

In reinforcement learning, a model is anything the agent can use to predict how the environment will react to an action.

Programming

When the Environment Is Not the Only Source of Answers

An agent often has to consider what might happen if it takes an action. Without a model, it must treat the environment's response as unknown until the action is actually carried out. With a model, the agent can predict the response and consider actions without waiting for the environment to respond every time.

Key point

In reinforcement learning, a model is anything the agent can use to predict how the environment will react to an action.

The Model's Prediction

A model prediction begins with two pieces of information: the current state and an action the agent is considering. The prediction concerns two parts of the environment's response: the reward and the next state. In other words, the model helps answer this question: if the agent is in this state and takes this action, what reward and next state should it expect?

What does a reinforcement learning model predict from a current state and an action?

Worked example

Tracing One Model Query

An agent is in the generated state named Junction A and considers the generated action Move east. What information does the model need to predict?

1
Start with the state
The model receives Junction A as the state in which the agent is currently situated.
2
Add the action
The model also receives Move east, the action the agent is considering.
3
Predict the response
The model predicts the reward associated with this response and the next state reached after the action.

Result

The model query is based on a state and an action, and its predicted response consists of a reward and a next state.

Two Ways to Represent Outcomes

The important difference between distribution models and sample models is how much of the possible response the model returns. A distribution model describes the complete set of possibilities and their probabilities. A sample model returns one possibility selected according to those probabilities.

How does each model represent the possible response to a state and action?

Feature	Distribution model	Sample model
Returned information	The complete set of possibilities	One possibility at a time
Probability information	Includes the probabilities of the possibilities	Uses the probabilities to select one possibility
Expressiveness	More expressive because it exposes the full set of possibilities and probabilities	Less expressive for a single query because it exposes one possibility
Source-grounded application	Used in dynamic-programming estimates of an MDP's dynamics	Used in the blackjack example described in the source material

The distinction concerns how much of the environment response is represented or returned.

Why Representation Changes Planning

Planning means considering possible actions by using predicted environment responses. Instead of treating every response as unknown, the agent can query its model, examine the predicted reward and next state, and use that information while deciding what to do. The model therefore gives the agent a way to consider actions without requiring an immediate response from the environment for every consideration.

How can an agent use predicted responses to simulate an action and choose a plan?

Worked example

Planning with Different Model Outputs

Suppose an agent is considering an action from a generated state called Room 1. How would planning differ depending on the model type?

1
Use a distribution model
The agent receives the complete set of possible responses to the state and action, together with their probabilities. It can therefore consider the full range of predicted outcomes.
2
Use a sample model
The agent receives one probability-based possibility for the state and action. It can use that possible response as one simulated result.
3
Compare the meaning
The distribution model exposes more information about the possible response, while the sample model provides one outcome at a time.

Result

Both model types let the agent predict an environment response, but they provide different amounts of information about the possible outcomes.

Mistakes About Models

Common mistake
Treating a model as a prediction of only the next state
What it looks like
The model tells the agent where it will be next, but says nothing about reward.
Why it fails
The model prediction concerns both the next state and the reward.
Do this instead
When describing a model response, name both predicted parts: reward and next state.
Common mistake
Assuming that a model is the environment's actual response
What it looks like
The agent must perform the action before it can use the model.
Why it fails
A model lets the agent predict how the environment will react without waiting for the environment to respond every time it considers an action.
Do this instead
Treat the model as a way to predict an environment response for planning.
Common mistake
Confusing one sampled outcome with the complete distribution
What it looks like
A sample model returns one outcome, so that outcome must be the only possible outcome.
Why it fails
A sample model exposes one probability-based possibility at a time; the distinction does not say that no other possibilities exist.
Do this instead
Use distribution model for the complete set and probabilities, and sample model for one selected possibility.
Common mistake
Calling distribution and sample models different kinds of predicted input
What it looks like
A distribution model starts from reward, while a sample model starts from the state.
Why it fails
Both forms begin with a state and an action. They differ in how much of the possible response they return.
Do this instead
Keep the input fixed in your explanation and compare the returned outcome information.

Practice the Distinction

EASY

An agent considers an action from a current state. One model describes every possible reward and next-state outcome along with the probability of each. Another model returns one probability-based reward and next-state outcome. Identify which model is the distribution model and which is the sample model, then explain why.

Hints

Look at whether the model returns the complete set of possibilities or one possibility.
Probability information is explicitly represented in the distribution model.
The sample model returns one possibility selected according to probabilities.

Your attempt (not saved)

1
A correct answer identifies the first model as a distribution model because it exposes every possible outcome and its probability. The second is a sample model because it exposes one probability-based possibility at a time.

Key Takeaways

A reinforcement learning model is anything an agent can use to predict how the environment will react to an action.
The prediction starts with a state and an action and concerns a reward and a next state.
A distribution model represents the complete set of possible responses and their probabilities.
A sample model returns one probability-based possibility at a time.
Models support planning because an agent can consider predicted responses without waiting for the environment to respond every time.

At a glance

A model predicts the environment's response to an action from a given state.
The predicted response concerns both reward and next state.
Distribution models expose all possible outcomes and their probabilities.
Sample models expose one probability-based outcome at a time.
Using a model allows an agent to consider actions and plan without treating every response as unknown.

Related concepts

More concepts starting with U