Sample Model

Planning can improve value estimates without waiting for another real interaction with the environment. It asks the model to generate simulated experience instead.

Programming

Planning Without a New Interaction

In reinforcement learning, an agent can improve its value estimates in two different ways. Learning uses experience collected from the real environment. Planning uses experience generated by a model of the environment. A sample model is the part of this process that produces one possible response to a chosen state and action: a reward and a next state.

Key point

Planning can improve value estimates without waiting for another real interaction with the environment. It asks the model to generate simulated experience instead.

What is the difference between experience generated by a model for planning and experience collected from the real environment for learning?

The One-Step Planning Path

Random-sample one-step tabular Q-planning combines random selection with a one-step tabular Q-learning update. First, the method selects a state S and an available action A at random. It sends that pair to the sample model. The model returns a sample reward R and a sample next state S'. The four quantities S, A, R, and S' then go into one-step tabular Q-learning, which changes the Q-function entry Q(S, A).

How does the process move from a randomly selected state-action pair through the sample model to a Q-function update?

What the Sample Model Returns

Definition

A model of the environment predicts what the environment may do after receiving a state and an action. The prediction concerns both parts of the response: the next state and the next reward.

A sample model returns one possible outcome for the supplied state and action. In the planning update, that outcome is represented by a sample reward R and a sample next state S'. The outcome is selected according to the environment's probabilities, but the model reports only the one result used for this iteration.

After receiving a state and action, how does the sample model produce one possible reward and next state?

Distribution and Sample Reports

Model type	Response to a state and action	Information exposed
Distribution model	All possible next states and rewards with their probabilities	Complete probability description
Sample model	One next-state and reward outcome	One outcome drawn according to the probabilities

Both model types predict the environment's response; they differ in how much of that response they expose at one time.

A distribution model lays out the complete set of possible outcomes and their probabilities. A sample model gives one outcome selected according to those probabilities. Both support planning because both let an agent anticipate what may happen after a state and action.

What does each model return for a given state and action, and how does a probability distribution differ from one sampled outcome?

The Dozen-Dice Difference

Worked example

Rolling a Dozen Dice

Consider a process that rolls a dozen dice and records the sum. What does each model type report?

1
Distribution model
It returns all possible sums together with the probability of each sum.
2
Sample model
It simulates the rolls and returns one sum selected according to the underlying probability distribution.
3
Compare the reports
The underlying dice process is unchanged. Only the model's report changes: one exposes the entire distribution, while the other exposes one sampled result.

Result

The distribution model reports every possible sum and its probability; the sample model reports one sum.

Given the same dozen-dice process, what output does the distribution model return compared with the sample model?

Key point

The dozen-dice example is not a comparison of two different probability distributions. It is a comparison of two ways to report the same underlying process.

Coverage and Convergence

Random selection supports convergence only when it provides sufficient coverage over time. To converge to the optimal policy for the model, every state-action pair must be selected an infinite number of times, and the learning-rate parameter α must decrease appropriately over time. Repeated model-generated updates then give the Q-function opportunities to incorporate information about every state-action pair.

Common mistake
Assuming that one model call describes the whole environment.
What it looks like
Treating one returned reward and next state as if they were every possible outcome.
Why it fails
A sample model returns one outcome, whereas a distribution model exposes the complete probability description.
Do this instead
Interpret the sample as one model-generated transition used for one planning update.
Common mistake
Treating the sample model as the component that performs the Q-function update.
What it looks like
Saying that the model itself changes Q(S, A).
Why it fails
The model generates the transition sample; one-step tabular Q-learning processes that sample and changes the Q-function.
Do this instead
Keep the roles separate: request R and S' from the model, then apply the one-step Q-learning backup.
Common mistake
Assuming random selection alone guarantees convergence.
What it looks like
Selecting some state-action pairs repeatedly while never selecting others.
Why it fails
Convergence requires every state-action pair to be selected an infinite number of times.
Do this instead
Check both required conditions: sufficient repeated coverage and an appropriately decreasing α.

Trace and Check

MEDIUM

Suppose random-sample one-step tabular Q-planning selects a state S and an available action A. The sample model returns a reward R and a next state S'. Identify which component creates the simulated transition, which component changes Q(S, A), and what additional conditions are required for convergence.

Hints

Separate the model's response from the Q-learning backup.
The convergence conditions concern how often state-action pairs are selected and how α changes over time.

Your attempt (not saved)

What do you think happens?

A model receives a state and an action. Which result should you expect from a sample model?

Every possible next state and reward with its probability
One possible reward and one possible next state
Only the current state
Only the chosen action

Reveal answer

Answer:

One possible reward and one possible next state

A sample model supplies one outcome drawn according to the environment's probabilities. A distribution model supplies the complete set of possible outcomes and their probabilities.

Essential Takeaways

1
Planning uses model-generated experience, while learning uses experience from the real environment.
2
Random-sample one-step tabular Q-planning selects S and A, asks the sample model for R and S', and applies a one-step Q-learning update to Q(S, A).
3
A sample model returns one possible outcome; a distribution model returns all possible outcomes and their probabilities.
4
Distribution models are more capable because they can generate samples, while sample models may be easier to construct.
5
Convergence to the model's optimal policy requires every state-action pair to be selected infinitely often and α to decrease appropriately.

At a glance

A sample model creates simulated experience for planning by returning one reward and one next state after receiving a state and action.
The model-generated transition is then processed by one-step tabular Q-learning to update Q(S, A).
A distribution model exposes the full probability description, while a sample model exposes one sampled outcome.
The dozen-dice example shows the difference between reporting all sums with probabilities and reporting one sum.
Convergence requires infinite selection of every state-action pair and an appropriately decreasing α.

Related concepts

More concepts starting with S