Stochastic Environments

Incorrect models give planning an inaccurate description of the environment.

Programming

When Planning Trusts the Wrong Description

Planning does not examine the environment directly at every step. Instead, it relies on a model that describes how the environment behaves. If that model is inaccurate, planning can still produce a clear policy, but the policy may be poor because it was computed from the wrong description of what actions will do.

Definition

An incorrect model gives planning an inaccurate description of the environment. The resulting policy may be suboptimal because it is based on incorrectly predicted behavior.

How does a policy chosen using an inaccurate model differ from the policy that is best in the real environment?

Key point

A policy can be internally consistent and still be suboptimal. The problem may not be the planning process itself; the problem may be that planning received an inaccurate model.

Tracing a Model Error

Worked example

An Action with a Missing Real-World Outcome

Suppose a model predicts that taking an action from a state leads to a favorable opportunity. Planning uses that prediction to choose a policy. In the actual environment, the favorable opportunity is not available.

1
Model prediction
The model represents the action as producing a favorable possibility.
2
Planning
Planning treats the represented advantage as an opportunity worth pursuing and includes the action in its policy.
3
Real interaction
The agent attempts to use the opportunity, but the environment does not provide what the model predicted.
4
Evidence
The failed attempt supplies evidence that the model's description needs correction.

Result

The policy was suboptimal because it was based on a predicted opportunity that did not exist in the environment. The interaction also exposed the modeling error.

What does the model predict, what actually happens in the environment, and where does the mismatch occur?

The important sequence is not simply model error followed by failure. Planning can actively bring the error to light. When a policy pursues an opportunity that exists in the model but not in the environment, the attempted use of that opportunity supplies evidence that the model needs correction.

Three Sources of Inaccuracy

Modeling errors can arise for at least three distinct reasons. The environment may behave stochastically, the model may use an imperfect function approximation, or the environment may change. These causes all produce an inaccurate model, but they describe different ways the model can fail to match reality.

Cause	What makes the model inaccurate	What the mismatch concerns
Stochastic behavior	The environment can produce different outcomes, while the model may represent only one predicted behavior.	The actual outcome can differ from the model's predicted outcome.
Imperfect function approximation	The model's approximation does not represent the environment accurately.	The approximation fails to match the environment's behavior.
Environmental change	The environment changes after the model's description was formed.	The model describes behavior that is no longer the current behavior.

Three causes of modeling errors identified in the source material.

How do stochastic behavior, imperfect function approximation, and environmental change differ as causes of an inaccurate model?

Why Stochastic Behavior Misleads a Model

Stochastic behavior is a source of modeling error because the environment may produce different outcomes. A model that predicts only one outcome can therefore fail to represent what happens on a particular interaction. Planning then evaluates actions using a description that may not capture the range of behavior in the environment.

How can the same action produce different outcomes in the real environment even when the model predicts only one outcome?

Key point

The central issue is not that planning becomes impossible. Planning can still select a policy. The issue is that the selected policy may be suboptimal when the model's predicted behavior does not match the environment's behavior.

Optimism as an Error-Discovery Mechanism

An optimistic model predicts more favorable possibilities than the environment can actually provide. It may represent greater reward or better state transitions than are really possible. Planning treats those imagined advantages as opportunities worth pursuing.

What do you think happens?

An optimistic model predicts a favorable opportunity that is not actually available. What is likely to happen when planning chooses a policy that pursues it?

The opportunity is pursued, and the mismatch can provide evidence that the model is wrong.
The model becomes accurate without any interaction.
Planning stops before the opportunity is tested.

Reveal answer

Answer:

The opportunity is pursued, and the mismatch can provide evidence that the model is wrong.

The policy generated from the optimistic model pursues the favorable possibility. If the environment does not provide it, the attempted use exposes the mismatch and supplies evidence for correcting the model.

How can planning with an optimistic model lead the agent to discover that the model is wrong and then correct it?

This is why optimistic models are especially useful for understanding error discovery. Their policies pursue favorable opportunities that may fail in reality. The failure is informative: it reveals that the model contains an opportunity the environment cannot provide, giving evidence that the model should be corrected.

Mistakes in Diagnosing Model Errors

Common mistake
Assuming that a clear policy must be a good policy.
What it looks like
Planning produces a definite policy from an inaccurate model.
Why it fails
A policy can be clear while still being based on incorrect predicted behavior.
Do this instead
Check whether the model accurately describes the environment before judging the policy.
Common mistake
Treating every modeling error as stochastic behavior.
What it looks like
The model is inaccurate because its function approximation is imperfect or because the environment has changed.
Why it fails
Stochastic behavior, imperfect function approximation, and environmental change are distinct causes of modeling errors.
Do this instead
Ask whether the mismatch comes from variable outcomes, an inaccurate representation, or a changed environment.
Common mistake
Describing optimistic planning as guaranteed success.
What it looks like
The model predicts a favorable opportunity, but the environment does not provide it.
Why it fails
Optimism predicts possibilities that may fail in reality.
Do this instead
Treat a failed optimistic opportunity as evidence that can support model correction.
Common mistake
Stopping the analysis at the policy failure.
What it looks like
The policy pursues an opportunity that does not exist, and the analysis ends there.
Why it fails
The attempted use can reveal the model error and provide evidence for correction.
Do this instead
Trace the full sequence from prediction to action, environmental outcome, evidence, and model revision.

Practice the Diagnosis

MEDIUM

A planner chooses a policy because its model predicts a better state transition. The agent follows the policy, but the environment produces a less favorable result. Identify the model-related explanation, state whether the resulting policy can be suboptimal, and describe how the observation can help the agent.

Hints

Start by comparing the model's predicted behavior with the environment's actual behavior.
Use the three causes of modeling errors to identify possible sources of the mismatch.
Explain why the observation is evidence rather than merely a failure.

Your attempt (not saved)

Worked example

Practice Solution

A planner chooses a policy because its model predicts a better state transition. The agent follows the policy, but the environment produces a less favorable result.

1
Identify the mismatch
The model predicts more favorable behavior than the environment provides.
2
Assess the policy
The policy may be suboptimal because planning selected it using the inaccurate prediction.
3
Classify possible causes
The mismatch may arise from stochastic behavior, imperfect function approximation, or environmental change; the observation alone does not establish which cause is responsible.
4
Use the observation
The less favorable result supplies evidence that the model's description needs correction.

Result

The policy can be suboptimal because it relies on an inaccurate model, and the observed mismatch can guide correction of that model.

Key Takeaways

1
Planning relies on a model rather than directly examining the environment at every step.
2
An inaccurate model can cause planning to produce a suboptimal policy because the policy is based on incorrect predicted behavior.
3
Modeling errors can arise from stochastic behavior, imperfect function approximation, or environmental change.
4
An optimistic model represents favorable possibilities that may not exist in reality.
5
When planning pursues a false opportunity, the failed attempt can reveal the model error and provide evidence for correction.

At a glance

A model is the description that planning uses to predict environmental behavior.
If that description is inaccurate, planning can select a policy that is clear but suboptimal.
Stochastic behavior, imperfect function approximation, and environmental change are three causes of modeling errors.
Optimistic models can make favorable but false opportunities attractive to planning.
Pursuing a false opportunity can expose the mismatch and supply evidence for correcting the model.

Related concepts

More concepts starting with S