Dyna-Q Agent
A Dyna-Q agent can fail when the environment changes after it has learned a model.
When the Maze Changes
A Dyna-Q agent can learn a successful route and still fail to use a better route later. The reason is that the agent plans from its learned model. If the environment changes but the model does not include that change, planning continues from incomplete information.
In the Shortcut Maze, the agent initially learns a route around the left side of a barrier. After 3000 steps, a shorter path opens on the right. The environment now offers a better route, but the regular Dyna-Q agent continues with the established left-side route.
The Model Falls Behind
A Dyna-Q agent uses both learning and planning. Information from interaction with the environment contributes to a learned model, and planning uses that model to make decisions. In the Shortcut Maze, the important failure is not simply one bad action. The model itself does not contain the newly opened shortcut. As a result, planning has no representation of that connection to reason about.
The sequence identifies the modeling problem: the environment changes, but the model has not incorporated the shortcut. Planning can reason from what the model contains, not from an unrepresented route. Therefore, additional planning from the same stale model does not by itself reveal the missing connection.
Planning Repeats the Old Route
What do you think happens?
After the right-side shortcut opens, what is most likely to happen if planning repeatedly uses a model that still contains only the old route?
Reveal answer
Planning continues to use the information supporting the old route
The model does not contain the shortcut, so more planning from that same model does not reveal it. The agent continues with the established left-side route.
Planning is powerful only with respect to the information available in the model. If repeated planning uses the same stale model, it can continue to support the established route. More planning does not automatically create knowledge of a shortcut that has never entered the model.
Why Exploration Can Miss the Shortcut
An ε-greedy policy does include exploratory action selection, but exploration does not guarantee that the agent will discover the new shortcut. The issue is not that the agent has no exploration at all. The issue is that discovery requires the particular sequence of exploratory actions that takes the agent through the newly opened route.
An occasional exploratory action may fail to place the agent on the right sequence of actions. Without reaching and learning the new route, the model remains missing the shortcut, and planning continues to use stale information.
Common mistake Assuming that any exploration will discover the shortcut.
What it looks likeThe agent sometimes explores, but it does not take the particular sequence of exploratory actions needed to reach the newly opened route.
Why it failsThe source example identifies a required sequence of exploratory actions, not merely the presence of occasional exploration.
Do this insteadDistinguish between having an exploratory policy and actually discovering the specific new connection.
Common mistake Assuming that more planning must reveal the optimal path.
What it looks likePlanning repeatedly uses a model that does not contain the right-side shortcut.
Why it failsPlanning reasons from the model, and the model is missing the new connection.
Do this insteadAsk first whether the model contains the route that planning is expected to use.
Common mistake Explaining the failure as one bad action only.
What it looks likeThe agent continues with the left-side route after the maze changes.
Why it failsThe deeper issue is that the model itself is missing the shortcut, so repeated planning also remains limited.
Do this insteadTrace the environment, the learned model, and the planning process together.
Planning Versus Optimality
| Situation | What planning can use | Likely consequence |
|---|---|---|
| Model contains the relevant route | Information representing that route | Planning can reason from it |
| Model misses the newly opened shortcut | Stale information about the established route | Planning does not reveal the missing shortcut by itself |
Planning operates on the agent's current model; it is not automatically equivalent to finding the environment's optimal path.
The misconception is that combining learning with planning means a Dyna-Q agent will always find the optimal path. The Shortcut Maze disproves that stronger claim. The agent initially learns the left-side route, the environment later opens a shorter right-side route, and the regular agent does not switch because its model does not contain the shortcut.
Check Your Understanding
Explain the Shortcut Maze failure in three linked statements: first describe what changes in the environment after 3000 steps, then state what is missing from the agent's model, and finally explain why additional planning does not automatically make the agent use the shorter path.
Hints
- Mention the shorter right-side path.
- State that the learned model does not contain the shortcut.
- Connect planning to the information represented in that model.
Tracing the Shortcut Maze
Why does the regular Dyna-Q agent continue using the left-side route after a shorter right-side path opens?
- 1Initial learning
The agent initially learns a path around the left side of a barrier.
- 2Environmental change
After 3000 steps, the maze opens a shorter path on the right.
- 3Model check
The learned model does not contain the newly opened shortcut.
- 4Planning consequence
Planning continues from stale information, so it does not by itself reveal the missing route.
- 5Exploration check
ε-greedy exploration does not guarantee the particular sequence of exploratory actions needed to discover and learn the shortcut.
The agent continues with the established left-side route because the environment changed before the model incorporated the new right-side connection.
Key Takeaways
- 1A regular Dyna-Q agent can fail when the environment changes after it has learned a model.
- 2In the Shortcut Maze, a shorter right-side path opens after 3000 steps, but the agent continues with the established left-side route.
- 3The model does not contain the shortcut, so planning uses stale information.
- 4ε-greedy exploration does not guarantee the sequence of actions needed to discover a new route.
- 5Planning is not the same as finding the optimal path when the model is incomplete or outdated.
At a glance
- A Dyna-Q agent plans from its learned model, so an environmental change can make that model stale.
- In the Shortcut Maze, the right-side shortcut opens after 3000 steps, but the agent keeps using the established left-side route.
- Exploration is not enough by itself: ε-greedy behavior may fail to produce the exact action sequence needed to discover the shortcut.
- Repeated planning cannot reveal a route that the model does not represent.
- Planning supports decisions from current knowledge; it does not guarantee the optimal path in a changed environment.