Planning and Learning in Dyna

Planning is an additional way to improve a policy from existing experience; in this comparison, it does not replace interaction with the maze.

Programming

Why Planning Matters

Dyna-Q addresses a practical question: after an agent has gathered experience, can it improve its policy more quickly by planning from that experience? The agent still interacts with the maze, but it also uses what it has already learned to generate additional learning opportunities. The central idea is that one real interaction can support both immediate learning and later simulated learning.

Key point

Planning is an additional way to improve a policy from existing experience; in this comparison, it does not replace interaction with the maze.

The Dyna Maze

The Dyna Maze contains 47 states. At each state, the agent has four directional actions available. Ordinary transitions provide reward 0. Entering the goal provides reward +1. The learning problem is therefore to discover actions that eventually lead to the goal and to improve the policy that reaches it.

How do ordinary movements differ from the transition that enters the goal?

The maze structure makes delayed information important. Most transitions have no immediate reward, while the useful reward appears when the agent enters the goal. A learning method must therefore use experience to improve actions that may occur several steps before the rewarding transition.

One Transition Two Uses

Worked example

A real experience becomes reusable

Suppose the agent experiences one transition: it takes action a in state s, receives reward r, and arrives in state s'. How can this one event contribute to Dyna-Q twice?

1
Direct learning
The real transition is used immediately by the direct reinforcement-learning part of Dyna-Q, described in the source as one-step tabular Q-learning.
2
Model update
The experience also teaches the table-based model the recorded reward and next-state prediction associated with the experienced state-action pair.
3
Search control
During a later planning phase, search control selects a starting state and action from previously experienced state-action pairs.
4
Simulated experience
The model returns its recorded reward and next-state prediction for that selected pair, creating a simulated experience.
5
Planning update
The simulated experience is used by one-step tabular Q-planning, so the original real transition can influence learning again without requiring the agent to immediately repeat the environmental interaction.

Result

A real transition supports an immediate direct-learning update and can later support one or more planning updates through the model.

What happens after the agent experiences one transition, and how can that transition be reused?

Key point

The model does not replace the environment. It preserves information from real interaction so that the planning component can reuse that information later.

The Planning Budget n

Definition

In Dyna-Q, n specifies the number of planning steps performed per real step. It is useful to interpret n as a planning budget: n = 0 means no additional planning steps, while larger values add more simulated updates after each real interaction.

Agent	Planning after each real step	Reported improvement
n = 0	No planning; direct reinforcement learning only	About 25 episodes
n = 5	Five planning steps	About 5 episodes
n = 50	Fifty planning steps	About 3 episodes

Reported learning speed in the Dyna Maze experiment. The values describe the experiment's settings and results, not a universal rule for every task.

How does increasing n change the number of simulated updates and the reported learning speed?

The comparison used the same kind of maze and varied how much planning occurred after each real step. The first episode was the same for every n value because the random-number-generator seed was held constant across the algorithms. That episode took about 1700 steps and was not shown in the reported curves.

How Planning Spreads Policy Information

Direct learning and planning differ in how quickly useful information can spread through the maze. With n = 0, the agent relies only on direct reinforcement learning. In the halfway-through-the-second-episode comparison, each episode adds only one more step to the learned policy, so only the final step has been learned at that point. With n = 50, the first episode still directly teaches only one step, but planning during the second episode develops a much larger policy while the agent is still wandering near the start. By the end of the second episode, the policy reaches almost back to the start state. By the end of the third episode, the source reports a complete optimal policy and perfect performance.

How does planning propagate information from the goal toward the start more quickly than direct learning alone?

Planning spreads information because the model allows previously recorded transitions to be replayed as simulated experiences. Once useful information has been obtained near the goal, repeated planning updates can affect earlier parts of the learned policy while the agent is still exploring other parts of the maze.

Known Pairs and Search Control

Search control selects the starting states and actions for simulated experiences. Dyna-Q restricts these selections to state-action pairs that the agent has already experienced. That restriction matters because the table-based model has recorded reward and next-state information for experienced pairs. When search control selects one of those pairs, the model can return a simulated experience based on what was recorded.

Why does planning begin with a previously experienced state-action pair?

Mistakes About Dyna-Q

Common mistake
Treating planning as a replacement for real interaction
What it looks like
Assuming that an agent with n = 50 no longer needs to experience the maze.
Why it fails
The comparison describes planning as additional computation performed after real steps. The agent still interacts with the maze and builds its model from real experience.
Do this instead
Think of n as an additional planning budget applied to what the agent has already experienced.
Common mistake
Assuming n changes the maze
What it looks like
Reading n = 5 and n = 50 as two different environments.
Why it fails
The experiment compares different amounts of planning on the same kind of task.
Do this instead
n changes the number of planning steps performed after each real step, not the maze structure.
Common mistake
Assuming the first episode proves that planning is faster
What it looks like
Comparing different first-episode paths as evidence for the effect of n.
Why it fails
The first episode was the same for every n value because the random-number-generator seed was held constant across the algorithms.
Do this instead
Compare what happens after the shared initial experience, when the agents use different planning budgets.
Common mistake
Planning from arbitrary unexperienced pairs
What it looks like
Asking the model for a transition for a state-action pair that the agent has never encountered.
Why it fails
The table-based model has no recorded reward and next-state prediction for that pair.
Do this instead
Let search control select from previously experienced state-action pairs.
Common mistake
Treating the reported n = 50 result as a universal rule
What it looks like
Concluding that the largest possible planning budget is always best.
Why it fails
The reported learning speeds came from a particular maze experiment and particular parameter settings.
Do this instead
Use the result to understand the effect of additional planning in this experiment, not as a universal guarantee.

Check Your Understanding

MEDIUM

A Dyna-Q agent has just experienced a state-action-reward-next-state transition. Explain two different ways this transition can affect the agent: one immediate use and one later use. Then explain why a planning step should select a previously experienced state-action pair.

Hints

Separate direct reinforcement learning from model-based planning.
Identify what the model records from the real transition.
Ask whether the model has a prediction for an unexperienced pair.

Your attempt (not saved)

MEDIUM

Compare n = 0 and n = 50 in the reported Dyna Maze experiment. Describe what happens after each real step and why the n = 50 agent can develop a larger policy while it is still wandering near the start.