Episodic and Continuing Tasks

An episodic task is organized into separate episodes. A continuing task does not naturally divide into identifiable episodes.

Programming

One Run or an Ongoing Interaction

A reinforcement-learning interaction can have a clear ending, or it can continue without a natural point at which one interaction ends and another begins. This distinction produces two task types: episodic tasks and continuing tasks. The key question is whether the interaction naturally divides into complete runs with identifiable beginnings and endings.

Key point

An episodic task is organized into separate episodes. A continuing task does not naturally divide into identifiable episodes.

The Episode Lifecycle

An episode is one complete run from a starting point to an ending point. To trace an episodic task, follow three stages. The agent begins from a standard starting state or from a state selected from a standard distribution of starting states. The agent and environment then interact while the episode is in progress. Finally, the interaction reaches a special terminal state. After the terminal state is reached, the task resets and another episode can begin.

What happens as an episodic interaction reaches its terminal state?

The State Sets S and S+

Episodic tasks use two related state sets. S contains all nonterminal states: states in which the episode has not yet ended. S+ contains all of the states in S together with the terminal state. The distinction matters because an agent can occupy states from S while an episode is in progress, but the terminal state must also be included when the complete state collection includes the episode boundary.

Which states belong to S, and what additional state does S+ include?

State collection	What it contains	Role in an episode
S	All nonterminal states	States before the episode has ended
S+	All states in S plus the terminal state	Includes the episode boundary

S names only the states before termination; S+ includes the terminal state as well.

Different Outcomes, One Boundary

Worked example

A win and a loss

How can two different game outcomes use the same terminal state?

1
First play
The agent moves through nonterminal states in S, reaches the terminal state, and receives a reward associated with a win.
2
Second play
The agent again moves through nonterminal states in S, reaches the same terminal state, and receives a reward associated with a loss.
3
Episode boundary
The terminal state identifies where each play ends. The reward identifies the outcome, so the boundary and the outcome do not need to be represented by separate terminal states.
4
Reset
After either outcome, the task resets before another play begins.

Result

Different outcomes can have different rewards while both episodes transition into one terminal state.

How can a win and a loss use one terminal state while keeping their rewards different?

Key point

The terminal state represents where an episode ends. The reward can represent what happened at that ending, so different outcomes do not require different terminal states.

Interactions Without Episodes

Definition

A continuing task is an agent-environment interaction that does not naturally divide into identifiable episodes. The interaction continues without limit instead of reaching a natural point where one run ends and another begins.

How does an interaction with identifiable episodes differ from one that continues indefinitely?

Feature	Episodic task	Continuing task
Organization	Naturally divides into separate episodes	Does not naturally divide into identifiable episodes
Ending	Reaches a terminal state	Has no natural point at which one interaction ends
After the ending	The task resets and another episode can begin	The interaction continues without limit

The presence or absence of a natural episode boundary distinguishes the two task types.

Why Infinite Returns Cause Trouble

For a continuing task, the interaction continues without limit, so the final time step is represented as T = ∞. This creates a problem for the standard return formulation: if rewards continue to accumulate across an unlimited sequence of time steps, the return can be infinite.

How can receiving +1 at every time step make accumulated return grow without bound?

Worked example

A reward of +1 forever

What happens when a continuing task supplies a reward of +1 at every time step?

1
First steps
The interaction supplies +1 at the first time step, then +1 again at the next time step, and continues doing so.
2
No final step
Because the task is continuing, its final time step is represented as T = ∞.
3
Accumulation
There is no terminal point at which reward accumulation stops. The standard return formulation can therefore produce an infinite return.

Result

Repeated +1 rewards across an interaction with T = ∞ illustrate why the standard return formulation becomes problematic for continuing tasks.

Common Classification Mistakes

Common mistake
Treating the terminal state as an ordinary middle-of-episode state.
What it looks like
Continuing to describe the same episode after the terminal state has been reached.
Why it fails
The terminal state marks the end of the episode, and a reset follows.
Do this instead
Separate the episode into its run toward the terminal state and the reset that begins the next episode.
Common mistake
Confusing S with S+.
What it looks like
S is described as containing the terminal state.
Why it fails
S contains only nonterminal states, while S+ also includes the terminal state.
Do this instead
Use S for states before termination and S+ when the terminal state must be included.
Common mistake
Assuming different outcomes require different terminal states.
What it looks like
Using one terminal state for a win and another terminal state for a loss solely because their rewards differ.
Why it fails
Different rewards can represent different outcomes even when both episodes share one terminal state.
Do this instead
Keep the episode boundary represented by the terminal state and use the reward to distinguish the outcome.
Common mistake
Calling every long interaction continuing.
What it looks like
Classifying an interaction as continuing only because one episode takes many time steps.
Why it fails
The defining issue is whether the interaction naturally divides into identifiable episodes, not merely how long one episode lasts.
Do this instead
Look for a natural terminal point followed by a reset.
Common mistake
Applying a finite-ending intuition to T = ∞.
What it looks like
Assuming that repeated +1 rewards must eventually stop because an ordinary episode would end.
Why it fails
A continuing task has no natural final time step, and the standard return formulation can produce an infinite return.
Do this instead
Check whether the task continues without limit before reasoning about its return.

Classify the Interaction

EASY

A task begins from a starting state, proceeds through nonterminal states, reaches an outcome, and then resets before another run begins. Classify the task and identify the roles of S, S+, the terminal state, and the reset.

Hints

Ask whether the interaction naturally divides into complete runs.
Identify which states occur before the episode ends.
Separate the episode boundary from the reward associated with its outcome.

Your attempt (not saved)

MEDIUM

A second task has no natural point at which one interaction ends and another begins. It continues without limit and supplies +1 at every time step. Classify the task and explain why the standard return formulation becomes problematic.