State-Action Associations

A state-action association focuses attention on what happens when an action is taken in a particular state.

Programming

A Change That Tests Learning

Imagine an animal encountering an environment before it has a reason to travel toward any particular location. Later, something changes unexpectedly. The animal now needs to decide how to behave. The important question is whether it can use an internal representation of the environment to plan, rather than relying only on previously learned state-action associations.

Psychology uses the term cognitive map for this kind of mental representation. Reinforcement learning uses the related idea of an environment model: a computational representation that mimics how the environment behaves. Both ideas describe a shift from remembering isolated action patterns toward representing relationships within an environment.

What contains the information that connects an environment's states, actions, and transitions?

Tracing an Environment Representation

1
The agent encounters an environment and gains experience about it.
2
Information from that experience contributes to a representation of the environment.
3
The representation captures relationships relevant to how the environment behaves.
4
At a later time, the agent can use the representation to support planning.
5
If circumstances change unexpectedly, planning can use the representation instead of depending only on previously repeated action patterns.

The representation is not described as merely a record of past actions. Its later purpose is planning behavior. This distinction matters because a record of what happened before may not directly answer what to do after an unexpected change, whereas a representation of the environment can support reasoning about possible behavior.

How can a learned representation support a new action sequence when the environment changes?

In the real world

Generated example: Suppose an animal has encountered several connected locations but has not yet needed to reach one particular location. If a familiar route later becomes unusable, the animal's behavior could be guided by its representation of how locations are related. The point of the example is not that every animal must behave this way; it illustrates the source's question about whether a learned representation can guide behavior after an unexpected change.

Representation Versus Reward Learning

Learning an environment representation and learning directly from reward signals are different perspectives on learning. In direct reward-based learning, the emphasis is on which actions become associated with outcomes in particular situations. In the representation perspective described by the source, the agent can learn about the environment without relying on reward signals, then use that representation later for planning.

This does not mean reward is irrelevant to all reinforcement learning. The narrower claim is that a cognitive map or environment model can be learned without reward serving as the signal that creates the representation. The source presents this as a shared learning property of these representations.

Learning perspective	Main emphasis	Role of reward	Later use
State-action associations	Associations between situations and actions	The source contrasts this perspective with representation learning	Can describe behavior through learned action patterns
Environment representation	How the environment is organized and behaves	The representation can be learned without relying on reward signals	Can support planning behavior

The source distinguishes a representation-based perspective from explaining behavior only through state-action associations.

What differs between learning environment relationships and learning action patterns from reward signals?

From State to Action to Outcome

A state-action association focuses attention on what happens when an action is taken in a particular state. The action may be connected with a resulting state and possibly a reward. An environment representation goes further in emphasis: it represents how parts of the environment are related and how the environment behaves, so that the representation can later support planning.

How does an action taken in one state connect to a new state and a possible reward?

Worked example

Interpreting a Changed Route

Generated example: An agent has learned relationships within an environment. A later change makes its familiar route unusable. What does the representation-based explanation focus on?

1
Identify the limitation
A previously repeated action pattern may no longer fit the changed circumstances.
2
Use the learned representation
The agent can use its internal representation of the environment as a basis for considering behavior after the change.
3
Plan behavior
The important function of the representation is to support planning, rather than merely to store the sequence of actions that occurred before.

Result

The representation-based account explains changed behavior through later planning supported by an environment representation, not only through previously learned state-action associations.

Mistakes About Cognitive Maps

Common mistake
Treating a cognitive map as a literal diagram with one required physical shape.
What it looks like
Assuming the concept requires a particular physical structure inside an animal.
Why it fails
The source says the sequence should be read as roles: experience provides information, a cognitive map is the resulting mental representation, and the representation can support planning.
Do this instead
Treat the cognitive map as a mental representation used to understand and navigate an environment, without claiming a specific physical shape.
Common mistake
Assuming the representation must be created by reward signals.
What it looks like
Concluding that an agent can represent an environment only after receiving a reason to reach a particular location.
Why it fails
The source identifies learning without relying on reward signals as a shared property of cognitive maps and environment models.
Do this instead
Separate learning the environment representation from learning which actions are favored by reward.
Common mistake
Using cognitive map and environment model as if they were perfectly identical terms.
What it looks like
Claiming that psychology and reinforcement learning use the same term for the same physical mechanism.
Why it fails
The source says the terms belong respectively to psychological and reinforcement learning descriptions and need not refer to identical physical mechanisms.
Do this instead
Describe them as related ideas that share a pattern: representation first, planning later.
Common mistake
Describing the representation as only a history of previous actions.
What it looks like
Saying that the representation matters only because it records what the agent did before.
Why it fails
The source emphasizes that the later purpose is planning behavior, not merely storing a record of past actions.
Do this instead
Explain how the representation can support behavior when circumstances change unexpectedly.

Practice the Distinction

MEDIUM

Generated practice: Write two or three sentences explaining why an environment model is more than a list of previously performed actions. Include the role of experience, the possibility of learning without relying on reward signals, and the later role of planning.

Hints

Begin with what an environment model represents.
Contrast planning with repeating a previously learned action.
Mention that the source connects cognitive maps and environment models without claiming they are identical physical mechanisms.

Your attempt (not saved)

What do you think happens?

An agent learned about an environment before having a reason to reach a particular location. If the environment later changes unexpectedly, which explanation best matches the representation-based perspective?

The agent must rely only on the exact actions that previously produced reward.
The agent may use a learned representation of the environment to support planning.
The agent cannot use earlier experience because it was not rewarded at the time.

Reveal answer

Answer:

The agent may use a learned representation of the environment to support planning.

The source describes cognitive maps and environment models as representations that can be learned without relying on reward signals and later used to plan behavior, especially when circumstances change unexpectedly.

The Planning Shift

State-action associations describe one way to understand behavior: actions can become associated with particular states and their outcomes. Cognitive maps and environment models add another perspective. They represent the environment, can be learned without relying on reward signals, and can later support planning. This perspective becomes especially important when an environment changes unexpectedly, because behavior need not be explained only as the repetition of previously learned action associations.

Key Takeaways

A cognitive map is a mental representation of an environment used to understand and navigate it.
An environment model is the related reinforcement learning idea of a computational representation that mimics how an environment behaves.
These representations can be learned without relying on reward signals.
Their later purpose is planning behavior, not merely storing a record of past actions.
Representation-based learning offers an alternative to explaining behavior only through state-action associations, particularly after unexpected environmental changes.

Related concepts

More concepts starting with S