States and Actions in Reinforcement Learning
An afterstate is the state reached immediately after an action.
From Situation to Result
Reinforcement learning tasks can be understood as transitions. An agent begins with the task in its current state, takes an action, and the task moves into a new state. The state reached immediately after that action is called the afterstate.
The defining question is: what does the task look like immediately after the chosen action occurs?
The Transition Pattern
An afterstate is not the situation before the action and not the action itself. It is the state that follows the action immediately. To identify it reliably, trace the task in order: first describe the current situation, then identify the action by its immediate effect, and finally describe the resulting state.
When locating an afterstate, avoid starting with a broad description of everything that might happen later. Focus first on the immediate result of the chosen action.
Separating Before and After
A Simple State Trace
A task is in a current state. An agent chooses an action that produces a known immediate result. Identify the afterstate.
- 1Describe the current state
Start with the task as it exists before the action. This is the current state, not the afterstate.
- 2Name the action by its effect
Identify what the action immediately does. The immediate effect is the point at which the transition is defined.
- 3Describe the resulting state
The task as it exists immediately after that effect occurs is the afterstate.
The afterstate is the new state immediately following the action.
The visual separates three things that are easy to blend together: the original situation, the chosen action, and the state produced by that action. The afterstate is the last of these three.
Queuing as an Afterstate Task
Afterstates are not limited to games. They also apply to queuing tasks, because a queuing action can have a known immediate effect. Examples of such effects include assigning a customer, rejecting a customer, or discarding information.
In a queuing task, identify the action through what it immediately does. If the effect is assignment, the afterstate is the queue-related state immediately after that assignment. If the effect is rejection, the afterstate is the state immediately after rejection. If the effect is discarding information, the afterstate is the state immediately after that information is discarded.
| Task setting | Current state | Action identified by | Afterstate |
|---|---|---|---|
| Game | The current game situation | The immediate game effect | The game state immediately afterward |
| Queuing task | The current queue situation | Assignment, rejection, or discarding information | The queue state immediately after that effect |
The setting changes, but the state-to-action-to-afterstate pattern remains the same.
A Queue Transition Walkthrough
Consider a generated queuing example. Begin with the queue in its current state. The selected action assigns a customer. The action is named by that immediate effect, not by a later consequence. The afterstate is the queue state immediately after the customer has been assigned.
Finding the Afterstate in a Queue
A queuing task is in its current state. The chosen action has the immediate effect of rejecting a customer. What should be called the afterstate?
- 1Current state
This is the queue situation before the rejection occurs.
- 2Immediate effect
The action is identified as rejection because rejection is what occurs immediately.
- 3Afterstate
The afterstate is the queue situation immediately after the customer has been rejected.
The queue situation immediately after rejection is the afterstate.
Mistakes in State Tracing
Common mistake Calling the current state the afterstate
What it looks likeDescribing the queue before an assignment and labeling that description as the afterstate.
Why it failsThe afterstate is reached immediately after the action, while the current state exists before the action.
Do this insteadKeep the before-action situation as the current state and label the immediately following situation as the afterstate.
Common mistake Treating the action and the afterstate as the same thing
What it looks likeUsing assignment as though it were the resulting queue state.
Why it failsAssignment is the immediate effect that defines the action; the afterstate is the state produced by that effect.
Do this insteadName the action first, then describe the task immediately after that action.
Common mistake Assuming afterstates belong only to games
What it looks likeLooking for afterstates in a game but not in a queuing task.
Why it failsThe concept applies whenever an action produces a known immediate result, including assignment, rejection, or discarding information in queuing tasks.
Do this insteadApply the same transition test to any task: current state, immediate action effect, resulting afterstate.
Common mistake Focusing on a later result instead of the immediate result
What it looks likeDescribing what might happen later in a queue rather than the state directly after rejection.
Why it failsThe definition is tied to the state reached immediately after the action.
Do this insteadStop the trace at the first state produced by the chosen effect.
Practice the Transition
A task is in a current state. The agent chooses an action whose immediate effect is discarding information. Write a three-part description: the current state, the action's immediate effect, and the afterstate.
Hints
- The current state is the situation before discarding occurs.
- Name the action by its immediate effect.
- The afterstate is the situation immediately after the information is discarded.
What do you think happens?
A queuing action rejects a customer. Which description is the afterstate?
Reveal answer
The queue immediately after the rejection
An afterstate is the state reached immediately after an action. The rejection names the immediate effect, while the resulting queue situation is the afterstate.
The Essential Distinction
- 1An afterstate is the state reached immediately after an action.
- 2The current state describes the task before the action; the afterstate describes the task directly afterward.
- 3To find an afterstate, trace the immediate effect of the chosen action.
- 4Queuing actions can be defined by immediate effects such as assignment, rejection, or discarding information.
- 5The state-to-action-to-afterstate pattern applies beyond games whenever an action produces a known immediate result.
Key Takeaways
- An afterstate is the state produced immediately after an action is taken.
- The current state comes before the action, while the afterstate comes immediately after it.
- In a queuing task, identify the action through its immediate effect, such as assignment, rejection, or discarding information.
- Afterstates are a general way to understand transitions in tasks beyond games.