State Values and Action Values
A transition graph organizes the possible one-step outcomes that follow a current state and a selected action.
A Decision Is Only the Start
When an agent chooses an action, the model must describe more than the choice itself. It must also describe what the environment may produce afterward: a new situation and a reward. The Recycling Robot makes this visible. It may search for a can, wait, or return to its home base to recharge. Each action can have different consequences, so the environment's response belongs in the model.
A transition graph organizes the possible one-step outcomes that follow a current state and a selected action.
Following One Transition
Read the graph from left to right. Begin with the current state. Follow the selected action. Then inspect every branch leaving that action. Each branch represents a possible pair: a next state together with a reward. The graph therefore records the environment's possible one-step behavior rather than merely listing which actions are available.
What the Dynamics Record
For a chosen current state and action, the MDP records the probability of each possible pair consisting of a next state and a reward. The source describes these one-step dynamics as the complete description of a finite MDP's environment behavior. This is the most complete view in the supplied material because it preserves both which next state can occur and which reward accompanies that outcome.
| Description | What it preserves | What it summarizes |
|---|---|---|
| Complete one-step dynamics | The probability of every next-state-and-reward pair | The full one-step environment behavior |
| State-transition probabilities | Probabilities of moving from one state to another | The state movement, without the complete pair detail |
| Expected rewards | The reward expected for an action in a state | The reward aspect for a state-action pair |
The source distinguishes complete dynamics from two useful quantities derived from them.
State-transition probabilities and expected rewards are useful views, but each leaves out some detail compared with recording the probability of every next-state-and-reward pair. This distinction matters when two outcomes have different combinations of next states and rewards.
Tracing the Recycling Robot
Following a search decision
Trace what must be recorded after the Recycling Robot chooses to search.
- 1Start with the current state
Identify the situation in which the robot reaches a decision point.
- 2Record the selected action
The robot chooses search. The action is part of the one-step description.
- 3List possible outcomes
Record every possible next state and the reward associated with that next state. The source emphasizes that one action may have multiple outcomes with different rewards.
- 4Attach probabilities
For each next-state-and-reward pair, record its probability. This produces the complete one-step description for that state and action.
The result is not merely the label search. It is the set of possible next-state-and-reward pairs, together with the probability of each pair.
The same tracing method applies to waiting and returning to the home base to recharge. Start at the current state, follow the chosen action, and then record all possible next states and rewards. The action name alone does not tell us the environment's full response.
Values and the Supplied Model
The article title names state values and action values, but the supplied source material focuses on the transition graph and the environment's one-step dynamics. It explains expected rewards for state-action pairs, yet it does not provide formal numerical definitions or calculation rules for state values or action values. Therefore, the grounded distinction available here is between the current state, the selected action, and the possible next-state-and-reward outcomes that follow.
| Model element | Question it organizes | Detail supplied by the source |
|---|---|---|
| Current state | What situation is the robot in at the decision point? | The starting point of a one-step transition |
| Selected action | What does the robot choose to do? | Search, wait, or return to the home base to recharge |
| Next state and reward | What can the environment produce after the action? | A possible pair recorded by the one-step dynamics |
| Expected reward | What reward is expected for this state-action pair? | A useful quantity derived from the complete dynamics |
This table separates the transition ingredients described in the supplied material.
Common Tracing Mistakes
Common mistake Treating an action as if it had only one guaranteed outcome.
What it looks likeAssuming that choosing search directly identifies one next state and one reward.
Why it failsThe Recycling Robot example shows why one action may have multiple possible outcomes with different rewards.
Do this insteadAfter naming the action, list every possible next-state-and-reward pair and its probability.
Common mistake Recording only the next state.
What it looks likeWriting down where the robot goes but omitting the reward.
Why it failsThe complete one-step description records pairs consisting of a next state and a reward.
Do this insteadKeep the next state and its associated reward together.
Common mistake Calling state-transition probabilities the complete environment dynamics.
What it looks likeUsing only the probabilities of moving from one state to another.
Why it failsState-transition probabilities are derived from the complete dynamics and leave out some detail.
Do this insteadUse the complete next-state-and-reward probabilities when the full one-step behavior is required.
Common mistake Confusing expected rewards with the full set of outcomes.
What it looks likeKeeping only the expected reward for a state-action pair.
Why it failsExpected rewards are useful summaries, but they do not preserve every possible next-state-and-reward pair.
Do this insteadTreat expected rewards as a derived view rather than a replacement for the complete dynamics.
Practice the Graph
The Recycling Robot is at a decision point and chooses to wait. Describe what a complete one-step record must contain, without inventing particular state names, reward amounts, or probabilities.
Hints
- Begin with the current state and the selected action.
- List every possible next-state-and-reward pair.
- Attach the probability of each pair.
- Explain why an expected reward alone would not be the complete record.
- 1A strong answer identifies waiting as the selected action, then records all possible next states and their rewards together with the probability of each combined outcome. It also distinguishes that complete record from an expected reward, which is only a derived summary.
Key Takeaways
- A transition graph begins with a current state and a selected action, then branches to possible next states and rewards.
- The complete one-step dynamics record the probability of every possible next-state-and-reward pair.
- State-transition probabilities and expected rewards are useful quantities derived from the complete dynamics, but each omits some detail.
- The Recycling Robot may search, wait, or return to its home base, and one action may lead to multiple outcomes with different rewards.
- The supplied material explains transition dynamics and expected rewards, but it does not provide formal numerical definitions for state values or action values.