Planning Backups
State-space planning can be understood as sequences of backups.
Planning as a Sequence
State-space planning can be understood as a sequence of backups. Instead of treating a planning method as a single indivisible procedure, examine it one backup at a time. Each backup performs an update, and the planning strategy also determines which state receives attention next.
Two Planning Decisions
The phrase planning backup contains two separate decisions. First, what kind of backup is performed? This is the computation dimension: the update made during a backup. Second, which state receives attention next? This is the ordering dimension: the sequence in which backups are directed toward states. Two planning strategies may therefore differ because they perform different kinds of backups, because they order attention differently, or because they differ along both dimensions.
Separating Update from Attention
Imagine two planning procedures that use the same kind of backup but choose states in different orders.
- 1Identify the backup type
Ask what update each procedure performs. This identifies the computation dimension.
- 2Identify the backup order
List which state receives attention first, second, and later. This identifies the ordering dimension.
- 3Compare the procedures
If the update is the same but the state order differs, the strategies differ in backup order rather than backup type.
A planning strategy is easier to analyze when its update rule and its attention order are examined separately.
Backward Focusing
Backward focusing is one way to organize attention during planning. The central idea is to begin with a target state or outcome and direct attention toward states that lead backward from it. In a conceptual sequence, the target receives attention first, followed by relevant predecessor states, with the order continuing according to the chosen backward organization.
Forward Focusing
Forward focusing organizes attention according to reachability from frequently visited states under the current policy. The starting point is therefore connected to the behavior currently being followed: states visited frequently under that policy receive importance, and other states are prioritized according to how reachable they are from those frequently visited states.
Following Reachability
Suppose a current policy frequently visits state A and can reach states B and C from states that are frequently visited. A different state, D, is not identified as reachable from those frequently visited states.
- 1Find the reference states
Use the states frequently visited under the current policy as the starting points for attention.
- 2Consider reachability
Compare other states according to whether they are reachable from those frequently visited states.
- 3Order attention
Prioritize reachable states such as B and C rather than treating every state as equally connected to the policy's frequently visited states.
This illustrates the organizing principle of forward focusing: policy-based frequent visitation supplies the reference, and reachability supplies the basis for prioritizing attention.
Two Ways to Order Attention
Backward and forward focusing answer the same broad planning question—where should attention go next—but use different reference points. Backward focusing organizes attention by looking from a target or outcome toward predecessor states. Forward focusing organizes attention from frequently visited states under the current policy and uses reachability to prioritize other states.
| Dimension | Backward focusing | Forward focusing |
|---|---|---|
| Reference point | A target state or outcome | Frequently visited states under the current policy |
| Ordering idea | Organize attention toward predecessor states | Prioritize states according to reachability |
| Role in planning | One way to organize backup attention | One way to organize backup attention |
| What it does not determine by itself | The backup type | The backup type |
The comparison concerns how attention is ordered; backup type is a separate planning dimension.
Common Reasoning Errors
Common mistake Treating backup type and backup order as the same thing.
What it looks likeDescribing only which states are selected and calling that the entire planning method.
Why it failsA planning strategy includes both the kind of backup being performed and the order in which backups receive attention.
Do this insteadAnalyze the update computation and the attention sequence separately.
Common mistake Assuming backward focusing is the definition of planning.
What it looks likeConcluding that every planning sequence must begin at an outcome and move backward.
Why it failsBackward focusing is presented as one way to organize attention during planning.
Do this insteadTreat backward focusing as one possible ordering strategy and compare it with forward focusing.
Common mistake Describing forward focusing as simply moving forward through every state.
What it looks likeIgnoring the current policy and listing states without considering frequent visitation or reachability.
Why it failsForward focusing selects attention according to reachability from frequently visited states under the current policy.
Do this insteadIdentify the policy-based frequently visited states first, then consider reachability from them.
Common mistake Using the names of methods without examining their underlying operations.
What it looks likeComparing two method names without asking what update is made or which state receives attention next.
Why it failsThe sequence-of-backups perspective directs attention beneath individual method names.
Do this insteadDescribe each strategy in terms of backup type and backup order.
Practice the Distinction
A planning description says: “The method uses a particular backup update, begins by considering states frequently visited under the current policy, and then prioritizes other states according to reachability from those states.” Identify the backup type dimension and the backup order dimension. Which focusing strategy does the ordering description resemble?
Hints
- The backup type is identified by asking what update the method performs.
- The backup order is identified by asking which states receive attention and how they are prioritized.
- Frequent visitation under the current policy followed by reachability describes forward focusing.
- 1To analyze the description, separate the update from the ordering. The particular backup update belongs to the backup type dimension. Starting from frequently visited states under the current policy and prioritizing according to reachability belongs to the backup order dimension and resembles forward focusing.
Practical Reading Strategy
When encountering a new planning method, read beneath its name by asking two questions. What kind of backup is being performed? Which state receives attention next, and why? This method of analysis keeps computation and attention distinct. It also makes it possible to compare methods without assuming that one ordering strategy is the only valid approach.
Key Takeaways
- State-space planning can be understood as a sequence of backups.
- A planning strategy has two important dimensions: the kind of backup performed and the order in which backups receive attention.
- Backward focusing organizes attention from a target or outcome toward predecessor states.
- Forward focusing prioritizes states according to reachability from frequently visited states under the current policy.
- Backward and forward focusing are alternative ways to organize attention, not a reason to ignore the separate backup type dimension.