Reinforcement Learning Problems
Two reinforcement learning problems may involve similar agents and environments but still require different approaches.
The Agent's Starting Knowledge
Two reinforcement learning problems may involve similar agents and environments but still require different approaches. The decisive question is not only what the environment is like. It is what the agent already knows about how that environment behaves.
The central classification question is: what model of the environment is available to the agent?
Reading an Environment Dynamics Model
A model of an environment's dynamics describes what can happen after the agent takes an allowable action in a state. In an MDP, the model contains two kinds of information for all states and allowable actions: one-step transition probabilities and expected rewards.
The model therefore connects a current state and an allowable action with information about the next state and the reward expected from that one-step interaction. To count as complete knowledge, this information must cover the relevant states and allowable actions and must be complete and accurate.
Complete Knowledge
A complete-knowledge problem provides the agent with a complete and accurate dynamics model of the environment.
For an MDP, completeness means that the model covers all states and allowable actions relevant to the problem. It supplies both one-step transition probabilities and expected rewards. Accuracy matters as well: a model that is available but not correct does not satisfy the requirement for complete knowledge.
Classifying a Supplied Model
An agent begins with a model that covers every relevant state and allowable action. For each one-step interaction, the model provides transition probabilities and expected rewards, and the model is accurate. What kind of reinforcement learning problem is this?
- 1Check coverage
The model covers the relevant states and allowable actions.
- 2Check information
The model includes one-step transition probabilities and expected rewards.
- 3Check accuracy
The model is described as accurate, so the available information correctly represents the environment's dynamics.
- 4Classify
Because the complete and accurate dynamics model is available from the beginning, this is a complete-knowledge problem.
Complete knowledge
Incomplete Knowledge
An incomplete-knowledge problem lacks a complete and perfect model of the environment's dynamics.
The model may be unavailable as a complete source of information about the relevant states, allowable actions, transition probabilities, and expected rewards. The key point is the absence of a complete and perfect model, not simply the fact that the agent knows little in a general sense.
Finding the Missing Model
An agent begins in an environment where the complete and perfect dynamics model is not available. The agent does not initially have complete transition-probability and expected-reward information for all relevant states and allowable actions. How should the problem be classified?
- 1Inspect the initial information
The agent does not begin with a complete and perfect model.
- 2Compare with the dividing line
Complete knowledge requires a complete and accurate dynamics model.
- 3Classify
Because that complete model is unavailable, the problem has incomplete knowledge.
Incomplete knowledge
Classifying by Initial Information
To classify a problem, inspect what the agent knows at the start. Do not classify it only from the environment's description. The same kind of environment can belong to different categories if one agent begins with a complete and accurate model while another does not.
A problem description says that an agent begins with a model covering all relevant states and allowable actions, including one-step transition probabilities and expected rewards. The model is accurate. Classify the problem and name the evidence that supports your answer.
Hints
- Check whether the model is complete.
- Check whether the model is accurate.
- Look for both transition-probability and expected-reward information.
A second problem uses a similar environment, but the agent begins without a complete and perfect model of the environment's dynamics. Classify the problem. Explain why the environment alone is not enough to determine the classification.
Hints
- The dividing line concerns the model available to the agent.
- Compare the agent's initial information with the requirements for complete knowledge.
Common mistake Treating any model as complete knowledge.
What it looks likeThe agent has some information about how the environment behaves, so the problem is called complete-knowledge.
Why it failsComplete knowledge requires a complete and accurate model covering the relevant states and allowable actions and supplying transition-probability and expected-reward information.
Do this insteadCheck completeness and accuracy rather than merely checking whether some model information exists.
Common mistake Classifying the problem from the environment alone.
What it looks likeThe environment is familiar, so the problem must have complete knowledge.
Why it failsThe classification depends on what the agent initially knows about the environment's behavior.
Do this insteadInspect the model available to the agent at the beginning of the problem.
Common mistake Ignoring expected rewards.
What it looks likeThe model gives transition probabilities, so it is complete.
Why it failsIn an MDP, the model contains both one-step transition probabilities and expected rewards for all states and allowable actions.
Do this insteadVerify that both kinds of information are included.
Common mistake Ignoring model accuracy.
What it looks likeThe agent has information covering every state and action, so knowledge is complete even if the information is not accurate.
Why it failsComplete knowledge requires a complete and accurate dynamics model.
Do this insteadTreat accuracy as a required part of the classification.
Classification Checklist
- 1Identify the model of the environment's dynamics available to the agent at the start.
- 2Check whether the model covers the relevant states and allowable actions.
- 3Check whether it provides one-step transition probabilities and expected rewards.
- 4Check whether the model is complete and accurate.
- 5Classify the problem as complete knowledge only when the complete and accurate model is available; otherwise classify it as incomplete knowledge.
- 1Complete knowledge means that the agent begins with a complete and accurate dynamics model. In an MDP, that model includes one-step transition probabilities and expected rewards for all states and allowable actions. Incomplete knowledge means that a complete and perfect model is unavailable. The classification depends on the agent's initial information, not merely on the environment itself.
Key Takeaways
- The central question in a reinforcement learning problem is what model of the environment is available to the agent.
- Complete knowledge requires a complete and accurate dynamics model.
- In an MDP, the model includes one-step transition probabilities and expected rewards for all states and allowable actions.
- Incomplete knowledge means that a complete and perfect model is unavailable.
- The classification is determined by the agent's initial information, so similar environments can produce different problem classifications.