Prediction and Control in Reinforcement Learning

Consider an animal that performs an action and then receives something it likes. If that action becomes more likely later, the important learning event is not simply that the animal experienced a reward. The reward followed a particular behavior, and the consequence changed the future tendency to produce that behavior.

This is a control relationship. A behavior leads to a consequence, and that consequence is associated with a change in future behavior. In reinforcement learning, the algorithmic counterpart is policy improvement: the agent's future action choices are changed in light of what follows its actions.

For the instrumental arrangement described here, a reinforcing stimulus must be contingent on the behavior. Contingent means that the consequence occurs because it follows a particular behavior in the learning arrangement, rather than being merely an event the learner experiences independently of that behavior.

This contingency-based distinction also limits the connection with classical conditioning. Prediction algorithms and classical conditioning can both involve predicting upcoming stimuli. However, the predicted stimulus does not have to be a reward or penalty that evaluates an earlier action. Therefore, prediction is not automatically an instrumental or classical-conditioning process; its meaning depends on what is being predicted and how that quantity relates to behavior and consequences.

A reinforcement-learning policy describes the agent's action choices. Policy improvement changes those choices so that some actions become more likely and others become less likely. Rewards increase the tendency toward rewarded behavior, while penalties decrease the tendency toward penalized behavior.

The diagram should be read as a control loop, not as a claim about one particular internal mechanism. The experimental consequence supplies the direction of change: a rewarded behavior becomes more likely, whereas a penalized behavior becomes less likely. Reinforcement learning's policy-improvement algorithms provide an algorithmic counterpart to this form of control.

A useful way to understand prediction is to follow one target quantity through time. First identify the environmental feature to be predicted. Next consider how that feature is expected to unfold as the agent continues interacting with the environment. Finally produce an estimate of the future-dependent quantity.

When a prediction algorithm estimates future reward under a particular policy, it is performing policy evaluation. The estimate answers a question about the policy being followed: what amount of future reward is expected as the agent continues interacting with the environment?

Future reward is the main reinforcement-learning example, but it is not the boundary of prediction. Prediction algorithms can estimate upcoming stimuli and other numerical-valued environmental features. The important question is not whether the target is called a reward; it is whether the target is a quantity whose expected future behavior can be estimated.

Many reinforcement-learning questions concern what has not happened yet. An agent may need an estimate of the reward it can expect while it continues interacting with the environment. That estimate provides information about the consequences associated with the policy being evaluated.

The connection is indirect but important. A prediction algorithm estimates future reward associated with the policy being evaluated. Policy evaluation is an integral component of algorithms for improving policies, so prediction can support improvement without being identical to improvement itself.