Value Prediction in Reinforcement Learning

A reinforcement learning agent often has to make decisions before it knows the complete result of those decisions. Value prediction gives the agent a forecast: how much total reward might be available in the future? A value estimate is therefore a long-run prediction of total future reward, not merely a description of the next immediate result.

The challenge becomes larger when an agent must predict values for many states and actions. Treating every prediction as an entirely separate case may not make good use of experience. Function approximation connects reinforcement learning with generalization: information from some states or actions can support value estimates for other states or actions. The central process is to use a reinforcement learning method to produce value-prediction backups, use those backups as training examples, and then evaluate the resulting approximation.

A useful way to follow the information is as a three-part chain. First, a reinforcement learning value-prediction method produces backups. Next, those backups serve as training examples for a function approximation method. Finally, the approximation is judged with a performance measure such as MSVE. The direction matters: MSVE evaluates the resulting approximation; it does not produce the training examples.

Function approximation is useful because a reinforcement learning system may need values for many states and actions. An approximation method can connect these predictions instead of treating every one as an entirely separate case. Experience associated with some states or actions can therefore contribute to estimating values for other states or actions.

Producing predictions is not enough to establish that an approximation method is performing well. The method is also considered through a measure such as MSVE. In practical reasoning, ask two separate questions: what information trains the approximation, and how is the resulting approximation evaluated? Backups answer the first question by providing training examples. MSVE helps address the second by measuring performance.

There is no single universal function approximation method for value prediction. The possible range is large, and the source material notes that too little is known about many alternatives to support a reliable evaluation or recommendation of all of them. For that reason, the discussion focuses on methods based on gradient principles.

Gradient-descent methods are one class of function approximation methods. Linear gradient-descent methods receive particular attention because they are simple, considered promising, and useful for exposing important theoretical issues. Their importance is therefore both practical and explanatory: their relative simplicity makes underlying questions easier to study.

The distinction is about what is being evaluated. V asks how promising a state is for the agent. Q asks how promising a particular action is. Both are long-run predictions rather than descriptions limited to the next immediate result.