Function Approximation and Reinforcement Learning
The defining operation is generalization. Function approximation is not simple example storage: it uses available examples to construct a representation intended to cover more of the desired function.
From Limited Experience to a Broader Estimate
A reinforcement-learning system may have only some examples of the function it needs. The central problem is then not simply how to store those examples, but how to use them to estimate the function more broadly. Function approximation addresses this problem by turning examples of a desired function into a broader estimate.
The defining operation is generalization. Function approximation is not simple example storage: it uses available examples to construct a representation intended to cover more of the desired function.
Three Different Objects
It is important to keep three parts of the process separate. The desired function is the function the system needs to represent. The examples are the evidence currently available from that function. The approximation is the broader representation constructed from those examples. In reinforcement learning, the desired function can be a value function.
Separating Evidence from the Estimate
Suppose a reinforcement-learning system has a few examples associated with a value function. Identify what each part represents.
- 1Desired function
The value function is the function the system is trying to represent.
- 2Examples
The available value-related examples are the evidence the system currently has.
- 3Approximation
The broader estimate constructed by generalizing from those examples is the function approximation.
- 4Boundary
The examples are inputs to the process; the approximation is the result produced by generalization.
The approximation is not identical to the examples. It is a broader representation built from them.
Why This Is Supervised Learning
Function approximation is an instance of supervised learning used within reinforcement-learning algorithms. The examples associated with the desired function provide the evidence from which an approximation is constructed. This connection means that reinforcement learning can combine its own methods with established generalization methods rather than requiring an entirely separate approach to generalization.
The word supervised identifies the role of examples in the process: they are associated with the function being represented and are used to construct a broader estimate. The reinforcement-learning setting determines where those examples come from and how the resulting approximation is used.
Inside the Reinforcement-Learning Method
Within a reinforcement-learning algorithm, a function approximator takes the role of the generalization component. Reinforcement-learning methods provide or use examples associated with the desired function, and the approximator constructs a broader estimate. That estimate can represent a value function and can then support the reinforcement-learning method. The exact practical fit depends on the particular approximation method and the particular reinforcement-learning algorithm.
When evaluating a possible approximator, ask two separate questions. First, can the method serve as a function approximator in principle? Second, how naturally does it fit inside the particular reinforcement-learning algorithm? Theoretical suitability does not imply equal practical convenience.
Available Approximation Traditions
The source identifies a broad range of fields that provide possible approximation methods. These include machine learning, artificial neural networks, pattern recognition, and statistical curve fitting. In theory, a method studied in any of these fields can take the role of a function approximator within a reinforcement-learning algorithm.
| Field | Possible contribution |
|---|---|
| Machine learning | General methods for constructing approximations from examples |
| Artificial neural networks | Methods that can be used as function approximators |
| Pattern recognition | Methods that can provide approximation approaches |
| Statistical curve fitting | Methods for fitting a broader representation to examples |
Fields identified as possible sources of function-approximation methods
Mistakes That Blur the Process
Common mistake Treating function approximation as example storage
What it looks likeThe system keeps the examples but makes no broader estimate.
Why it failsThe defining operation of function approximation is generalization, not simple example storage.
Do this insteadDescribe the approximation as a broader representation constructed from available examples.
Common mistake Calling the available examples the desired function
What it looks likeA few examples are presented as if they were the complete value function.
Why it failsThe desired function, the examples, and the approximation are distinct parts of the process.
Do this insteadIdentify the examples as evidence from the desired function and the approximation as the constructed estimate.
Common mistake Assuming reinforcement learning must invent its own generalization methods
What it looks likeThe learner is treated as unable to use methods from other fields.
Why it failsFunction approximation is an instance of supervised learning, and methods from several machine-learning and curve-fitting fields can provide approximators.
Do this insteadRecognize that reinforcement-learning methods can be combined with existing generalization methods.
Common mistake Assuming every possible approximator fits every algorithm equally well
What it looks likeA method that can serve as an approximator in theory is automatically treated as the most convenient choice.
Why it failsThe practical fit differs across methods and reinforcement-learning algorithms.
Do this insteadSeparate theoretical possibility from how naturally a method fits a particular algorithm.
Check Your Understanding
A reinforcement-learning system has several examples associated with a desired value function. Explain what function approximation contributes beyond retaining those examples.
Hints
- Name the role of the examples.
- Describe the result produced by generalization.
- State why the result is broader than simple storage.
Classify each item as the desired function, available evidence, or constructed approximation: the value function the system needs to represent; a set of examples associated with that function; a broader estimate built from those examples.
Hints
- The desired function is the target of representation.
- Examples are the evidence available to the learner.
- The approximation is the result of generalization.
A proposed method comes from statistical curve fitting. What two questions should you ask before using it as a function approximator in reinforcement learning?
Hints
- First ask whether it can serve as an approximator in principle.
- Then ask how naturally it fits the particular reinforcement-learning algorithm.
The Central Idea
- 1Function approximation turns examples of a desired function into a broader estimate.
- 2In reinforcement learning, the desired function can be a value function.
- 3The examples are evidence; the approximation is the generalized representation constructed from that evidence.
- 4Function approximation is an instance of supervised learning used within reinforcement-learning algorithms.
- 5Possible approximators can come from machine learning, artificial neural networks, pattern recognition, and statistical curve fitting, although practical fit differs across methods.
Key Takeaways
- Function approximation generalizes from limited examples to represent a desired function more broadly.
- The desired function, its available examples, and the constructed approximation are different things.
- In reinforcement learning, the desired function can be a value function.
- Function approximation is supervised learning used within reinforcement-learning algorithms.
- Methods from several related fields can provide approximators, but their practical fit depends on the reinforcement-learning algorithm.