Concepts / Function Approximation for Continuous State Spaces

Function Approximation for Continuous State Spaces

Polynomial basis functions provide a family of candidate functions for approximation.

  • Programming

From Samples to a State-Space Function

A learning problem may provide function values only at some samples, while the desired result is a function defined between those samples or even beyond them. Reinforcement learning faces a closely related challenge when its state space is continuous and has multiple dimensions: the learning system needs a way to approximate a function across that space. Polynomial basis functions are relevant because they provide one family of candidate functions for this approximation task.

Observed statesampleslimited informationApproximatingfunctionvalues between or beyondsamplesContinuous statespacemultiple dimensionsprovides sampled informationdefines the domain
How can a finite set of observed states and values represent a function defined over an entire continuous state space?

A Candidate Family in Action

Think of approximation as choosing a candidate function that can stand in for the function the learning system wants to represent. A polynomial family supplies several possible candidate shapes. The important introductory question is not whether you can memorize every polynomial family. It is whether a polynomial family is an appropriate approximation tool for the problem at hand.

Stateone point in the statespacePolynomial basisfunction Acandidate componentFunction estimateapproximated valuePolynomial basisfunction Bcandidate componentevaluated atevaluated atcontributescontributes
How do individual polynomial basis functions contribute to an approximating function for a given state?
Worked example

A Small Approximation Scenario

Imagine that a learning problem reveals values for a few states, but the state space contains many continuous, multi-dimensional possibilities. How could a polynomial family help?

  1. 1
    Identify the information available

    The system has values at some samples rather than a complete description of the function over every state.

  2. 2
    Identify the missing result

    The system needs a function that can provide values between the samples or beyond them.

  3. 3
    Choose a candidate family

    A polynomial family can serve as one family of candidate functions for constructing an approximation.

  4. 4
    Relate the choice to reinforcement learning

    The same general challenge appears when reinforcement learning must approximate a function over a continuous state space with multiple dimensions.

Result

Polynomial basis functions do not replace the overall approximation task. They provide one family of candidate functions that may be used within that task.

Interpolation, Regression, and Reinforcement Learning

Interpolation and regression provide a useful way to understand the structure of reinforcement-learning function approximation. In the interpolation and regression setting, function values are available at some samples, and the goal is to define a function between those samples or even beyond them. In reinforcement learning, the corresponding challenge appears over a continuous, multi-dimensional state space: the system must approximate a function across states using limited or sampled information. The transfer is useful because the underlying challenge is related in both settings: a function must be defined from incomplete or sampled information.

Sampled informationknown function valuesInterpolationbetween samplesApproximated functionvalues across a domainRegressionbetween or beyond samplesReinforcementlearningcontinuousmulti-dimensional statessupportssupportsmotivates approximationdefines valuesdefines valuesapproximates over states
How do known state-value samples become predictions for unseen states, and what shared structure connects interpolation, regression, and reinforcement learning?
IdeaRole in the shared structure
Sampled function valuesThe available information is limited to some samples.
InterpolationA function is defined between known samples.
RegressionA function may be defined between samples or beyond them.
Reinforcement-learning approximationA function is approximated over a continuous state space with multiple dimensions.

The source of the connection is the shared need to define a function from limited or sampled information.

Basis Functions Versus Approximation

Function approximation is the broader task: define an approximate function when the desired function is known only through limited or sampled information. Polynomial basis functions belong to the toolset for carrying out that task. They provide a family of candidate functions, but naming a family is not the same as completing the approximation. Keeping these roles separate prevents a common misunderstanding: polynomial basis functions are a possible representation choice, while function approximation is the overall problem being solved.

Functionapproximationoverall taskCandidate functionfamilychoice of representationPolynomial basisfunctionsone available familymay usecan be
What is the difference between choosing a family of basis functions and carrying out the overall process of fitting an approximating function?

Choosing the Right Mental Model

MEDIUM

A learning problem provides values at some samples and needs a function for a continuous state space with multiple dimensions. Explain, in your own words, why function approximation is needed and where a polynomial family could fit.

Hints
  • Start with the difference between sampled information and a function defined across a space.
  • Mention the connection to interpolation and regression.
  • Separate the overall approximation task from the choice of polynomial basis functions.
  • Common mistake

    Treating polynomial basis functions as the same thing as function approximation.

    What it looks like

    Saying that selecting a polynomial family automatically completes the approximation problem.

    Why it fails

    A polynomial family is one family of candidate functions, while function approximation is the broader task of defining an approximate function from limited or sampled information.

    Do this instead

    Describe the polynomial family as a possible tool within the larger approximation process.

  • Common mistake

    Missing the connection to interpolation and regression.

    What it looks like

    Discussing reinforcement-learning approximation without relating it to the problem of defining values between or beyond known samples.

    Why it fails

    The relevance of polynomial families to reinforcement learning comes from this shared approximation structure.

    Do this instead

    Begin with the sampled-information setting, then connect it to approximation over continuous, multi-dimensional state spaces.

  • Common mistake

    Assuming that continuous, multi-dimensional state spaces do not change the problem.

    What it looks like

    Treating a continuous state space as though a complete list of every state and its value were already available.

    Why it fails

    The challenge is to approximate a function over the space when information is limited or sampled.

    Do this instead

    Emphasize the need for a function that extends across the state space rather than only recording the observed samples.

  • Common mistake

    Focusing on memorizing polynomial families instead of recognizing when a family is appropriate.

    What it looks like

    Trying to recall every polynomial family before explaining why a polynomial family might be useful.

    Why it fails

    The central introductory skill is identifying when a polynomial family is an appropriate approximation tool.

    Do this instead

    First identify the approximation problem and its sampled information; only then consider a candidate family.

Key Takeaways

  1. 1
    Polynomial basis functions provide one family of candidate functions for approximation.
  2. 2
    Interpolation and regression involve defining a function from values available at some samples, between those samples or even beyond them.
  3. 3
    Reinforcement learning encounters a related challenge when it approximates a function over a continuous, multi-dimensional state space.
  4. 4
    Function approximation is the broader task; selecting a polynomial family is one possible tool within that task.
  5. 5
    The key introductory skill is recognizing when a polynomial family is an appropriate approximation tool rather than memorizing every family.

At a glance

  • Polynomial basis functions are candidate functions used within the broader task of function approximation.
  • The connection to interpolation and regression comes from the shared need to define a function from limited or sampled information.
  • Continuous, multi-dimensional state spaces make approximation important because the learning system must represent a function across the space.
  • The main introductory skill is deciding when a polynomial family is an appropriate approximation tool.

Related Topics

Related concepts

More concepts starting with F