Concepts / True and Approximate Value Functions

True and Approximate Value Functions

Value function approximation makes exact prediction for every state impossible in the described setting.

  • Programming

When Exact Prediction Is Impossible

A true value function assigns a true value to each state under a policy. An approximate value function tries to predict those values, but in the setting described here it cannot represent every state perfectly. That creates a problem: checking whether each individual state is exactly correct is no longer enough to judge the quality of the whole prediction.

Approximation couples state predictions. The same parameters used by an approximate value function can influence several state estimates. As a result, improving one estimate may change other estimates as well. Since all states may not be made exact at the same time, the learner needs an explicit objective that says how the total prediction quality should be evaluated.

State s1true valueState s1approximate valueState s2true valueState s2approximate valueState s3true valueState s3approximate valuecomparecomparecompare
What does it mean for an approximate value function to match or differ from the true value function at multiple states?

A Coupled Update

What do you think happens?

Suppose an update improves the approximate estimate for one state. What can happen to the estimates for other states when those estimates are coupled?

  • Only the updated state can change

  • Other state estimates can also change

  • All state estimates must become exact

Reveal answer
Answer:

Other state estimates can also change

The source describes approximate predictions as coupled: changing the parameters used by the value function can change several state estimates. Therefore, an update can improve one state while making another state more or less accurate.

State s1approximate valueState s1new approximate valueState s2approximate valueParameter updateshared approximationState s2new approximate valueState s3approximate valueState s3new approximate valueaffected by shared parametersaffected by shared parametersaffected by shared parameterschanges estimatechanges estimatechanges estimate
When one approximate value estimate is updated, which states can become more or less accurate, and how can their errors change?

The important consequence is that an update does not have to produce one isolated error change. One state may become more accurate, while another becomes less accurate or remains inaccurate. The prediction method therefore needs a way to combine errors across the state space.

The State Weighting Distribution

Definition

The state weighting distribution d(s) expresses which state errors matter more when prediction quality is evaluated. It gives each state a role in the overall objective, so the same set of approximate values can receive different evaluations when the state weights change.

State s1d(s1)State weightimportance for s1State s2d(s2)State weightimportance for s2State s3d(s3)State weightimportance for s3assigned by d(s)assigned by d(s)assigned by d(s)
How does d(s) assign different importance or frequency weights to states when evaluating prediction quality?
Worked example

Same errors, different priorities

Consider three states whose prediction errors are already known. Compare the evaluation when one state is given a larger weight.

  1. 1
    Identify the errors

    For each state, compare the approximate value with the true value. The resulting differences are the per-state prediction errors.

  2. 2
    Apply the state weights

    Each squared error is multiplied by that state's d(s). A larger d(s) makes that state's error contribute more strongly to the total objective.

  3. 3
    Change one priority

    If the weight for state s2 increases while the approximate and true values stay the same, the contribution from s2 increases relative to the contributions from the other states.

Result

Changing d(s) changes which errors matter most, even when the predictions and true values do not change.

Mean Squared Value Error

Definition

For a state s, the prediction error is the difference between the approximate value, written as ˆv(s, θ), and the true value under policy π, written as vπ(s). Mean Squared Value Error, or MSVE, is the weighted sum of the squared differences between these approximate and true state values.

Formula, step by step
  1. 1
    For each state s, find the difference ˆv(s, θ) − vπ(s).
  2. 2
    Square that difference so the state contributes through a squared error.
  3. 3
    Multiply the squared error by the state's weight d(s).
  4. 4
    Add the weighted squared errors over the state space.
Approximate and truevaluesValue differenceˆv(s, θ) − vπ(s)Squared error[ˆv(s, θ) − vπ(s)]²Weighted errord(s) times squared errorMSVEsum over statessubtractsquaremultiply by d(s)add over states
How are per-state value errors squared, weighted by d(s), and combined into one overall error measure?

MSVE turns many state-level comparisons into one objective. It does not ask only whether one state is correct. Instead, it combines every state's squared prediction error after applying the state weighting distribution. The arithmetic is less important than the structure: error, square, weight, and sum.

Reading the Objective

Worked example

Why a large error may not dominate

Suppose two states have different prediction errors. State s1 has a larger squared error, but state s2 has a larger d(s). What determines each state's contribution to MSVE?

  1. 1
    Evaluate state s1

    Square the difference between ˆv(s1, θ) and vπ(s1), then multiply that squared error by d(s1).

  2. 2
    Evaluate state s2

    Square the difference between ˆv(s2, θ) and vπ(s2), then multiply that squared error by d(s2).

  3. 3
    Compare contributions

    The larger contribution is determined by the product of the squared error and the state's weight, not by the error alone.

Result

A state affects MSVE through both its prediction accuracy and its state weight.

State s1squared errord(s1)lower weightd(s1)same weightState s2squared errord(s2)lower weightd(s2)higher weightweighted contributionbefore weighting changeafter weighting change
How does increasing the weight of one state change its contribution to the overall prediction error compared with other states?

If the approximate and true values remain fixed, changing d(s) can still change MSVE. Increasing the weight of one state makes that state's squared error more important in the combined measure. This is why the prediction objective must specify not only what counts as error, but also which state errors receive greater attention.

Common Reasoning Mistakes

  • Common mistake

    Assuming that improving one state leaves every other state unchanged.

    What it looks like

    Treating an update as if it changes only the state currently being considered.

    Why it fails

    Approximate state predictions are coupled, so changing the parameters can change several state estimates.

    Do this instead

    Track the possible effects of an update across multiple states.

  • Common mistake

    Evaluating approximation using only the largest individual error.

    What it looks like

    Choosing the state with the largest difference between approximate and true values and ignoring all other states.

    Why it fails

    MSVE combines weighted squared errors over the state space.

    Do this instead

    Consider every state's squared error together with its d(s) weight.

  • Common mistake

    Treating d(s) as irrelevant bookkeeping.

    What it looks like

    Assuming two states contribute equally even when their state weights differ.

    Why it fails

    The state weighting distribution expresses which state errors matter more.

    Do this instead

    Include d(s) when interpreting each state's contribution to the objective.

  • Common mistake

    Thinking that a lower error always means a lower contribution to MSVE.

    What it looks like

    Assuming the state with the smaller squared error must matter less.

    Why it fails

    A larger state weight can make a smaller squared error contribute more strongly.

    Do this instead

    Compare weighted squared errors rather than errors alone.

Practice

MEDIUM

Explain in your own words why a value function approximation method needs an explicit prediction objective. Then describe what happens to the objective if the approximate and true values stay fixed but d(s) is increased for one state.

Hints
  • Mention that approximate predictions can be coupled across states.
  • Use the structure of MSVE: difference, square, weight, and sum.
  • Focus on how the selected state contributes more strongly after its weight increases.

Key Takeaways

  1. 1
    Approximation can make exact prediction for every state impossible.
  2. 2
    Because state estimates are coupled, improving one estimate can change predictions and errors in other states.
  3. 3
    The state weighting distribution d(s) expresses which state errors matter more.
  4. 4
    MSVE is the sum over states of d(s) multiplied by the squared difference between ˆv(s, θ) and vπ(s).
  5. 5
    Changing state weights changes the priorities of the prediction objective even when the predictions themselves stay the same.

At a glance

  • Approximate value functions may not be able to match the true value at every state.
  • Coupled predictions mean that one update can affect errors in multiple states.
  • The distribution d(s) determines how strongly each state's error matters.
  • MSVE combines squared state errors after weighting each one by d(s).
  • A prediction objective makes approximation trade-offs explicit.

Related Topics

Related concepts

More concepts starting with T