Episodic Semi-gradient Control Methods
In this topic, the curves concern semi-gradient Sarsa combined with tile-coding function approximation in the Mountain Car example.
The Question Behind the Curves
A learning algorithm is not judged only by one final result. Its performance can change as learning proceeds. A learning curve represents that performance over time, allowing you to examine how the plotted outcome changes across the period shown. In this topic, the curves concern semi-gradient Sarsa combined with tile-coding function approximation in the Mountain Car example.
The central reading task is not to memorize a claimed winner. It is to identify what was varied, identify what was measured, and then describe only the behavior displayed by the curves.
Following Performance Over Time
A learning curve gives a time-oriented view of algorithm performance. To read one, begin with the labels on the plotted axes and any legend. Then follow a curve across the represented period. The curve's changing position tells you how the displayed performance changes over time. This reading is different from stating an unsupported numerical result: the supplied description identifies the presence of several curves and different step sizes, but it does not provide exact numerical values or a definite ranking.
Treat the axis labels and legend as part of the evidence. First determine what the plotted quantity represents and what the horizontal progression represents. Only then describe whether the displayed performance changes, differs between curves, or remains difficult to distinguish.
The Mountain Car Setting
The reported experiment combines three elements: semi-gradient Sarsa, tile-coding function approximation, and the Mountain Car example. The Mountain Car example supplies the concrete problem in which learning performance is examined. Tile-coding function approximation is the representation method used with semi-gradient Sarsa in that setting. The source pack does not supply the exact tile layout, the numerical parameters, or the internal value assignments, so those details should not be inferred from the figure.
What Figure 10.2 Compares
Identifying the Comparison
Describe what Figure 10.2 compares without claiming an unsupported numerical result.
- 1Identify the method
The curves concern semi-gradient Sarsa used with tile-coding function approximation.
- 2Identify the problem
The method is examined on the Mountain Car example.
- 3Identify the varied setting
Figure 10.2 shows several curves associated with various step sizes.
- 4Identify the measured evidence
The curves show performance over the period represented in the figure.
- 5Limit the conclusion
Describe the visible differences between the curves only after checking their labels and plotted behavior. Do not name a best step size unless the figure itself clearly supports that claim.
Figure 10.2 compares performance curves for semi-gradient Sarsa with tile-coding function approximation on Mountain Car under different step-size settings.
Why Step Sizes Produce Multiple Curves
A step size is one of the settings being varied in the reported comparison. Because Figure 10.2 considers various step sizes, the figure uses separate curves to let the reader compare the performance associated with those settings over the represented period. The important question is not whether multiple curves exist in isolation, but how the plotted performance differs when the step-size setting differs.
Before comparing curve behavior, match every curve to its step-size label. A statement about one curve cannot be transferred to another curve unless the labels show that they represent the same setting.
Mistakes in Figure Interpretation
Common mistake Claiming a definite best step size without evidence from the plotted curves.
What it looks likeStep size B is definitely the best choice.
Why it failsThe supplied material states that Figure 10.2 contains curves for various step sizes, but it does not state a particular ranking.
Do this insteadDescribe the relative curve behavior only when the labels and plotted evidence support it.
Common mistake Treating the curve as a result for semi-gradient Sarsa alone.
What it looks likeThis curve shows how Sarsa performs independently of the representation.
Why it failsThe reported setting combines semi-gradient Sarsa with tile-coding function approximation on Mountain Car.
Do this insteadAttribute the displayed result to the combined experimental setting.
Common mistake Inventing exact numerical values that are not supplied.
What it looks likeThe algorithm reaches a performance value of 20 at a particular point.
Why it failsThe source identifies the comparison but does not provide exact numerical values.
Do this insteadUse qualitative descriptions tied to the visible axes, labels, and curve patterns.
Common mistake Describing tile coding in more detail than the evidence allows.
What it looks likeThe figure proves which individual tiles are active and what numerical weights they contain.
Why it failsThe supplied material identifies tile-coding function approximation as part of the setting but does not provide an internal tile map or weight values.
Do this insteadState its role as the function-approximation method used in the Mountain Car experiment.
Evidence-Based Reading Practice
Suppose you are shown Figure 10.2 and asked to explain one visible difference between two curves. Write a four-part answer: name the method and problem setting, identify the two step-size labels, describe the displayed performance difference over the represented period, and state one conclusion you cannot make without additional evidence.
Hints
- Include semi-gradient Sarsa, tile-coding function approximation, and the Mountain Car example.
- Use the legend to identify which curve belongs to which step size.
- Describe only the behavior visible in the plotted curves.
- Do not invent exact values or a ranking that the figure does not establish.
A Careful Written Interpretation
Construct a defensible interpretation when the figure shows several step-size curves but the source description does not supply their numerical values.
- 1State the setting
Say that the curves concern semi-gradient Sarsa with tile-coding function approximation on the Mountain Car example.
- 2State the comparison
Say that various step sizes are represented by separate curves.
- 3Report the observation
Refer to the curve labels and describe the differences visible over the period represented in the figure.
- 4State the limit
Do not claim exact values or a universal best setting unless those claims are directly supported by the figure.
The figure compares how semi-gradient Sarsa with tile-coding function approximation performs on Mountain Car under various step-size settings. The defensible conclusion is whatever the labeled curves visibly show; unsupported numerical rankings should be omitted.
Key Takeaways
- 1A learning curve represents algorithm performance over time.
- 2Figure 10.2 concerns semi-gradient Sarsa with tile-coding function approximation on the Mountain Car example.
- 3Different step sizes are represented by different curves so their displayed performance can be compared.
- 4The curve labels, axis labels, and plotted patterns are the evidence for interpretation.
- 5Exact numerical results and a particular curve ranking should not be claimed unless the figure supports them.
At a glance
- A learning curve shows how an algorithm's displayed performance changes over time.
- Figure 10.2 compares several step-size settings for semi-gradient Sarsa with tile-coding function approximation on Mountain Car.
- Tile coding is part of the reported function-approximation setting, while the supplied material does not specify its internal tile values.
- Reliable interpretation depends on matching curves to labels and describing only supported behavior.
- Do not infer exact numerical values or a definite ranking that the plotted evidence does not establish.