AlphaGo's Policy and Value Networks
AlphaGo evaluated game states with both a value network and rollouts.
Two Views of a Go Position
AlphaGo evaluated a Go game state in two ways rather than relying on a single estimate. One method used a value network. The other used rollouts. The central design question was how much influence each method should have in the final evaluation.
The value network evaluated the high-performance policy pρ. Rollouts used the weaker but faster policy pπ. These methods therefore contributed different strengths: the value network represented a stronger policy, while rollouts could use a faster policy to add precision for particular game states.
Following One Position Through the System
A Hypothetical Position
Suppose AlphaGo is evaluating one Go position and must combine the value-network evaluation with the rollout evaluation.
- 1Start with the position
The same Go game state is presented to both evaluation methods.
- 2Use the value network
The value network evaluates the high-performance policy pρ.
- 3Use rollouts
Rollouts evaluate the position using the weaker but faster policy pπ.
- 4Apply λ
λ determines how much the final evaluation relies on the rollout result and how much it relies on the value-network result.
- 5Produce one estimate
The two contributions are combined into the evaluation used for that game state.
The position is judged using both methods, with λ controlling their relative influence.
This example is a mechanism trace, not a numerical calculation. The source describes the roles of the two methods and λ, but it does not provide particular evaluation scores for a specific board position. The important process is that one state receives two evaluations before they are combined.
λ as the Mixing Control
λ is a mixing control, not a third evaluation method. It determines how much the final game-state evaluation relies on the value network and how much it relies on rollouts. Changing λ changes the balance between the two existing evaluations.
| λ value | Evaluation used | Meaning |
|---|---|---|
| 0 | Value network only | Rollouts contribute nothing |
| 0.5 | Combined value-network and rollout evaluation | Both methods influence the final evaluation |
| 1 | Rollout only | The evaluation relies only on rollouts |
The endpoint meanings come directly from the way λ controls the mixture. The λ = 0.5 row describes the reported best setting as a combined evaluation, not as a claim that the two numerical outputs are necessarily identical.
What the Three Key Settings Mean
What do you think happens?
Which evaluation contributes when λ = 0, and which contributes when λ = 1?
Reveal answer
λ = 0 uses the value network only, and λ = 1 uses rollouts only.
At λ = 0, rollouts contribute nothing. At λ = 1, the evaluation relies only on rollouts.
At λ = 0, AlphaGo uses value-network-only evaluation. This removes the rollout contribution. At λ = 1, AlphaGo uses rollout-only evaluation. This removes the value-network contribution. At λ = 0.5, both methods contribute to the final evaluation, representing a balanced combination of the two methods.
Why the Combination Was Stronger
The value network and rollouts were useful because they were imperfectly matched in a productive way. The value network evaluated the high-performance policy pρ, but that policy was too slow for direct live play. Rollouts used the weaker but faster policy pπ, allowing them to add precision for particular states. Combining their outputs gave AlphaGo more than either isolated evaluation method.
The reported comparison supports this conclusion. AlphaGo using only the value network played better than AlphaGo using only rollouts. However, the best play came from λ = 0.5 rather than from either method alone. The source also reports that the value-network-only version played better than the strongest of the other Go programs mentioned in the evaluation.
Mistakes in Reading λ
Common mistake Reversing the endpoint meanings
What it looks likeTreating λ = 0 as rollout-only and λ = 1 as value-network-only
Why it failsThe source defines λ = 0 as value-network-only evaluation and λ = 1 as rollout-only evaluation.
Do this insteadRemember that rollouts contribute nothing at λ = 0, while λ = 1 relies only on rollouts.
Common mistake Calling λ a third evaluator
What it looks likeDescribing λ as another network that judges the Go position
Why it failsλ is a mixing control. It determines the relative influence of the value-network and rollout evaluations.
Do this insteadName the two evaluators first, then describe λ as the control that combines their outputs.
Common mistake Assuming one method was always sufficient
What it looks likeConcluding that the value network should replace rollouts because value-network-only play was stronger than rollout-only play
Why it failsThe best reported play came from λ = 0.5, which combined both methods.
Do this insteadDistinguish between the stronger endpoint and the best combined setting.
Common mistake Confusing pρ and pπ
What it looks likeSaying that rollouts used the high-performance policy pρ
Why it failsThe value network evaluated pρ, while rollouts used the weaker but faster policy pπ.
Do this insteadAssociate pρ with the value network and pπ with rollouts.
Check Your Understanding
Explain, in your own words, what changes when λ moves from 0 to 1. Then explain why λ = 0.5 can be stronger than either endpoint even though the value network alone performed better than rollouts alone.
Hints
- Start by identifying which method is used at each endpoint.
- Then compare the roles of pρ and pπ.
- Finally, connect the reported λ = 0.5 result to the complementary strengths of the two methods.
A Complete Explanation
Give a concise explanation of why AlphaGo used both a value network and rollouts.
- 1Identify the value network
It evaluated the high-performance policy pρ.
- 2Identify rollouts
They used the weaker but faster policy pπ.
- 3Identify λ
λ controlled how much each evaluation contributed.
- 4Interpret the result
The reported best play at λ = 0.5 showed that combining the methods was better than relying on either one alone.
AlphaGo gained strength by combining a high-performance value-network evaluation with a faster rollout evaluation instead of selecting only one.
Key Takeaways
- 1AlphaGo evaluated a Go state with both a value network and rollouts.
- 2The value network evaluated the high-performance policy pρ, while rollouts used the weaker but faster policy pπ.
- 3λ controlled the mixture: λ = 0 meant value-network-only evaluation, and λ = 1 meant rollout-only evaluation.
- 4λ = 0.5 combined both evaluation methods and produced the best reported play.
- 5The combination worked because the two imperfect methods contributed complementary strengths.
At a glance
- AlphaGo used a value network and rollouts to evaluate the same Go game state.
- The value network evaluated pρ, the high-performance policy; rollouts used pπ, the weaker but faster policy.
- λ controlled the balance between the two evaluations.
- λ = 0 used only the value network, λ = 1 used only rollouts, and λ = 0.5 combined them.
- The reported best play at λ = 0.5 demonstrated the strength of combining complementary evaluation methods.