Watson's Question Answering and Game-Playing Architecture

Watson compared the legal bets available in the current situation. Each bet received an action value representing an estimate of the chance of winning from that situation, and Watson usually selected the bet with the highest action value.

Programming

A Wager Is More Than a Number

Finding a Daily-Double clue was only the beginning of Watson's decision. Watson also had to decide how much to wager. That choice depended on the entire game situation, not only on the clue. Instead of following one fixed betting rule, Watson treated wagering as a decision under uncertainty.

Key point

How does Watson move from the current game situation to possible Daily-Double bets and then select a wager under uncertainty?

Comparing Candidate Bets

An action value belongs to a possible bet in a particular game situation. It estimates the chance of winning that would result from making that bet from the current state. The value is therefore not a permanent rating of the bet itself. The same wager can have a different action value when the surrounding game situation changes.

Candidate	What Watson evaluates	Decision role
Legal bet	The possible game situation after making that bet	One candidate action
Another legal bet	A different possible game situation after making that bet	Another candidate action
Highest action value	The estimated winning chance associated with the candidate	Usually selected by Watson

Action values let Watson compare legal wagers in the current game situation.

How does each possible bet map to an action value, and why does Watson usually choose the wager with the highest value?

Two Estimates Behind One Value

The action-value comparison used two learned estimates. First, Watson evaluated the afterstate: the game situation that would result from making a particular bet. Second, it used in-category Daily-Double confidence, written as pDD. This estimate represented how likely Watson was to answer the unrevealed clue correctly in that category.

Definition

The state value function evaluates the broader game consequences of a possible bet. It represents Watson's estimated chance of winning from a game state or resulting afterstate.

Definition

In-category Daily-Double confidence, or pDD, estimates Watson's likelihood of answering the unrevealed Daily-Double clue correctly in its category. It was estimated from Watson's historical accuracy on clues in different categories over a large number of games.

What does the overall state value represent, what does in-category confidence represent, and how do they provide different information for wagering?

An Illustrative Bet Comparison

Worked example

Choosing Between Three Legal Bets

Suppose Watson has found a Daily-Double clue and three legal bets are available. How should the decision be analyzed?

1
Inspect the current situation
Watson begins with the current game state, including the situation that would lead to each possible wager being considered.
2
Evaluate each afterstate
For each candidate bet, Watson considers the resulting afterstate and uses the state value function to estimate the broader game consequences.
3
Estimate clue confidence
Watson also considers pDD, its estimated likelihood of answering the unrevealed clue correctly in that category.
4
Compare action values
The two estimates support an action value for each legal bet. Watson compares those action values rather than applying one fixed wager rule.
5
Select the leading candidate
The candidate with the highest action value is usually selected because its estimated chance of winning is highest among the compared legal bets.

Result

The wager is chosen by comparing predicted consequences and clue-answer confidence. The numerical values in this example are intentionally omitted because the source describes the decision process, not a particular game position.

How does a fixed betting rule differ from a strategy that considers the current board, confidence, possible outcomes, and expected rewards?

Learning Through Simulated Games

Reinforcement learning supplied Watson's state value function. Watson trained a multilayer neural network with nonlinear TD(lambda) on millions of simulated games against models of human players. During these games, temporal-difference errors were backpropagated to adjust the network's weights.

The training connected a representation of a game state with an estimate of Watson's chance of winning from that state. The learned function could then contribute to the evaluation of a possible Daily-Double wager by assessing the afterstate produced by that wager.

How do simulated game states, wagers, outcomes, and rewards flow through repeated trials to produce a learned wagering strategy?

Features Designed for Jeopardy!

The neural network did not learn from an unspecified game representation. It received feature vectors designed for Jeopardy!. These features represented the game state used by the wagering model, while the question-answering system supplied information relevant to Watson's expected success on the unrevealed clue.

Watson-specific information therefore entered the decision in two related ways. The learned state value function represented the broader consequences of the game situation, and pDD represented Watson's category-specific likelihood of answering the clue correctly. Together, these estimates supported the action value for each possible legal wager.

How do Watson's game and question-answering features enter the decision process and influence the predicted value of each possible wager?

Mistakes About Watson's Wager

Common mistake
Thinking that finding the Daily-Double clue determines the whole decision.
What it looks like
Watson finds the clue and is treated as though the wager is automatic.
Why it fails
Finding the clue was only the beginning. Watson still had to choose how much to wager.
Do this instead
Separate clue identification from wager selection. The wager depends on the entire game situation.
Common mistake
Treating pDD as the complete value of a bet.
What it looks like
A high pDD is assumed to mean that the corresponding wager must be best.
Why it fails
pDD estimates the chance of answering the unrevealed clue correctly in its category. It does not by itself evaluate the broader game consequences.
Do this instead
Use pDD as one learned estimate and keep it separate from the state value function.
Common mistake
Treating the state value function as a clue-answer confidence score.
What it looks like
The state value is interpreted as Watson's confidence that it knows the clue answer.
Why it fails
The state value function evaluates the broader game consequences of a state or afterstate.
Do this instead
Interpret the state value as an estimate of winning from the game situation, not as category-specific clue confidence.
Common mistake
Assuming Watson used one fixed wager rule.
What it looks like
Every Daily-Double situation is expected to produce the same kind of bet.
Why it fails
Watson treated wagering as a decision under uncertainty and compared action values for legal bets.
Do this instead
Analyze the current situation, the resulting afterstates, and pDD before comparing candidate wagers.

Check Your Reasoning

MEDIUM

Explain why an action value cannot be understood as only Watson's confidence in answering the Daily-Double clue. In your answer, identify the separate role of the state value function and the separate role of pDD.

Hints

Start with what the state value function evaluates.
Then describe what pDD estimates.
Finish by explaining why both estimates contribute to comparing legal bets.

Your attempt (not saved)

MEDIUM

Trace the learning pipeline in order: simulated game state, wager, later outcome, temporal-difference error, neural-network update, and learned state value. Then explain how that learned value can later help evaluate a possible Daily-Double wager.