Policies and Sequential Decision Problems
The policy and value roles should not be confused. The policy is associated with choosing behavior, whereas the value function is associated with estimating or evaluating situations or decisions.
From Situations to Decisions
A reinforcement-learning agent must often respond to many possible situations. Storing a separate answer for every situation can be impractical, so the agent can use a function that receives information about a situation and produces an estimate or decision. An artificial neural network provides one way to represent such a function.
The loop shows why the problem is sequential. The agent receives information about a current situation, produces a decision, and then receives reinforcement-learning information connected to what happened next. A neural network can help produce the decision or estimate the usefulness of the situation and decision.
Neural Networks as Function Approximators
An artificial neural network is a function representation that can be used within reinforcement learning. Information describing a situation enters the network, passes through one or more layers, and produces an output.
The network does not need to store a separate answer for every possible situation. Instead, its connected computational units and weights represent a function that maps situation information to an output. Learning changes the weights so that the output becomes more useful for the task.
Replacing a Separate Answer Table with an Approximation
An agent encounters many possible situations in a sequential decision problem. How can a neural network help it respond without storing a separate answer for every situation?
- 1Receive information
Information describing the current situation enters the artificial neural network.
- 2Process information
The information passes through the network's connected computational units and layers.
- 3Produce an output
The network produces either a decision rule output or an estimate associated with the situation.
- 4Improve the representation
Learning changes the network's weights so that later outputs become more useful for the reinforcement-learning task.
The network serves as a function representation: it maps situation information to an approximate decision or estimate rather than requiring a separately stored answer for every situation.
Single-Layer and Multi-Layer Roles
The important structural distinction is whether the network has only one layer of neurons or several. A single-layer artificial neural network is the simpler arrangement. A multi-layer network adds intermediate layers, whose role can include learning a suitable representation before the final output is produced.
| Structure | Role described in the source | Reinforcement-learning significance |
|---|---|---|
| Single-layer network | A simpler arrangement of neurons | Provides a basic network structure for representing a function |
| Multi-layer network | Adds intermediate layers that can learn a suitable representation | Increases the ability to learn complex representations and nonlinear policies |
The central difference is the presence of intermediate layers and the representational role they can provide.
The historical progression moved from the abstract model neuron proposed by McCulloch and Pitts in 1943, through single-layer systems such as the Perceptron and ADALINE, to multi-layer networks trained with error backpropagation. The significance of adding layers is not merely that the network becomes larger. Intermediate layers make it possible to learn more complex representations and nonlinear control policies.
Forward Output and Backward Error
Training a multi-layer neural network involves comparing what the network produces with the output required for training, measuring the error, and using that error to adjust the network. Error backpropagation is the method associated with this process.
The defining direction is two-part. Information first travels forward through the layers to produce an output. Information about the error then travels backward through the network to guide learning. The purpose of this backward pass is to adjust the network's weights, including weights associated with intermediate layers, so that future outputs become more useful.
Tracing One Training Adjustment
A multi-layer network produces an output that differs from the output required for training. What happens next?
- 1Forward pass
Situation information moves through the network and produces an output.
- 2Error measurement
The produced output is compared with the output required for training, and the difference provides error information.
- 3Backward pass
The error information is propagated backward through the network.
- 4Weight adjustment
The backward information guides changes to the network's weights.
The next output can become more useful because the network has used error information to adjust its internal parameters.
Policies and Value Functions
In a sequential decision problem, a neural network can approximate more than one kind of function. A policy is a mapping used to choose behavior. A value function provides an estimate associated with situations or decisions. Both can be represented by artificial neural networks, while reinforcement-learning information is used to improve the approximations.
The distinction is about the network's role in the agent's interaction with the task. A policy approximation is connected to behavior: it helps determine what the agent does. A value-function approximation is connected to evaluation: it estimates something associated with situations or decisions. These roles can be represented separately or used together.
The source discusses actor-critic systems to illustrate the two roles. In reported pole-balancing work, both an actor and a critic were represented by artificial neural networks. The actor concerned behavior, while the critic concerned evaluation. The example shows that neural networks can participate in more than one part of a reinforcement-learning method.
A Complete Decision Trace
Consider a generated abstract example. At one point in a sequential task, the agent receives information describing its current situation. A policy-approximating network processes that information and produces a behavior choice. The environment then supplies reinforcement-learning information connected to the result, including a reward and a next situation. A value-approximating network can provide an estimate associated with the situation or decision. The process repeats as the agent encounters the next situation.
The policy and value roles should not be confused. The policy is associated with choosing behavior, whereas the value function is associated with estimating or evaluating situations or decisions.
What do you think happens?
An artificial neural network receives information about a situation and produces an estimate rather than an action choice. Which role does this output most directly represent?
Reveal answer
A value-function approximation
The source distinguishes a policy as a mapping used to choose behavior from a value function as an estimate associated with situations or decisions. Backpropagation is the training process used to adjust a multi-layer network; it is not the output role.
Mistakes About Network Roles
Common mistake Treating a neural network as a fixed lookup table with one separately stored answer for every situation.
What it looks likeAssuming that the agent must store an individual answer for each possible situation.
Why it failsThe source presents the network as a function representation that receives situation information and produces an estimate or decision.
Do this insteadDescribe the network as an approximation of a function whose weights are changed through learning.
Common mistake Assuming that adding layers only increases the number of computational units without changing representation ability.
What it looks likeDescribing a multi-layer network as merely a larger single-layer network.
Why it failsIntermediate layers can learn a suitable representation, and the move to multi-layer networks increased the ability to learn complex representations and nonlinear policies.
Do this insteadExplain the representational role of the intermediate layers.
Common mistake Calling the forward production of an output backpropagation.
What it looks likeSaying that backpropagation is the process by which situation information travels to the output.
Why it failsThe source distinguishes the forward movement of information from the backward propagation of error information.
Do this insteadUse backpropagation for the backward error-guided training process that adjusts weights.
Common mistake Using policy and value function as interchangeable terms.
What it looks likeCalling an estimate associated with a situation an action choice.
Why it failsA policy is used to choose behavior, while a value function provides an estimate associated with situations or decisions.
Do this insteadIdentify whether the network output directs behavior or evaluates a situation or decision.
Check Your Understanding
Explain, in your own words, why a neural network can be useful when a reinforcement-learning agent must respond to many possible situations. Then distinguish the roles of a policy approximation, a value-function approximation, and error backpropagation.
Hints
- Start with the idea of representing a function instead of storing a separate answer for every situation.
- For the policy, focus on behavior and action choice.
- For the value function, focus on estimation or evaluation.
- For backpropagation, focus on error information traveling backward to guide weight adjustment.
Create a short trace for a sequential decision problem using these terms in order: current situation, network output, action or estimate, error information, weight adjustment, next situation.
Hints
- A policy output is connected to behavior.
- A value output is connected to evaluation.
- Error information is used during training rather than being the policy itself.
Key Takeaways
- 1Artificial neural networks can represent functions that map situation information to decisions or estimates in reinforcement learning.
- 2Single-layer networks are simpler arrangements, while multi-layer networks add intermediate layers that can learn more complex representations and nonlinear policies.
- 3Backpropagation sends error information backward through a multi-layer network to guide changes to its weights.
- 4A policy approximation is connected to choosing behavior, while a value-function approximation is connected to estimating situations or decisions.
- 5Actor-critic systems illustrate how separate neural-network roles can support behavior and evaluation within one reinforcement-learning method.
At a glance
- Neural networks provide function representations for handling many possible situations in reinforcement learning.
- Adding intermediate layers enables learning of more complex representations and nonlinear control policies.
- Backpropagation uses error information to adjust the weights of a multi-layer network.
- Networks can approximate policies that guide behavior and value functions that evaluate situations or decisions.