Evolutionary Methods

Reinforcement learning methods can search for better behavior from different starting points. A policy gradient method searches through policies defined by numerical parameters. It uses interaction with the environment to estimate a direction for changing those parameters. An evolutionary method takes another route: it compares the complete behavior of several candidate policies and favors candidates that obtain more reward over their interaction period.

A policy gradient method searches over parameter-defined policies rather than treating a value estimate as the primary search object. The policy is described by a collection of numerical parameters. Those parameters determine the policy the agent uses, so changing the parameters changes the policy itself.

The diagram separates three ideas that can otherwise be confused. Policy gradient search changes the numerical parameters that define a policy. Value-function estimation focuses on how valuable a state or action is. Evolutionary search compares candidate policies by the behavior those policies produce over an interaction period. These ideas can appear together in reinforcement learning, but they are not identical.

Environment interaction matters because the method needs evidence about how the current policy behaves. The policy uses its current numerical parameters, the agent interacts with the environment, and the resulting behavioral interactions provide information for estimating a direction in which the parameters should move. The method then adjusts the parameters and searches again with another parameter-defined policy.

Value functions are not required in every method in this family. Some policy gradient methods use value-function estimates to improve their gradient estimates. In that situation, the value estimate is not the same thing as the policy-gradient estimate. It helps make the estimated direction for changing policy parameters more useful.

Evolutionary methods approach reinforcement learning by comparing complete behaviors rather than constructing a value estimate. Imagine a collection of agents that use different policies. Each agent interacts with the environment during its lifetime. After that interaction period, the agent is evaluated by the reward obtained from its complete behavior. Candidates with more reward are favored as the basis for further search, while weaker candidates are not favored for the next stage.

An evolutionary method can compare whole policies even when it never constructs a value function. Its important unit of evaluation is the agent's lifetime behavior. A value function instead estimates the value of a state or action. These approaches are different, although both can be used in reinforcement learning and some policy gradient methods can use value estimates to improve their gradient estimates.

Evolutionary methods may be effective when the policy space is small, when the policy space can be organized so that good policies are common or easy to find, or when substantial time is available for searching. These conditions make it more plausible that repeated comparison and selection will encounter useful policies.