Source-linked AI summary
Evolved Policy Gradients
Rein Houthooft, Richard Y. Chen, Phillip Isola, Bradly C. Stadie, Filip Wolski, Jonathan Ho, Pieter Abbeel
TL;DR
RL agents often learn new tasks from scratch, so this paper asks whether a reusable learning algorithm can provide better progress signals than standard policy gradients. EPG evolves a differentiable, history-dependent loss whose inner-loop policy training is optimized for final performance. It learns faster on several randomized environments, generalizes to out-of-distribution test tasks, and can sometimes operate without environmental rewards at test time.
Problem
RL agents typically approach new tasks from scratch using external rewards, motivating a reusable learning algorithm that supplies a prior notion of progress.
Method
EPG evolves a differentiable loss over agent experience, using temporal convolutions and an outer evolution-strategy loop to optimize the final performance of inner-loop policy training.
Results
EPG trains agents faster than an off-the-shelf policy-gradient method and generalizes to out-of-distribution test-time tasks with qualitatively different behavior from other metalearning algorithms.
Takeaways & Limitations
The learned loss is adaptive and instructive, enabling quicker learning of new tasks and sometimes eliminating environmental rewards at test time.
Takeaways & Limitations
EPG is currently effective for one small family of tasks at a time and is unlikely to transfer to radically different tasks; its metalearning cycle also creates large computational demands.
Abstract
from arXiv · showhide
We propose a metalearning approach for learning gradient-based reinforcement learning (RL) algorithms. The idea is to evolve a differentiable loss function, such that an agent, which optimizes its policy to minimize this loss, will achieve high rewards. The loss is parametrized via temporal convolutions over the agent's experience. Because this loss is highly flexible in its ability to take into account the agent's history, it enables fast task learning. Empirical results show that our evolved policy gradient algorithm (EPG) achieves faster learning on several randomized environments compared to an off-the-shelf policy gradient method. We also demonstrate that EPG's learned loss can generalize to out-of-distribution test time tasks, and exhibits qualitatively different behavior from other popular metalearning algorithms.
1. Introduction
EPG metalearns a differentiable RL loss rather than a policy, aiming to give agents reusable progress signals for rapid learning across tasks. Its outer loop evolves the loss so inner-loop policy training maximizes final performance, with evidence of faster learning and out-of-distribution generalization.
- RL agents typically learn new tasks from scratch using external rewards, unlike humans who draw on prior experience to recognize progress.
- EPG encodes prior knowledge implicitly in a learned loss function, rather than explicitly in a learned behavioral policy.
- EPG uses inner-loop gradient descent for policy learning and an outer loop that adjusts loss parameters to maximize final returns.
- The loss parameters are optimized with evolution strategies because final returns are not explicit functions of those parameters.
- The loss architecture incorporates agent history through temporal convolutions, supporting adaptive learning signals across sequential experience.
- EPG trains agents faster than an off-the-shelf policy-gradient method and can generalize to out-of-distribution test-time tasks with qualitatively different behavior from other metalearning algorithms.
2. Notation and Background
This section formulates RL as an MDP and describes policy-gradient optimization through a surrogate loss. It also identifies fixed discounting and policy steps as limitations that motivate learning the loss itself.
- An MDP is defined by states, actions, transitions, rewards, an initial-state distribution, and a discount factor.
- An episode is a sequence of states, actions, and rewards, and its discounted return depends on the initial-state distribution, policy, and transition distribution.
- The RL objective is to find policy parameters that attain the highest expected episodic return.
- Policy-gradient methods optimize expected return by stepping in the direction of E[Rτ∇log π(τ)], which can be represented with a surrogate loss.
- The standard procedure is limited by its particular return discounting and fixed gradient step with respect to the policy.
3. Methodology
EPG learns a history-dependent loss that trains policies in an inner loop while evolution strategies optimize that loss in an outer loop. The objective is high expected return after training on sampled MDPs.
- EPG learns a loss Lφ built from temporal convolutions over recent agent history, with potential to internalize rewards and incentivize extended behaviors such as exploration.
- The evolved loss trains an on-policy policy πθ using stochastic gradient descent, while evolution strategies optimize its parameters φ.
- The inner loop minimizes Lφ for a policy on an MDP sampled from a task distribution p(M).
- The outer loop learns Lφ so the policy trained with it achieves high expected returns across the MDP distribution.
- Because trained-policy returns are not explicit functions of Lφ, gradient-based optimization cannot directly solve the outer objective.
2 Initialize buffer with N zero tuples
The inner loop trains a policy on sampled task experience using losses computed from recent sequential data, while the outer loop evolves the loss toward higher final returns.
- The outer loop evaluates perturbed loss functions on randomly sampled MDPs using final returns, applies an evolution-strategy update, and produces Lφ after E epochs.Relative-return ranking is used in the implementation to reduce sensitivity to tasks that consistently generate higher returns.
- Each worker samples an MDP, trains a policy for U experience steps, and updates the policy through SGD every M steps using recent minibatches.Workers return their final returns to the outer loop after inner-loop training.
- At test time, the learned loss trains a policy from scratch on a test MDP using the same schedule as inner-loop training.The loss can also receive rewards as inputs when environmental rewards cannot be fully inferred from observations.
- The loss uses an experience buffer of N recent tuples and computes losses for a batch of M sequential steps, with M < N.Transitions contain state, action, reward, and termination information.
- Temporal convolutions transform sequential policy, memory, and transition information into a context vector used by dense layers to output the loss.The memory unit is updated by gradient descent and can retain information beyond the finite experience buffer.
4. Experiments
Experiments evaluate EPG across randomized continuous-control environments, showing faster learning, higher returns, adaptive gradient behavior, exploration, and some transfer beyond metatraining conditions. EPG also generalizes to unseen goal directions, although transfer across policy architectures is inferior and longer-horizon generalization is limited.
- Performance: EPG is evaluated on randomized MuJoCo tasks requiring identification of randomized physics, morphology, reward directions, or target locations.The environments include RandomHopper, RandomWalker, RandomReacher, DirectionalHopper, DirectionalHalfCheetah, GoalAnt, and Fetch.
- Performance: EPG agents learn more quickly and obtain higher returns than PPO agents across the randomized environments tested.This advantage holds even when the learned loss does not observe rewards at test time.
- Analysis: EPG gradients are related to but different from PPO gradients, with Spearman correlation ρ = 0.5 across the tested environments.The learned loss therefore produces an optimization signal that is not equivalent to the PPO objective.
- Analysis: In DirectionalHopper, EPG explores both hopping directions and eventually selects the rewarded direction, while PPO agents often stagnate.Negative reward spikes indicate exploratory backward behavior before the agent learns the correct direction.
- Analysis: EPG adapts gradient magnitudes to the agent’s history, often increasing them as environmental information accumulates and occasionally producing strong spikes.In RandomHopper, learned-loss updates generally shift the policy distribution less per step but sometimes make sudden changes.
- Generalization: Simultaneously evolving policy initialization changes EPG’s learning profile, producing lower initial but higher final returns than PPO with evolved initialization.The comparison reflects a trade-off between rapid initial performance and later learning progress.
- Generalization: Transfer to policy architectures differing in layers, activations, or hidden units is inferior to transfer with the same train-time and test-time architecture.Training used a fixed two-layer policy with 64 tanh units on RandomWalker.
- Generalization: EPG reaches unseen negative-x GoalAnt targets and matches RL2’s final performance on positive-x targets after learning from random initialization.MAML with multiple updates can outperform RL2 but learns more slowly than EPG in experience timesteps.
5. Relation to Existing Literature
The paper situates EPG within metalearning, learned-loss, and reward-learning research, emphasizing its decoupling of the policy from the learning algorithm and its less restrictive optimization setup.
- Compared with concurrent learned-loss work for behavioral cloning, EPG needs no demonstrations, permits a long-horizon nondifferentiable outer objective, and supports inner loops lasting tens of thousands of timesteps.
- Unlike recurrent approaches such as RL2, EPG separates the policy from the learning algorithm instead of coupling both through recurrence.
- EPG is related to methods that learn reward functions and universal goal-directed value functions, whose critics can be interpreted as learned loss-like objects.
6. Discussion
The discussion presents EPG as an adaptive and sometimes instructive learned loss, while identifying limited task generality and substantial sequential computation as important boundaries.
- EPG learns a differentiable loss over thousands of sequential environmental actions that can adapt to tasks and sometimes operate without test-time environmental rewards.
- When test-time rewards are observable in DirectionalHopper, EPG improves learning speed, although those rewards cannot be completely internalized from observations alone.
- Without test-time environmental rewards, EPG can use its learned loss to internalize reward structures encountered during training, a step toward intrinsic motivation.
- EPG shows rudimentary generalization beyond methods such as RL2, but a loss trained for one small task family is unlikely to work effectively on a radically different task.
- EPG's sequential metalearning cycle creates large computational demands; a RandomReacher worker required approximately 196 million steps to evolve its loss.
A. Environment Description
The experiments use randomized continuous-control environments that vary physical parameters, reward functions, or target locations across tasks.
- RandomHopper and RandomWalker randomize gravity, friction, body mass, and link thickness during metatraining, with forward velocity as the reward.
- RandomReacher randomizes link lengths and uses negative distance as the metatraining reward, while exposing the target location at test time.
- DirectionalHopper and DirectionalHalfCheetah3 use randomized velocity reward functions.
- GoalAnt randomizes target location and initial rotation, provides velocity toward the target as reward, and does not expose the target location.
- Fetch randomizes target location and uses negative distance to the target as its reward, while providing the target location as input to EPG.
B. Additional Experiments
Additional experiments test whether EPG can continue learning without routine environment resets, including on GoalAnt where the agent retains its post-episode location and configuration.
- EPG can perform well in no-reset learning, continuing from the location and configuration after each episode instead of returning to a fixed start.
- On GoalAnt, the ant resets to its starting point only when the target is reached, and Figure 19 reports average return against epoch.
Training performance w.r.t. evolution epoch
EPG training shows stable metatraining performance across evolution epochs, while the learned loss uses both current and broader temporal experience. Test-time curves illustrate rapid behavioral adaptation through policy updates.
- Training performance w.r.t. evolution epoch: EPG metatraining performance is averaged across 256 inner-loop workers and multiple random seeds to assess the stability of evolving well-performing loss functions.The experiment uses 256 workers, 64 noise vectors, and 256 updates every 32 steps over an 8196-step inner loop.
- Loss input sensitivity: Figure 21 measures the gradient magnitude of L_t=25 with respect to observations, actions, termination signals, and policy outputs across buffer time points.The analysis is conducted in reward-free randomized environments including RandomHopper, RandomWalker, RandomReacher, and Fetch.
- Loss input sensitivity: The EPG loss is especially sensitive to inputs at its evaluation time step while also depending on the entire temporal context in the buffer.This supports the role of temporal convolutions in using the agent’s history and future experience to score behavior.
- Test-time training curves: Individual test-time curves track return against sampled trajectories and policy-update KL divergence across randomized environments.Figure 22 uses columns for different sampled environments, red curves for return, blue curves for KL divergence, and final-return labels averaged over the last three trajectories.
- Test-time training curves: In DirectionalHalfCheetah and GoalAnt, the agent first ramps up velocity before determining whether it is moving in the correct direction.For the hopper, increasing KL divergences visibly accompany rapid velocity ramp-up.
C. Experiment Hyperparameters
The section identifies the experiment hyperparameters and presents additional test-time training curves across sampled randomized environments. The curves jointly show return learning and policy-update divergence over training.
- C. Experiment Hyperparameters: Table 1 lists the EPG hyperparameters used for different environments.The surrounding text identifies these as the hyperparameters used in Section 4.
- C. Experiment Hyperparameters: Figure 22 organizes each sampled environment by column and plots return against trajectories alongside policy-update KL divergence.Final return values in the first row are averaged over the final three trajectories.