Source-linked AI summary
Evolution Strategies as a Scalable Alternative to Reinforcement Learning
Tim Salimans, Jonathan Ho, Xi Chen, Szymon Sidor, Ilya Sutskever
TL;DR
The paper asks whether Evolution Strategies can provide a viable alternative to popular MDP-based reinforcement-learning methods for challenging control and game tasks. It evaluates ES on MuJoCo and Atari, using reparameterized policies and highly parallel evaluation, and finds competitive performance with strong scaling across workers. ES also offers tolerance of delayed rewards and long horizons without temporal discounting or value-function approximation, although the study leaves adaptive mutation-scale exploration for future work.
Problem
The paper examines whether black-box optimization can compete with popular MDP-based RL methods on challenging environments despite limited access to gradients.
Method
The paper trains neural-network policies with Evolution Strategies, using policy reparameterizations, common-random-number communication, mirrored sampling, fitness shaping, and distributed workers.
Results
ES is competitive on MuJoCo and most Atari environments, scales to over a thousand workers, and solves MuJoCo 3D humanoid walking in under 10 minutes with 1,440 workers.
Takeaways & Limitations
ES is a viable RL strategy when parallel computation can offset lower data efficiency, while avoiding temporal discounting and value-function approximation.
Takeaways & Limitations
The study fixes σ during training after finding no benefit from adapting it, and leaves indirect policy encodings for future work.
Abstract
from arXiv · showhide
We explore the use of Evolution Strategies (ES), a class of black box optimization algorithms, as an alternative to popular MDP-based RL techniques such as Q-learning and Policy Gradients. Experiments on MuJoCo and Atari show that ES is a viable solution strategy that scales extremely well with the number of CPUs available: By using a novel communication strategy based on common random numbers, our ES implementation only needs to communicate scalars, making it possible to scale to over a thousand parallel workers. This allows us to solve 3D humanoid walking in 10 minutes and obtain competitive results on most Atari games after one hour of training. In addition, we highlight several advantages of ES as a black box optimization technique: it is invariant to action frequency and delayed rewards, tolerant of extremely long horizons, and does not need temporal discounting or value function approximation.
1 Introduction
The paper studies Evolution Strategies as a scalable black-box alternative to MDP-based reinforcement learning, evaluating it on MuJoCo and Atari. It reports reliable training, strong parallel scaling, competitive performance, and robustness across environments.
- Evolution Strategies reliably trains neural-network policies for MuJoCo control and Atari from pixel inputs.
- Virtual batch normalization and other policy reparameterizations improve ES reliability, changing it from brittle to strong across varied environments.
- 1,440 workers solve the MuJoCo 3D humanoid task in under 10 minutes through ES’s highly parallelizable implementation.The reported communication strategy enables linear runtime speedups with more than a thousand workers.
- ES matches A3C’s final performance on most Atari environments using 3x–10x as much data, while reducing required computation by roughly 3x.Its one-hour results use about the same computation as published one-day A3C results, performing better on 23 games and worse on 28.
- On MuJoCo tasks, ES matches TRPO’s learned-policy performance with no more than 10x as much data and exhibits broader gait exploration on humanoid walking.The observed gaits include walking sideways and backwards, which were never observed with TRPO.
- ES combines reward-distribution indifference, no need for backpropagated gradients, and tolerance of potentially arbitrarily long horizons with scalability to many workers.The paper presents these properties as advantages over commonly used RL approaches.
2 Evolution Strategies
Evolution Strategies optimize policy parameters through population perturbations and episode returns, with methods designed for reliable training and large-scale parallelism. The approach uses shared randomness and related stabilization techniques to support efficient distributed optimization across RL environments.
- Core method: Evolution Strategies perturb policy parameters, evaluate complete episodes, and update parameters from the resulting returns.The method optimizes a distribution over policy parameters using stochastic gradient ascent and sampled evaluations.
- Scaling and parallelizing ES: Shared random seeds let workers reconstruct perturbations while communicating only scalar episode returns, sharply reducing distributed bandwidth.Workers know the perturbations used by others and can agree on parameter updates without exchanging full gradients.
- Stabilization and exploration: Antithetic sampling evaluates paired perturbations ϵ and −ϵ to reduce variance, while fitness shaping limits outlier influence and early local optima.Weight decay also keeps policy parameters from growing large relative to perturbations.
- Implementation boundaries: The implementation keeps σ fixed because adapting it during training showed no benefit, while indirect policy encodings remain future work.Full-length episodes can also lower CPU utilization when episode durations vary substantially, so episode caps are dynamically adjusted.
- Stabilization and exploration: ES explores through parameter perturbations rather than action sampling, and action discretization can encourage broader behaviors on some MuJoCo tasks.Gaussian perturbations on Atari convolutional policies sometimes produced nearly constant-action behaviors, motivating virtual batch normalization.
- Stabilization and exploration: Virtual batch normalization and policy reparameterizations greatly improve ES reliability across diverse environments.Without these methods, ES was brittle in the reported experiments.
3 Smoothing in parameter space versus smoothing in action space
The paper contrasts action-space and parameter-space smoothing for estimating policy gradients. ES smooths the objective through parameter perturbations, avoiding time-horizon-dependent variance growth and reducing distributed communication and computation costs.
- Policy-gradient optimization can be difficult because discrete actions, deterministic policies, and inaccessible transition dynamics make performance non-smooth and gradients unavailable to backpropagation.
- Policy gradients add noise in action space, whereas ES perturbs policy parameters with Gaussian noise, producing a smoothed objective.
- For policy gradients, estimator variance grows nearly linearly with episode length T, while ES’s corresponding score-function term is independent of T.
- ES is attractive for long episodes and long-lasting action effects because discounting and value-function approximation can bias policy-gradient estimates.
- ES can be interpreted as randomized finite-difference optimization, whose theoretical optimization-step requirement scales linearly with parameter dimension for general non-smooth problems.
- Black-box optimization communicates scalar returns and random seeds, reduces per-episode computation by about two thirds, and may substantially reduce memory use.
4 Experiments
Experiments compare ES with TRPO on MuJoCo, evaluate ES on Atari, and test its distributed scaling and temporal-resolution robustness. ES reaches competitive control performance, scales to 1,440 cores, and shows similar Pong learning across frame-skip settings.
- 4.1 MuJoCo: ES and TRPO were evaluated on OpenAI Gym continuous-control tasks simulated by MuJoCo, including balancing, hopping, and walking.
- 4.1 MuJoCo: ES reached TRPO’s final performance after 5 million environment timesteps, with less than a 10x sample-complexity penalty on Hopper and Walker2d.
- 4.2 Atari: ES was evaluated on 51 Atari 2600 games using the preprocessing and feedforward CNN architecture from prior A3C work.
- 4.3 Parallelization: ES uses low communication bandwidth for distributed execution, enabling testing on public cloud infrastructure without special networking setup.
- 4.3 Parallelization: 1,440 CPU cores reduced 3D Humanoid solving time to 10 minutes, with Figure 1 reporting linear speedup across CPU-core counts.
- 4.4 Invariance to temporal resolution: Pong learning curves were similarly fast for frame-skip values 1, 2, 3, and 4, with runs converging in about 100 weight updates.
5 Related work
The paper situates ES within prior neuro-evolution, direct policy-search, and black-box optimization work. Its stated contribution is demonstrating competitive performance alongside strong distributed scalability and efficiency.
- Earlier work applied ES-related methods and compressed neural-network policies to reinforcement learning and Atari, while natural evolution strategies also addressed black-box optimization.
- The paper’s main contribution is showing that ES can be highly scalable and efficient on distributed hardware.
- The authors report that carefully implemented ES is competitive with RL algorithms on hard problems, close in data efficiency, and faster in wall-clock training time.
6 Conclusion
The paper concludes that Evolution Strategies are a viable alternative to MDP-based reinforcement learning, with advantages for action frequency, delayed rewards, long horizons, and distributed scaling. Future work targets long-horizon problems, complex rewards, meta-learning, and low-precision implementations.
- ES is a viable alternative to Q-learning and policy-gradient reinforcement learning, with invariance to action frequency and delayed rewards.
- ES does not require temporal discounting or value function approximation and tolerates extremely long horizons.
- ES’s high parallelizability can offset decreased data efficiency by scaling computation across more workers.
- Future work will apply ES to long-horizon, complicated-reward problems and investigate meta-learning.
- The authors also plan to combine ES with fast low-precision neural-network implementations.