Source-linked AI summary
Benchmarking Deep Reinforcement Learning for Continuous Control
Yan Duan, Xi Chen, Rein Houthooft, John Schulman, Pieter Abbeel
TL;DR
Continuous-control research lacked a standardized, challenging benchmark, making progress difficult to quantify. The paper introduces a varied task suite and systematically evaluates reinforcement-learning algorithms, finding TNPG, TRPO, and DDPG effective while hierarchical tasks remain difficult.
Problem
The lack of a standardized, challenging testbed makes scientific progress in reinforcement learning and continuous control difficult to quantify.
Method
The paper provides a variety of challenging continuous-control tasks, implements several reinforcement-learning algorithms, and releases the benchmark and reference implementations openly.
Results
Among the implemented algorithms, TNPG, TRPO, and DDPG are effective for training deep neural network policies, while hierarchical tasks remain difficult.
Takeaways & Limitations
The open-source benchmark encourages reproducible evaluation and comparison of existing and newly proposed algorithms on challenging continuous-control tasks.
Takeaways & Limitations
Implemented algorithms achieve poor performance on hierarchical tasks even after extensive hyperparameter search and 500 training iterations.
Abstract
from arXiv · showhide
Recently, researchers have made significant progress combining the advances in deep learning for learning feature representations with reinforcement learning. Some notable examples include training agents to play Atari games based on raw pixel data and to acquire advanced manipulation skills using raw sensory inputs. However, it has been difficult to quantify progress in the domain of continuous control due to the lack of a commonly adopted benchmark. In this work, we present a benchmark suite of continuous control tasks, including classic tasks like cart-pole swing-up, tasks with very high state and action dimensionality such as 3D humanoid locomotion, tasks with partial observations, and tasks with hierarchical structure. We report novel findings based on the systematic evaluation of a range of implemented reinforcement learning algorithms. Both the benchmark and reference implementations are released at https://github.com/rllab/rllab in order to facilitate experimental reproducibility and to encourage adoption by other researchers.
1. Introduction
Recent deep reinforcement learning combines learned feature representations with reinforcement learning, but continuous control lacks a standardized, challenging benchmark for measuring progress. The paper addresses this gap with a 31-task benchmark, systematic algorithm evaluation, and open-source reference implementations.
- Continuous-control algorithms do not generalize straightforwardly from discrete-action benchmarks, while naive action discretization becomes infeasible in high dimensions.This creates a gap in understanding continuous-action reinforcement learning.
- A standardized and challenging continuous-control testbed is needed to quantify scientific progress and compare algorithm strengths and limitations.
- The benchmark contains 31 tasks spanning cart-pole balancing, high-degree-of-freedom locomotion, partial observations, and hierarchical structure.The suite ranges from simple tasks to challenging settings with high state and action dimensionality.
- The authors systematically evaluate implemented reinforcement learning algorithms for training deep neural network policies.
- The benchmark and reference implementations are released openly to support development, implementation, and evaluation of new algorithms and tasks.
2. Preliminaries
The benchmark formalizes tasks as finite-horizon discounted Markov decision processes, extending them to partially observable settings when observations do not reveal the full state. Algorithms optimize stochastic or deterministic policies over trajectories generated by the environment.
- The task interface is a finite-horizon discounted Markov decision process defined by states, actions, transitions, rewards, initial-state distribution, discount factor, and horizon.
- Partially observable tasks additionally specify an observation set and an observation probability distribution.
- Most implemented algorithms optimize a stochastic policy parameterized by θ.
- A trajectory consists of states and actions sampled from the initial-state distribution, policy, and transition dynamics.
- Deterministic policies map states directly to actions through a policy µθ.
3. Tasks
The benchmark spans basic control, locomotion, partial observability, and hierarchical tasks, using physics simulators to provide varied and increasingly realistic continuous-control challenges. These tasks test state inference, model generalization, motor control, and high-level decision-making.
- The benchmark divides tasks into basic, locomotion, partially observable, and hierarchical categories.Detailed specifications are provided in supplementary materials and source code.
- Physics simulators are used instead of symbolic equations, with Box2D for simple dynamics and MuJoCo for more complicated contact-rich dynamics.
- The five basic tasks provide relatively low-dimensional, quick evaluations, including balancing, swing-up, Mountain Car, Acrobot, and double inverted pendulum control.
- Locomotion tasks: Seven locomotion tasks vary in dynamics and difficulty, from Swimmer and Hopper to Ant, Simple Humanoid, and Full Humanoid.
- Partially observable tasks: Fifteen partially observable variants require agents to infer hidden state from limited, noisy, delayed, or model-varying observations and action histories.
- Hierarchical tasks: Hierarchical tasks combine low-level locomotion with higher-level objectives such as food collection or maze navigation.The tasks use different time scales for motor control and high-level decisions.
4. Algorithms
The benchmark evaluates gradient-based policy search, gradient-free optimization, and online actor-critic methods under general policy parameterizations. The algorithms differ in how they estimate policy updates, constrain distribution changes, explore parameters, and use replay data.
- The benchmark implements gradient-based policy search methods alongside two gradient-free methods for comparison.
- Batch algorithms: Batch algorithms collect trajectory data and use either current-policy or perturbed-policy samples, depending on the method.
- Batch algorithms: REINFORCE estimates the expected-return gradient with a likelihood-ratio method and a state-dependent baseline to reduce variance.
- Batch algorithms: TNPG computes a natural-gradient direction that approximately limits policy-distribution change without explicitly forming the Fisher-matrix inverse.Conjugate gradients make the computation practical for large neural-network policies.
- Batch algorithms: RWR uses expectation-maximization to optimize a lower bound on log-expected return, transforming returns to nonnegative weights.
- Batch algorithms: REPS limits information loss per iteration by solving for dual parameters and controlling the policy step with δKL.Its sample Bellman error enters the dual formulation.
- Batch algorithms: TRPO constrains policy updates through a surrogate-loss optimization with a KL step-size parameter and line search.
- Gradient-free methods: CEM explores directly in policy-parameter space by retaining perturbations from the top q-quantile of sampled returns, while CMA-ES adapts covariance along evolution paths.
5. Experiment Setup
The experiments define performance from undiscounted trajectory returns and tune hyperparameters using repeated trials across selected tasks. Policies and variance-reduction baselines are specified across task categories and algorithms.
- Performance Metrics: Performance is defined for each algorithm-task report unit using undiscounted returns averaged across training iterations and collected trajectories.The number of iterations, trajectories per iteration, and trajectory returns determine the metric.
- Hyperparameter Tuning: DDPG uses hyperparameters from Lillicrap et al. (2015), while other algorithms undergo category-based grid searches on two tasks.Each hyperparameter choice is evaluated under five random seeds.
- Hyperparameter Tuning: The best hyperparameters maximize mean(returns) −std(returns), selecting against performance fluctuations caused by overly large step sizes.This criterion balances average performance with variability across runs.
- Hyperparameter Tuning: On remaining tasks, the two best category-level hyperparameter settings are tested and the better performance is reported.This evaluates both maximum performance under tuning and robustness across tasks.
- Policy Representation: Most policy classes use a feedforward neural network with hidden layers of 100, 50, and 25 units, while partially observable tasks use recurrent policies.The feedforward networks use tanh nonlinearities in the first two hidden layers and map states to Gaussian means.
- Policy Representation: DDPG uses two-layer feedforward networks with 400 and 300 units and ReLU activations for both its policy and Q function.The policy and Q function share the described architecture.
- Baseline: Gradient-based algorithms except REPS use a linear baseline with time-varying features to reduce variance in empirical returns.The baseline is subtracted from empirical returns during optimization.
6. Results and Discussion
The systematic evaluation reveals distinct strengths and limitations across policy-search algorithms, with stability, sample efficiency, observability, and task structure shaping performance. No implemented algorithm performs well on hierarchical tasks, even after extensive tuning.
- Policy-gradient methods: REINFORCE is effective on most basic and locomotion tasks, but can prematurely converge to local optima, producing unstable jumping behavior on Walker.The observed performance gap between REINFORCE and TNPG on Walker is associated with large policy-distribution changes from individual steps.
- Policy-gradient methods: Both TNPG and TRPO outperform other batch algorithms on most tasks, consistent with more stable learning from constrained policy-distribution changes.TRPO additionally controls each update through a line search that ensures improvement in the surrogate loss.
- Other methods: RWR solves some basic tasks but fails on more challenging locomotion tasks, showing fast initial improvement followed by significant slowdown.REPS is especially prone to early convergence to local optima for continuous states and actions, with outcomes strongly affected by the initial policy.
- Gradient-free methods: CEM performs well on some basic tasks despite thousands of policy parameters, whereas performance declines as dynamics become more complicated and CMA-ES can run out of memory.CEM outperforms CMA-ES; on Full Humanoid, CMA-ES fails to produce results because of high-dimensional observations.
- Policy-gradient methods: DDPG converges significantly faster than batch algorithms on some tasks such as Half-Cheetah, but is less stable and can degrade during training.Rescaling rewards by 0.1 seems to improve DDPG stability, while the method remains more susceptible to reward scaling.
- Partially observable tasks: Recurrent policies find better solutions than feedforward policies in partially observable tasks, but are more difficult to train.CEM and CMA-ES perform considerably worse with recurrent policies, while the performance gap between REINFORCE and TNPG widens.
- Hierarchical tasks: All implemented algorithms achieve poor performance on hierarchical tasks despite extensive hyperparameter search and 500 training iterations.The results motivate algorithms that can automatically discover and exploit hierarchical structure.
7. Related Work
Earlier reinforcement-learning benchmarks range from individual low-dimensional control problems to high-dimensional robots and multi-agent tasks. This benchmark instead emphasizes simulated environments, reproducibility, and a larger collection spanning varied difficulty.
- Earlier benchmarks commonly incorporated tasks such as inverted pendulum, mountain car, and Acrobot into broader evaluation suites.
- Several existing libraries and competitions focused on low-dimensional continuous-control tasks.
- Prior high-dimensional benchmarks included pole balancing, octopus-arm and cheetah models, humanoids, multi-agent games, and hardware interfaces.
- Unlike these testbeds, the proposed benchmark uses simulated environments to reduce computation time and encourage experimental reproducibility.
- The benchmark also provides a much larger collection of tasks with varying difficulty than the aforementioned testbeds.
8. Conclusion
The paper presents a broad continuous-control benchmark and evaluates several reinforcement-learning algorithms with general policy parameterizations. TNPG, TRPO, and DDPG are effective among the implemented methods, while hierarchical tasks remain difficult.
- The benchmark covers a wide variety of challenging continuous-control tasks and evaluates several reinforcement-learning algorithms.
- TNPG, TRPO, and DDPG are effective methods for training deep neural network policies among the implemented algorithms.
- Poor performance on the proposed hierarchical tasks indicates a need for algorithms that can address those tasks more effectively.
1. Task Specifications
The benchmark specifies continuous-control environments ranging from basic balancing and swing-up tasks to locomotion, partial-observation, system-identification, and hierarchical settings. Tasks define observations, actions, rewards, termination conditions, and selected disturbances or model variations.
- Basic Tasks: The task suite includes basic control problems such as cart-pole balancing, swing-up, Mountain Car, Acrobot Swing Up, and double inverted-pendulum balancing.
- Locomotion Tasks: Locomotion tasks vary in morphology and difficulty, including Swimmer, Hopper, Walker, Half-Cheetah, Ant, and humanoid models.
- Other Tasks: The benchmark includes limited-sensor, noisy-observation, delayed-action, system-identification, and food-collection settings.
- Noisy Observations and Delayed Actions: Gaussian observation noise uses σ = 0.1, while delayed actions correspond to three discretization frames in the specified tasks.
2. Experiment Parameters
The experiments use a shared time-varying feature encoding for linear baselines and report task-category parameters alongside hyperparameter search ranges and selected best settings. The tables organize settings for the principal algorithms.
- All batch gradient-based algorithms use the same time-varying feature encoding for the linear baseline.
- The feature encoding operates on the state vector using element-wise products.
- Table 2 reports experiment parameters for all four task categories.