Source-linked AI summary
Multi-Goal Reinforcement Learning: Challenging Robotics Environments and Request for Research
Matthias Plappert, Marcin Andrychowicz, Alex Ray, Bob McGrew, Bowen Baker, Glenn Powell, Jonas Schneider, Josh Tobin, Maciek Chociej, Peter Welinder, Vikash Kumar, Wojciech Zaremba
TL;DR
The report addresses the need for challenging multi-goal continuous-control benchmarks and improved reinforcement-learning methods. It introduces Fetch and Shadow Hand environments with Gym-compatible goal interfaces and evaluates DDPG with HER, finding that HER substantially outperforms vanilla DDPG across Fetch and hand tasks while motivating further research questions.
Problem
Reinforcement learning needs challenging multi-goal robotics environments and improved methods for learning from sparse rewards.
Method
The paper releases Fetch and Shadow Dexterous Hand environments with explicit goals, an extended gym.GoalEnv interface, exposed reward recomputation, and DDPG/HER benchmark evaluations.
Results
DDPG+HER substantially outperforms vanilla DDPG across the Fetch and hand environments, learning especially well with sparse rewards, though HandManipulatePen is not fully solved.
Takeaways & Limitations
The suite provides challenging multi-goal robotics benchmarks and motivates research on unbiased HER, faster information propagation, and combining HER with on-policy algorithms.
Takeaways & Limitations
HER changes the joint distribution of replayed tuples in an unprincipled way and could make training impossible in extremely stochastic environments, although this was not observed in practice.
Abstract
from arXiv · showhide
The purpose of this technical report is two-fold. First of all, it introduces a suite of challenging continuous control tasks (integrated with OpenAI Gym) based on currently existing robotics hardware. The tasks include pushing, sliding and pick & place with a Fetch robotic arm as well as in-hand object manipulation with a Shadow Dexterous Hand. All tasks have sparse binary rewards and follow a Multi-Goal Reinforcement Learning (RL) framework in which an agent is told what to do using an additional input. The second part of the paper presents a set of concrete research ideas for improving RL algorithms, most of which are related to Multi-Goal RL and Hindsight Experience Replay.
1 Environments
The report introduces challenging continuous-control environments built on Fetch and Shadow Dexterous Hand hardware, released through OpenAI Gym with MuJoCo simulation. These tasks use sparse binary rewards, explicit goals, and a shared multi-goal interface, while benchmark results show HER substantially improves learning across Fetch and hand environments.
- Environment platform: All environments are released in OpenAI Gym and use the MuJoCo physics engine for simulation.The interface extends Gym through the newly introduced gym.GoalEnv.
- Environments: The suite covers Fetch reaching, pushing, sliding, and pick-and-place, plus Shadow Hand reaching and manipulation of blocks, eggs, and pens.Fetch tasks use a 7-DoF arm, while the Shadow Hand has 24 degrees of freedom.
- Task formulation: Fetch and hand tasks use sparse binary rewards, returning 0 when the goal is achieved within task-specific tolerance and −1 otherwise.Fetch actions control Cartesian gripper movement and gripper opening; hand actions control non-coupled joints.
- Multi-goal interface: Goals encode desired task outcomes, while desired_goal and achieved_goal expose target and current outcomes for goal-conditioned learning.The reward function is exposed so rewards can be recomputed under substituted goals, enabling HER-style algorithms.
- Benchmark results: On Fetch tasks beyond reaching, DDPG+HER outperforms other configurations and learns effectively with sparse rewards.FetchReach is easy enough that all four evaluated configurations can solve it, whereas vanilla DDPG generally finds dense rewards easier than sparse rewards.
- Benchmark results: Across hand environments, DDPG+HER significantly outperforms vanilla DDPG, although HandManipulatePen remains unsolved and reaching alone is an unreliable benchmark.HER learns partly successful policies on all hand environments, while some baseline runs fail to learn at all.
2 Request for Research
The report identifies open research problems in Multi-Goal RL and HER, spanning goal selection, bias, hierarchy, value functions, information propagation, returns, and on-policy learning. It frames these as concrete directions for broadly applicable RL improvements.
- Hindsight Experience Replay: The report proposes automatic hindsight-goal generation that learns which replay goals are most valuable, potentially using Bellman error.Candidate goals could come from achieved or observed goals, or a separate neural network; the authors connect this idea to prioritized replay.
- Hindsight Experience Replay: Unbiased HER is proposed because HER changes the replayed tuple distribution and could theoretically make training impossible in highly stochastic environments.Importance sampling might correct the bias but could produce prohibitively high gradient variance.
- Hierarchical RL: HER could be extended to hierarchical RL by replaying hindsight higher-level actions, although this approach may be unprincipled and destabilize training.The proposed example replaces a requested subgoal A with the state B actually reached by the lower-level policy.
- Value Functions: The report asks whether richer value functions should condition on additional quantities such as discount factors or binary-reward success thresholds.It relates this question to prior value functions conditioned on goals or time horizons.
- Faster Information Propagation: Target networks limit learning speed because one-step bootstrapping propagates return information backward only one step per update.The authors note that DDPG+HER often learns slowly during early training and seek faster propagation methods.
- Additional RL Directions: Further directions include combining HER with multi-step returns or on-policy algorithms, and improving RL under very frequent actions.Multi-step returns require corrections because HER data are extremely off-policy, while frequent actions create exploration and bootstrapping difficulties.
A Goal-based API Examples
The goal-based API exposes desired and achieved goals, supports reward recomputation under substituted goals, and provides wrappers for standard array-based RL implementations. The examples demonstrate these interfaces in a FetchReach environment.
- Exposed reward function: The exposed reward function lets algorithms recompute rewards after substituting goals, which HER-style methods require.The environment’s info dictionary can retain goal-independent simulation information needed for recomputation.
- Goal-based observations: A normal FetchReach step returns a dictionary observation containing achieved and desired goals, and the original reward equals compute_reward applied to them.The example explicitly checks this equality after sampling an action.
- Compatibility wrapper: FlattenDictWrapper converts selected dictionary observation keys into a standard array representation for compatibility with existing RL implementations.The example selects observation and desired_goal, producing a NumPy array with shape (13,).
B Hyperparameters
The experiments use a hyperparameter search followed by fixed settings for DDPG and HER comparisons. The search samples configurations on HandManipulateBlockRotateZ-v0 and selects the best by area under the test-success-rate curve across tasks.
- Search space: The search varies actor and critic learning rates, Polyak averaging, batch size, random-action probability, and Gaussian-noise scale.Each parameter is evaluated over the explicitly listed candidate values.
- Search procedure: The study randomly samples 40 hyperparameter combinations and trains each across four DDPG/HER sparse or dense configurations.HandManipulateBlockRotateZ-v0 is used because all four configurations can learn there, with three random seeds per configuration and combination.
- Selection criterion: Hyperparameter combinations are ranked by the area under the test success rate curve, selecting the best performance across all tasks.The report states that detailed hyperparameters are described in prior work.
- Fixed settings: The selected settings use three 256-unit ReLU layers, 1 · 10^-3 actor and critic learning rates, a 10^6-transition buffer, batch size 256, and Polyak coefficient 0.95.The fixed setup also uses 19 MPI workers, 10 test rollouts per epoch, and the other listed training parameters.
C Full Benchmark Results
The benchmark includes multiple Shadow Hand manipulation environment variants, covering full block manipulation, several block-rotation objectives, and egg and pen rotation.
- Shadow Hand environments: The suite includes HandManipulateBlockFull-v0 for full block manipulation.
- Shadow Hand environments: Block rotation is represented by parallel, XYZ, and Z-specific environment variants.
- Shadow Hand environments: The suite also includes HandManipulateEggRotate-v0 and HandManipulatePenRotate-v0.