Source-linked AI summary
Hindsight Experience Replay
Marcin Andrychowicz, Filip Wolski, Alex Ray, Jonas Schneider, Rachel Fong, Peter Welinder, Bob McGrew, Josh Tobin, Pieter Abbeel, Wojciech Zaremba
TL;DR
Sparse binary rewards make reinforcement learning difficult and often motivate complicated reward engineering. The paper introduces Hindsight Experience Replay, which relabels replayed episodes with achieved goals and combines with off-policy RL. HER enables robotic pushing, sliding, and pick-and-place policies, including successful physical-robot deployment without finetuning.
Problem
Reward shaping is difficult to engineer for sparse binary task-completion signals, especially in complex robotic environments.
Method
HER replays each episode with an alternative goal achieved in that episode while retaining the original goal, using off-policy learning.
Results
HER trained pushing, sliding, and pick-and-place policies while vanilla RL failed, and the pick-and-place policy worked on a physical robot without finetuning.
Takeaways & Limitations
HER makes applying off-policy RL to problems with sparse and binary rewards possible across the demonstrated robotic tasks.
Takeaways & Limitations
Replaying goals close to those achieved in the near future did not outperform the future strategy.
Abstract
from arXiv · showhide
Dealing with sparse rewards is one of the biggest challenges in Reinforcement Learning (RL). We present a novel technique called Hindsight Experience Replay which allows sample-efficient learning from rewards which are sparse and binary and therefore avoid the need for complicated reward engineering. It can be combined with an arbitrary off-policy RL algorithm and may be seen as a form of implicit curriculum. We demonstrate our approach on the task of manipulating objects with a robotic arm. In particular, we run experiments on three different tasks: pushing, sliding, and pick-and-place, in each case using only binary rewards indicating whether or not the task is completed. Our ablation studies show that Hindsight Experience Replay is a crucial ingredient which makes training possible in these challenging environments. We show that our policies trained on a physics simulation can be deployed on a physical robot and successfully complete the task.
1 Introduction
RL has achieved successes across simulated and robotic tasks, but robotics often requires carefully engineered reward functions. HER addresses this challenge by replaying experience with alternative achieved goals, enabling learning from sparse binary rewards.
- RL with neural networks has succeeded in simulated decision-making and diverse robotic tasks.
- Robotic RL commonly requires carefully shaped reward functions, demanding both RL expertise and domain knowledge.This limits applicability when admissible behavior is unknown.
- Humans can learn from undesired outcomes by treating them as evidence for alternative goals.A missed hockey shot can indicate actions would succeed if the net were farther right.
- HER replays episodes with goals achieved during those episodes and combines with arbitrary off-policy RL algorithms.It uses universal policies conditioned on both the current state and a goal state.
2 Background
The paper formalizes RL as agent-environment interaction and reviews DQN, DDPG, and UVFA. These methods differ in action spaces, value or policy representations, and whether goals condition the policy and value function.
- RL consists of states, actions, rewards, transitions, initial-state sampling, and a discount factor in an agent-environment interaction.
- A deterministic policy maps states to actions, while an episode repeatedly applies that policy and accumulates discounted rewards.
- Deep Q-Networks (DQN): DQN approximates the optimal Q-function for discrete actions and trains from replay-buffer transitions toward Bellman targets.A separate, slower-changing target network can stabilize target computation.
- Deep Deterministic Policy Gradients (DDPG): DDPG uses actor and critic networks for continuous actions, training the critic on actor-generated targets and the actor through the critic.
- Universal Value Function Approximators (UVFA): UVFA extends value-function approximation to multiple goals by conditioning policies and Q-functions on both states and goals.
3 Hindsight Experience Replay
Hindsight Experience Replay addresses sparse-reward learning by replaying trajectories with alternative goals, including goals achieved in the final state. It combines with off-policy RL and can act as an implicit curriculum while avoiding reward shaping.
- 3.1 A motivating example: Standard RL can fail in bit-flipping environments with n > 40 because it never experiences rewards other than −1.Exploration methods do not solve the impracticality of exploring the large state space; reward shaping is the standard alternative.
- 3.1 A motivating example: HER re-examines failed trajectories under alternative goals, such as the final achieved state, and stores those transitions alongside the original-goal transitions.A trajectory that fails for the intended goal still provides information about achieving its final state.
- 3.2 Multi-goal RL: HER trains goal-conditioned policies for multiple goals and can be combined with arbitrary off-policy RL algorithms through experience replay.The approach uses goals as policy inputs and relies on off-policy replay to reinterpret trajectories.
- 3.2 Multi-goal RL: Goals may represent exact target states or predicates specifying only selected properties of the state.The paper assumes a mapping m from each state to a goal satisfied by that state, which is the identity when goals equal states.
- 3.3 Algorithm: The simplest replay strategy adds the goal achieved in the episode’s final state while retaining the original pursued goal.The paper experimentally compares different types and quantities of additional replay goals.
- 3.3 Algorithm: HER can function as an implicit curriculum because replayed goals naturally progress from easy goals to more difficult ones.Unlike explicit curricula, this does not require control over the distribution of initial environment states.
4 Experiments
The experiments evaluate HER in simulated robotic manipulation across pushing, sliding, and pick-and-place tasks under sparse rewards, then test replay strategies, reward shaping, and physical deployment. HER enables learning across these tasks, performs better than shaped-reward variants in the reported experiments, and transfers to a physical robot after observation-noise retraining.
- Environments: The study uses a simulated 7-DOF Fetch arm to evaluate pushing, sliding, and pick-and-place with randomized object and target positions.The tasks require moving a box or puck to a target; pick-and-place uses an airborne target and unlocked fingers.
- Training setup: Goals represent desired object positions within a fixed tolerance, and the sparse reward is binary according to whether the goal is achieved.The policy receives gripper, object, target, and finger-distance information, while the Q-function also receives velocity features.
- HER evaluation: DDPG without HER cannot solve any task, whereas DDPG with HER solves all three tasks almost perfectly.Count-based exploration makes only some progress on sliding, supporting HER's central role in these experiments.
- Single-goal evaluation: HER remains much better than pure DDPG in the single-goal setting, while training with multiple goals makes HER learn faster.The authors therefore advise training on multiple goals even when only one goal matters in practice.
- Reward shaping: Neither DDPG nor DDPG+HER successfully solves the tested tasks with the shaped reward functions.The authors attribute this to mismatch with the success condition and penalties that can hinder exploration, and conclude that simple domain-agnostic shaping performs poorly.
- Replay strategies: Future-goal replay with k = 4 or 8 performs best, and it is the only tested strategy that solves sliding almost perfectly.Increasing k above 8 degrades performance because normal replay data becomes a very small fraction of the buffer.
- Physical deployment: After adding Gaussian observation noise with std=1cm, the simulated pick-and-place policy's physical-robot success rate increases from 2/5 to 5/5 without finetuning.The initial failures reflected sensitivity to box-position estimation errors caused by training on perfect simulated state.
5 Related work
HER relates to experience replay, multi-task value-function learning, and implicit curriculum learning. Its goal-sampling strategy is contrasted with prior curriculum approaches and includes evidence that near-future goals are most valuable.
- Experience replay: Prioritized experience replay is orthogonal to HER, so the approaches can be combined.Prioritized replay changes which transitions are replayed, whereas HER changes the goals used for replay.
- Multi-goal learning: HER builds on universal value function approximators for learning policies and value functions conditioned on both states and goals.The related work identifies multi-task policy and off-policy value-function learning as relevant precedents.
- Curriculum learning: HER is framed as implicit curriculum learning because replay goals shift from easy-to-achieve states toward more difficult ones.Unlike explicit curriculum methods, HER does not require control over the initial-state distribution.
- Deployment assumption: A caveat in the related discussion is that the Q-function approximator used for deployment was trained with exact observations rather than noisy observations.It therefore did not need robustness to noisy observations during physical deployment.
6 Conclusions
The paper concludes that HER enables sparse, binary-reward learning for robotic manipulation tasks where vanilla RL fails. It also reports successful transfer of a pick-and-place policy from simulation to a physical robot without fine-tuning.
- Method scope: HER was demonstrated as compatible with arbitrary off-policy RL algorithms, including DQN and DDPG.The conclusion presents sparse- and binary-reward learning as the central capability.
- Main conclusion: HER trained policies to push, slide, and pick-and-place objects to specified positions, while vanilla RL failed on these tasks.The experiments used a robotic arm and sparse, binary rewards.
- Physical transfer: The pick-and-place policy trained in simulation performed well on a physical robot without any fine-tuning.This was reported as a deployment result for the trained policy.
A Experiment details
The experiments used simulated Fetch Robotics manipulation environments with specified network, training, input-scaling, exploration, and physics-simulation procedures. Training durations varied by task because sliding simulation was slower.
- Network architecture: Both actor and critic networks had 3 hidden layers with 64 units per layer, using ReLU hidden activations and a tanh actor output.The actor output was rescaled to [−5cm, 5cm].
- Training procedure: Training used 200 epochs, with 50 cycles per epoch, 16 episodes per cycle, and 40 minibatch optimization steps per cycle.Minibatches contained 128 samples drawn uniformly from a replay buffer of 10^6 transitions.
- Input scaling: Inputs were normalized to mean zero and standard deviation one, then clipped to [−5, 5].Scaling statistics were computed from observations encountered during training.
- Exploration: Exploration selected a uniformly random valid action with probability 20%; otherwise it added Gaussian noise with standard deviation equal to 5% of each action range.The noisy action was based on the policy-network output.
- Training time: Training took approximately 2.5h for pushing and pick-and-place and 6h for sliding using 8 CPU cores.The longer sliding time was attributed to slower physics simulation.
B Deployment on the physical robot
The physical-robot deployment used a CNN trained on randomized MuJoCo-rendered images to estimate box position, while the control policy itself ran through a simulator connected to the real robot.
- Visual state estimation: A CNN predicted box position from raw Fetch head-camera images and was trained only on randomized MuJoCo-rendered images.The authors report that domain randomization enabled the network to perform well on real-world data despite non-photorealistic training images.
- Sim-to-real control: At each episode start, the simulator was initialized with the CNN-predicted box position and the physical robot state.The simulated policy then generated joint-angle targets for the real position-controlled robot after each timestep.