Source-linked AI summary
Leveraging Demonstrations for Deep Reinforcement Learning on Robotics Problems with Sparse Rewards
Mel Vecerik, Todd Hester, Jonathan Scholz, Fumin Wang, Olivier Pietquin, Bilal Piot, Nicolas Heess, Thomas Rothörl, Thomas Lampe, Martin Riedmiller
TL;DR
High-dimensional robotics RL often depends on carefully tuned shaping rewards, while sparse rewards make exploration difficult. The paper extends DDPG with human demonstrations in prioritized replay, and reports strong results across simulated insertion tasks plus a real clip-insertion task without requiring shaping rewards.
Problem
High-dimensional continuous-control RL commonly requires carefully tuned shaping rewards to communicate goals and ease exploration, especially for challenging multi-stage insertion tasks.
Method
DDPGfD loads human demonstration and agent transitions into replay, prioritizes sampling across both sources, and uses n-step returns to propagate sparse rewards.
Results
Across simulated insertion tasks and a real Sawyer clip-insertion task, DDPGfD outperforms DDPG and learns with sparse rewards without extra shaping-reward engineering.
Takeaways & Limitations
Human demonstrations provide a practical alternative to shaping rewards for teaching challenging continuous-control insertion policies, including richer force-sensitive policies on real robots.
Takeaways & Limitations
Shaped rewards require careful tuning of dimension weights and delicate balancing between reaching and inserting stages.
Abstract
from arXiv · showhide
We propose a general and model-free approach for Reinforcement Learning (RL) on real robotics with sparse rewards. We build upon the Deep Deterministic Policy Gradient (DDPG) algorithm to use demonstrations. Both demonstrations and actual interactions are used to fill a replay buffer and the sampling ratio between demonstrations and transitions is automatically tuned via a prioritized replay mechanism. Typically, carefully engineered shaping rewards are required to enable the agents to efficiently explore on high dimensional control problems such as robotics. They are also required for model-based acceleration methods relying on local solvers such as iLQG (e.g. Guided Policy Search and Normalized Advantage Function). The demonstrations replace the need for carefully engineered rewards, and reduce the exploration problem encountered by classical RL approaches in these domains. Demonstrations are collected by a robot kinesthetically force-controlled by a human demonstrator. Results on four simulated insertion tasks show that DDPG from demonstrations out-performs DDPG, and does not require engineered rewards. Finally, we demonstrate the method on a real robotics task consisting of inserting a clip (flexible object) into a rigid object.
1 Introduction
Robotics RL needs feedback-rich policies for variable manipulation tasks, but sparse rewards make exploration difficult. The paper uses human demonstrations with replay-based DDPG to avoid carefully engineered shaping rewards.
- Motivation: Kinesthetic guidance reduces programming demands but still depends on scripted trajectories that cannot readily handle variability in tasks such as wire insertion.Richer policies must use feedback including states, forces, and raw images.
- Motivation: Reward shaping provides dense guidance but requires considerable engineering and can produce bizarre behavior or premature convergence to local minima.Sparse rewards are often natural task specifications, such as rewarding successful wire insertion.
- Contribution: The approach injects demonstrations into off-policy replay-based RL, using DDPG to learn from arbitrary transition data and eliminate reward shaping.Replay memory retains transitions long enough to propagate sparse rewards through the value function.
- Results: Simulation insertion experiments and a real Sawyer clip-insertion task suggest that sparse rewards plus a few human demonstrations can teach challenging continuous-control policies.The real-task demonstrations are collected by kinesthetically controlling the robot arm by the wrist.
2 Background
The paper frames continuous-control learning with MDPs and actor-critic neural networks. DDPG learns from off-policy replay data, making human demonstrations compatible with the algorithm.
- MDP background: An MDP comprises states, actions, rewards, transitions, and a discount factor, with actions producing rewards and successor states.The transition function specifies the probability of reaching a successor state given the current state and action.
- DDPG: DDPG represents both the policy actor and action-value critic with neural networks and trains the policy using action gradients from the critic.The actor explores by adding a random process to its selected action, while resulting transitions enter a replay buffer.
- DDPG: DDPG evaluates transitions off-policy using replay-buffer distributions and target networks to update the critic and stabilize learning.Target networks are updated periodically from their associated actor and critic networks.
- Demonstrations: Because DDPG is off-policy, it can incorporate arbitrary transition data such as human demonstrations.The experiments also use general deep-RL techniques that substantially improve DDPG, especially with demonstrations.
3 DDPG from Demonstrations
DDPGfD extends DDPG by retaining demonstrations alongside agent experience and prioritizing replay. Additional return targets and update strategies improve sparse-reward propagation and training efficiency.
- Demonstration replay: DDPGfD loads demonstration transitions into the replay buffer before training and retains both demonstration and agent transitions permanently.Demonstrations use the same reinforcement-learning transition format as agent experience.
- Prioritized replay: Prioritized replay samples important transitions more frequently and naturally controls the sampling balance between demonstrations and agent data.Demonstration transitions receive an additional priority term, while importance-sampling weights correct for the altered distribution.
- Prioritized replay: The prioritized replay probability depends on each transition’s priority, with α = 0.3 and β = 1 used to learn from the intended distribution from the beginning.The method increases the probability of sampling demonstration transitions through a positive demonstration-priority constant.
- Return targets: A mixture of 1-step and n-step critic returns propagates Q-values along trajectories in sparse-reward tasks.The n-step target evaluates a rollout of size n following a policy close to the current policy.
- Optimization: Multiple learning updates per environment step increase replay utilization but require balancing fresher data against stale samples that may misrepresent the current policy’s state distribution.Stale replay data can produce incorrect Q-values in states not previously visited by the current policy.
- Algorithm summary: The complete modification set includes demonstration replay, prioritized sampling, mixed 1-step and n-step losses, multiple updates per step, and L2 regularization.L2 regularization is applied to both actor and critic network weights.
4 Experimental setup
The experiments span insertion tasks with varied exploration difficulties, using sparse or shaped rewards and demonstrations collected through kinesthetic robot control. The setup includes four simulated tasks and a real deformable-clip insertion task.
- Reward design: A simple goal-distance reward led DDPG to a local minimum outside the socket, preventing exploration around the opening.The task goal was at the socket bottom, whereas the only viable path passed through the socket opening.
- Insertion tasks: The task suite ranges from rigid peg-in-hole and drive insertion to deformable clip and highly under-actuated cable insertion.The cable task uses a 20-link capsule chain, while the clip uses spring-loaded hinge joints at each prong.
- Reward design: The experiments compare sparse rewards of +10 at goal proximity with shaped rewards combining reaching and inserting phases.The sparse reward terminates the episode when all plug tip sites reach the specified tolerance.
- Reward design: Shaped rewards require careful weighting and delicate balancing between reaching and inserting stages.The weighting matrices determine goal and opening errors, while scaling parameters transform distances into rewards from 0 to 1.
- Robot and demonstrations: A Sawyer 7-DOF torque-controlled arm was used in simulation and real experiments, with a cuff enabling kinesthetic teaching.The real task mounted a deformable nylon clip rigidly to the robot gripper for insertion into a rigid object.
- Robot and demonstrations: Human demonstrators kinesthetically force-controlled the arm, producing transitions that could be added directly to the replay buffer; each experiment collected 100 episodes.Demonstrations averaged about 25 steps, or approximately 5 seconds, and required 10–15 minutes of robot interaction per task.
- Evaluation: Learning curves compare four approaches across four tasks using reward versus environment interaction time over 64 trials.Each episode lasted at most 5 seconds, with an agent control rate of about 6 Hz; the plots also include human demonstrations and supervised imitators.
5 Results
Across simulated insertion tasks and a real clip-insertion task, DDPGfD outperformed DDPG while reducing reliance on engineered shaping rewards. It could learn the sparse-reward real-robot task from very limited demonstrations.
- Simulated tasks: DDPGfD consistently outperformed DDPG on the evaluated simulated insertion tasks, including when DDPG received well-tuned shaping rewards.DDPGfD also learned more reliably and typically solved tasks in 2–4x fewer steps than the demonstrations.
- Simulated tasks: DDPGfD performed nearly as well with sparse rewards as with shaping rewards, whereas supervised imitation performed poorly on every task.
- Demonstration quantity: A single human demonstration was sufficient for DDPGfD to solve the sparse-reward clip-insertion task.Increasing demonstrations to 50–100 produced diminishing returns.
- Real robot: On the physical Sawyer robot, DDPGfD learned a robust clip-insertion policy with sparse rewards and faster learning than shaped DDPG.The comparison used two real-robot runs and did not require the engineered reward function.
6 Related work
Related work addresses continuous-control learning through imitation, inverse reinforcement learning, reward shaping, or model-based acceleration. DDPGfD is distinguished from these approaches by targeting continuous robotics with demonstrations and off-policy learning.
- Imitation learning: Imitation-learning methods primarily match expert demonstrations, whereas this work combines demonstrations with task rewards to improve beyond observed behavior.
- Inverse reinforcement learning: Inverse reinforcement learning learns a reward or cost under which demonstrations are optimal, but the cited approach does not scale to continuous state-action spaces and requires known dynamics.
- Inverse reinforcement learning: GCL and GAIL learn rewards and policies by matching expert-trajectory distributions without requiring known dynamics or hand-crafted features.
- Model-based control: iLQG and Guided Policy Search rely on smooth or quadratic shaped rewards and locally linear dynamics approximations for control optimization.Guided Policy Search subsequently trains a neural network to fit trajectories generated by locally optimal policies.
- Model-based control: NAF accelerates model-free continuous-control learning with learned dynamics, imagination rollouts, and iLQG-generated model-guided trajectories.
- Learning from demonstrations: DQfD is the closest related method but is restricted to discrete action spaces and therefore is not applicable to robotics.
7 Conclusion
The paper concludes that DDPGfD uses demonstrations and prioritized replay to bootstrap deep reinforcement learning for sparse-reward motor tasks. Its experiments support demonstrations as an alternative to carefully tuned shaping rewards for challenging insertion problems.
- Conclusion: DDPGfD prioritizes replay samples from both demonstrations and self-generated data, while n-step returns propagate sparse rewards across trajectories.
- Conclusion: The approach replaces difficult shaping-reward design with demonstrations for high-dimensional continuous-control tasks, including multi-stage insertion.Insertion requires reaching the insertion point and then inserting the object.
- Conclusion: Experiments suggest that sparse rewards and a few human demonstrations can provide a practical alternative to shaping for robot insertion tasks.The authors frame this as extending learning from demonstration toward richer, force-sensitive policies on real robots.
A Real robot safety
The real-robot implementation inserts a safety-oriented impedance controller between the learned agent and the robot. High-rate force and velocity updates constrain the agent while preserving a lower control frequency.
- Safety controller: An intermediate impedance controller modifies the agent’s target velocity according to externally applied forces before commands reach the robot.
- Safety controller: The controller combines the agent signal, applied forces, and sensitivity constants, then limits velocity increases while allowing rapid stopping.This design is reported to increase control stability.
- Control rates: The agent operates at 5Hz while force and control-velocity updates run at 1kHz, maintaining a physically safe system.
- Training procedure: Algorithm 1 initializes DDPGfD with demonstrations in the replay buffer and outputs actor and critic networks for interaction-based learning.
- Training procedure: Training adds each environment transition to the replay buffer and samples minibatches with prioritization for critic updates.