Source-linked AI summary
Data-efficient Deep Reinforcement Learning for Dexterous Manipulation
Ivaylo Popov, Nicolas Heess, Timothy Lillicrap, Roland Hafner, Gabriel Barth-Maron, Matej Vecerik, Thomas Lampe, Yuval Tassa, Tom Erez, Martin Riedmiller
TL;DR
Dexterous manipulation requires data-efficient learning because real-robot interactions are costly and complex tasks demand large amounts of data. The paper evaluates simulation-based end-to-end learning and extends DDPG with decoupled replay updates, distributed training, composite rewards, and instructive starting states. The combined approach learns robust Lego grasping and stacking in fewer than 10 million transitions, with successful-trajectory starts reducing this to 1 million transitions.
Problem
Robotics needs broadly applicable dexterous-manipulation learning that keeps prior assumptions low while using a reasonable amount of expensive interaction data.
Method
The paper extends model-free DDPG with independently scheduled replay updates, distributed data collection and training, composite shaping rewards, and biased initial states.
Results
The combined methods reliably learn robust full-task policies from scratch in less than 10 million environment transitions and in 1 million transitions from successful-trajectory start states.
Takeaways & Limitations
The results suggest that successful stacking policies may be feasible to train through interactions on real robots.
Abstract
from arXiv · showhide
Deep learning and reinforcement learning methods have recently been used to solve a variety of problems in continuous control domains. An obvious application of these techniques is dexterous manipulation tasks in robotics which are difficult to solve using traditional control theory or hand-engineered approaches. One example of such a task is to grasp an object and precisely stack it on another. Solving this difficult and practically relevant problem in the real world is an important long-term goal for the field of robotics. Here we take a step towards this goal by examining the problem in simulation and providing models and techniques aimed at solving it. We introduce two extensions to the Deep Deterministic Policy Gradient algorithm (DDPG), a model-free Q-learning based method, which make it significantly more data-efficient and scalable. Our results show that by making extensive use of off-policy data and replay, it is possible to find control policies that robustly grasp objects and stack them. Further, our results hint that it may soon be feasible to train successful stacking policies by collecting interactions on real robots.
I. INTRODUCTION
The paper studies data-efficient end-to-end learning for dexterous Lego manipulation in simulation, targeting methods practical enough to assess on real robots. It extends DDPG with improved replay, distributed training, shaping rewards, and biased initial states, enabling robust full-task policies with relatively few transitions.
- Dexterous manipulation requires flexible interaction with varied objects and conditions, which is difficult for manually designed controllers.
- Learning methods typically require data volumes that grow with task complexity, making simulated training difficult to transfer to real robots because of unrealistic learning times.
- The study evaluates end-to-end learning of a contact-rich Lego task requiring reaching, grasping, and stacking with a 9-DoF robotic arm.
- The authors extend DDPG by decoupling network-update frequency from environment interactions and distributing data collection and training across computers and robots.
- Two additional strategies inject prior knowledge through shaping rewards for compositional tasks and biased initial states resembling curriculum or apprenticeship learning.
- The combined methods learn robust full-task policies from scratch in less than 10 million transitions, or 1 million transitions from successful-trajectory start states.The former corresponds to less than 10 hours on 16 robots, while the latter corresponds to less than 1 hour.
II. RELATED WORK
Related work frames the paper around the tension between expressive learning policies and data-efficient robotics. Model-free value-function methods offer off-policy data reuse without requiring a complete model or fully observed state space, while demonstrations and guided methods impose different practical requirements.
- Policy-search methods handle continuous, high-dimensional actions but can scale poorly with the number of parameters.
- Restrictive trajectory representations can make real-hardware learning tractable, but they limit the problems that the policy space can solve.
- Neural-network reinforcement learning provides richer function approximators for end-to-end learning on challenging continuous-control problems.
- Guided policy search improves data efficiency through aggressive local updates and extensive neural-policy training before collecting more real-world data.
- Guided policy search relies on a well-defined, fully observed state space, while model-based variants can struggle with strongly discontinuous dynamics.
- Model-free value-function approaches reuse data effectively and do not require full access to the state space or an environment model.
- Demonstrations can supply cost functions or bootstrap learning, but kinesthetic teaching requires a human to guide the robot through the full movement.
III. BACKGROUND
The background formalizes reinforcement learning and places DDPG within continuous-action, Q-function-based methods. DDPG combines deterministic policy gradients with experience replay and target networks, enabling off-policy reuse and decoupling environment data collection from network updates.
- Reinforcement learning trains an agent through sequential environment interactions to maximize expected cumulative reward.
- DPG improves a deterministic continuous-action policy by backpropagating action-value gradients from a learned Q-function.
- During learning, actions may be noisy versions of the deterministic policy, producing a transition distribution that differs from the policy’s marginal distribution.
- DDPG adds experience replay and slowly updated target networks to DPG, improving training stability.
- NAF is a DDPG variant with an action-quadratic Q-function, while DDPG and NAF are continuous-action counterparts of DQN.
- Off-policy learning decouples experience collection from policy and value-network updates, allowing many parameter updates per environment step.
IV. TASK AND EXPERIMENTAL SETUP
Experiments use a MuJoCo simulation matched to a 9-DoF Jaco-arm setup and evaluate grasping, stacking, and their composite Lego task. Episodes last 150 steps, and performance is averaged over repeated agents and evaluation episodes.
- The composite task picks up one Lego Duplo brick and stacks it onto another, with grasping and stacking also tested as isolated subtasks.
- The task’s subtask configurations pair starting states with rewards for grasping, StackInHand, and stacking.
- Experiments run in physically plausible MuJoCo simulation with an arm closely matched to a real-world Jaco setup.
- Each episode lasts 150 steps of 50ms, giving the agent 7.5 seconds to complete the task.
- The agent observes arm and gripper joint states, brick poses, and brick-to-gripper distances, while nine continuous actions set arm and finger velocities.
- Increasing replay updates changed the number of environment interactions needed by roughly a factor of two to three.
- For each experimental condition, ten agents with different initial network parameters are trained and evaluated every 30 episodes for ten episodes.
V. ASYNCHRONOUS DPG WITH VARIABLE REPLAY STEPS
Variable replay steps improve DDPG’s data efficiency, while asynchronous workers parallelize training and interaction to reduce experimental runtime. Together, these extensions enable successful manipulation policies with substantially fewer interactions and faster wall-clock execution.
- Multiple mini-batch replay steps: DPG-R configures a fixed number of mini-batch updates per environment step, and increasing this ratio improves data efficiency on Grasp and StackInHand.With 40 updates, successful stacking and grasping policies appear after 200,000 and 300,000 interactions, respectively.
- Asynchronous DPG: Increasing replay steps can make training computation scale with the number of updates, motivating parallelization of network training and environment interaction.This concern applies both when simulation runtime is update-dominated and when real-robot runtime is interaction-dominated.
- Asynchronous DPG: Asynchronous DPG uses multiple workers whose local actor and critic copies apply gradients to shared parameters without synchronization.Workers can be configured with shared or independent replay buffers.
- Asynchronous DPG: For ADPG-R with 16 workers, increasing replay steps improves data efficiency, although less strongly than for single-worker DPG-R.The comparison is reported for both Grasp and StackInHand.
- Asynchronous DPG: With best replay-step and learning-rate choices, multiple workers reduce theoretical wall-clock time by about 16x for StackInHand and 8x for Grasp.The wall-clock measure uses environment steps per worker and assumes communication is negligible relative to interaction and gradient computation.
- Asynchronous DPG: Distributing training and data collection across computers and robots can reduce experiment runtime enough to make more challenging experiments feasible.The authors use asynchronous DPG extensively in the remaining experiments.
VI. COMPOSITE SHAPING REWARDS
The paper addresses sparse-reward exploration in the composite Stack task by designing rewards that provide progress signals across reaching, grasping, and stacking. These rewards can enable full-task learning, but their design is effortful and error-prone.
- Motivation: Sparse rewards provide no useful learning signal for the full Stack task because naive exploration is unlikely to reach successful grasp-and-stack states.The same sparse reward supports isolated subtasks but fails to guide exploration through the full composite task.
- Limitations: Shaping rewards require careful construction because small design mistakes can alter the optimization target or cause complete learning failure.The authors observed unsuitable grasps, premature stopping before stacking, and brick flipping from an incorrect reference point.
- Reward design: Composite shaping rewards assign increasing rewards as the agent completes components of the full task, using piecewise-constant or smoothly varying functions.For Stack, the components correspond to progress through subtasks such as reaching, grasping, and stacking.
- Reward design: The evaluated reward variants add intermediate rewards for grasping, reaching, or both to the original sparse task reward.The grasp-shaping variant rewards grasping and final stacking, while reach-and-grasp shaping also rewards proximity to the first brick.
- Results: Only continuous reach, grasp, and stack shaping solves the full task; grasp shaping enables grasping but is slow, while reach-and-grasp shaping does not progress beyond grasping.Figure 5 compares the sparse reward with three composite reward designs.
VII. LEARNING FROM INSTRUCTIVE STATES
The paper improves exploration by initializing episodes from states that reflect the task’s compositional structure, including manually defined states and states along successful trajectories. This strategy accelerates and strengthens learning while reducing reliance on elaborate reward shaping.
- Approach: Instructive-state initialization samples episode starts closer to later task stages, such as states where the first brick is already grasped for stacking.The distribution of initial states is chosen to reflect the compositional structure of the task.
- Approach: States along or near successful trajectories can be manually defined or obtained from a human demonstrator or partially successful policy.The authors describe this as apprenticeship-like teacher information conveyed through the state visitation distribution.
- Results: Augmented start-state distributions improve all non-sparse reward variants, and even simple grasp shaping sometimes yields controllers that solve the full task.The basic sparse-reward case still shows no learning under the simple augmented distribution.
- Implications: Full composite shaping becomes faster and more robust with instructive states, while trajectory-based initialization allows the simplest overall-success reward to solve the task.These results suggest that suitable start-state distributions can both speed learning and simplify reward design.
- Results: 1 million environment transitions suffice to learn a robust policy for all seeds when episodes start along trajectories from a pre-trained controller, using only sparse reward.This corresponds to less than 1 hour of interaction time on 16 simulated robots.
- Evaluation: The learned policies are evaluated for robustness across different starting states using success rates from 1000 trials.The paper also reports videos of policies performing Grasp, StackInHand, and Stack from different initial states.
VIII. CONCLUSION
The paper extends DDPG and exploration strategies to make robust dexterous manipulation more data-efficient and scalable. These techniques enable Stack-task learning at transition counts feasible for real-robot systems, while simulation-to-reality transfer and state sensing remain challenges.
- Decoupling network-update frequency from environment interaction substantially improves data efficiency and can determine whether a solution is found.
- Distributed asynchronous DDPG provides close to linear wall-clock speedup with 16 parallel workers by spreading data collection and training across computers and robots.
- Composite rewards and instructive starting states guide exploration, reduce pressure on exploration strategies, and accelerate learning for sequential or long-horizon tasks.
- The combined methods learn robust Stack policies within transition counts feasible to collect on real robots within a few days, or faster with multiple robots.
- The work is positioned as a step toward real-robot dexterous manipulation with perception, but simulation-to-reality transfer and environment-state instrumentation remain caveats.
A. Reward function
The reward functions are built from geometric and state quantities extracted from the simulated environment. These quantities describe brick height, brick locations, and the hand’s pinch position.
- Brick 1 height above the table is used as a reward-function quantity.
- The center-position site of Brick 1 provides its x, y, and z coordinates.
- A site above Brick 2 marks where the center of Brick 1 should be located after stacking.
- The hand’s pinch site approximates where the fingertips meet when the fingers close.
1) Sparse reward components:
Sparse reward components define successful subtasks using geometric conditions for reaching, grasping, and stacking. The stacking condition is evaluated relative to Brick 2’s freely moving coordinate system.
- Reach Brick 1: Reaching Brick 1 succeeds when the fingers’ pinch site lies within a virtual box around the brick’s position.
- Grasp Brick 1: Grasping Brick 1 succeeds when its height above the table exceeds threshold θ, indicating that the brick has been lifted.
- Stack: Stacking succeeds when Brick 1 lies within a displacement box relative to Brick 2.
- Stack: The stacking box uses Brick 2’s coordinate system because Brick 2 is allowed to move freely.
- Stack: The stacking criterion could be implemented in hardware using a contact sensor attached to Brick 2.
2) Shaping components:
Shaping components add distance-based guidance to the reward, directing the hand toward Brick 1 and then Brick 1 toward Brick 2. Composite reward functions combine these components with subtask predicates.
- Distance-based shaping guides the hand toward Brick 1 and then guides Brick 1 toward Brick 2.
- The shaping terms could be implemented with hardware vision that approximately identifies object centroids.
- Reward functions for Stack, grasp shaping, reach-and-grasp shaping, and full composite shaping use predicates to detect completed subtasks.
- The full reward construction combines subtask conditions with distance-based shaping components.