Source-linked AI summary
Deep Reinforcement Learning for Robotic Manipulation with Asynchronous Off-Policy Updates
Shixiang Gu, Ethan Holly, Timothy Lillicrap, Sergey Levine
TL;DR
Real-robot reinforcement learning has been limited by sample complexity, hand-engineered representations, and demonstrations. The paper introduces asynchronous off-policy NAF with parallel experience collection, showing complex manipulation learned from scratch, including door opening in about 2.5 hours with 100% success across 20 trials. The approach still relies on task-specific reward guidance, and binary rewards make learning substantially harder.
Problem
Direct deep reinforcement learning has been restricted in robotics because of apparent high sample complexity and reliance on engineered representations or demonstrations for practical training.
Method
The paper extends NAF asynchronously, separating centralized Q-function training from parallel robot data collection and using general-purpose neural-network representations.
Results
2.5 hours and 100% success across 20 consecutive trials were achieved for door opening with two workers, learned from scratch without human demonstrations.
Takeaways & Limitations
Asynchronous off-policy deep reinforcement learning can learn complex manipulation skills from scratch on real physical robots without purpose-built representations or human demonstrations.
Takeaways & Limitations
The tasks use shaped reward guidance; replacing it with only binary success rewards makes learning substantially more difficult and exploration considerably greater.
Abstract
from arXiv · showhide
Reinforcement learning holds the promise of enabling autonomous robots to learn large repertoires of behavioral skills with minimal human intervention. However, robotic applications of reinforcement learning often compromise the autonomy of the learning process in favor of achieving training times that are practical for real physical systems. This typically involves introducing hand-engineered policy representations and human-supplied demonstrations. Deep reinforcement learning alleviates this limitation by training general-purpose neural network policies, but applications of direct deep reinforcement learning algorithms have so far been restricted to simulated settings and relatively simple tasks, due to their apparent high sample complexity. In this paper, we demonstrate that a recent deep reinforcement learning algorithm based on off-policy training of deep Q-functions can scale to complex 3D manipulation tasks and can learn deep neural network policies efficiently enough to train on real physical robots. We demonstrate that the training times can be further reduced by parallelizing the algorithm across multiple robots which pool their policy updates asynchronously. Our experimental evaluation shows that our method can learn a variety of 3D manipulation skills in simulation and a complex door opening skill on real robots without any prior demonstrations or manually designed representations.
I. INTRODUCTION
The paper addresses the sample complexity and engineering burden that have limited direct deep reinforcement learning on real robots. It extends off-policy deep Q-function methods to complex manipulation and asynchronous multi-robot learning.
- Real-world robotic reinforcement learning commonly requires hand-engineered policy representations and human demonstrations.
- High apparent sample complexity has limited direct deep reinforcement learning on real robotic platforms.
- Off-policy deep Q-function algorithms such as DDPG and NAF can achieve training times suitable for real robotic systems.
- Asynchronous NAF parallelizes learning across multiple robots that pool policy updates.
- The method learns door opening from scratch using general-purpose neural networks without human demonstrations.
III. BACKGROUND
The paper formulates robotic reinforcement learning as maximizing discounted reward and motivates off-policy Q-function learning for data efficiency. It focuses on NAF for continuous actions and real-system training.
- The reinforcement learning objective is to maximize expected discounted future reward under an optimal policy.
- Off-policy Q-learning improves data efficiency by reusing past experience through Q-function estimation.
- Continuous-action Q-learning requires specialized updates because directly maximizing a deep-neural-network Q-function is intractable.
- DDPG and NAF perform comparably in simulation, with NAF slightly better overall on the examined tasks.
- The paper develops parallelized NAF because robotic learning time is often constrained by real-time data collection.
IV. ASYNCHRONOUS TRAINING OF NORMALIZED ADVANTAGE FUNCTIONS
The proposed asynchronous NAF separates network training from robot experience collection. A central learner updates a shared replay buffer while distributed workers collect transitions and receive updated policies.
- Asynchronous NAF uses one learner thread for training and one or more worker threads for data collection.
- Parallel workers make experience collection across multiple robots straightforward when data collection limits training speed.
- The learner trains the deep Q-function asynchronously using experience stored in a replay buffer.
- Worker threads execute the current policy on individual robots and send observations, actions, rewards, and next states to the central server.
- Decoupling training from collection lets robot controllers run in real time without backpropagation delays.
B. Safety Constraints
The method constrains exploration to make real-world reinforcement learning safer. It limits joint motion and end-effector position, adding contact-specific heuristics when needed.
- Each experiment limits commanded joint velocity and imposes strict joint-position bounds.
- A bounding sphere constrains end-effector motion, projecting unsafe commands onto its surface with a corrective inward velocity.
- These constraints suffice for contact-free experiments, while contact tasks require additional safety heuristics.
C. Network Architectures
The paper uses compact state representations with feed-forward neural networks to parameterize NAF and DDPG action-value functions and policies. Linear-NAF provides a simpler comparison based on bounded linear policies.
- C. Network Architectures: The state combines joint angles, end-effector positions, their time derivatives, and task-dependent target information.Target inputs differ by task, including end-effector goals for reaching and handle-related information for door opening.
- C. Network Architectures: Two-hidden-layer networks with 100 units per layer parameterize the NAF and DDPG functions and policies.NAF parameterizes µ(x), L(x), and V(x), while DDPG parameterizes µ(x) and Q(x,u).
- C. Network Architectures: Linear-NAF replaces the deep representation with µ(x) = f(k + Kx), using learnable linear parameters and Tanh-bounded actions.The comparison tests whether simpler generic parameterizations can represent the policies and action-value functions adequately.
- C. Network Architectures: With identity f, the model corresponds to a globally quadratic Q-function and a linear feedback policy.The Tanh nonlinearity makes the Q-function nonlinear with respect to state-action features.
V. SIMULATED EXPERIMENTS
The simulated experiments use MuJoCo environments to compare learning design choices across robotic arms, control settings, and 3D manipulation tasks. The setup includes reaching, door manipulation, and pick-and-place scenarios.
- V. SIMULATED EXPERIMENTS: MuJoCo simulation enables rapid comparisons of update frequencies, parallelism, network architectures, and other hyperparameters.The simulated 7-DoF arm also models the physical robot used in experiments.
- V. SIMULATED EXPERIMENTS: The 7-DoF arm runs at 20Hz, while the JACO arm runs at 100Hz.Gravity is disabled for the 7-DoF arm and enabled for JACO, reflecting different robotic settings.
- V. SIMULATED EXPERIMENTS: The tasks include random-target reaching, door pushing, door pulling, and pick & place in 3D environments.The 7-DoF arm handles reaching and door tasks, whereas JACO performs pick & place.
- V. SIMULATED EXPERIMENTS: The reaching task samples a new target uniformly from a 0.2m cube around a fixed point.The arm starts from a fixed configuration, and the state contains 20 dimensions.
1) Reaching (7-DoF arm):
The simulated door task requires a 7-DoF arm to manipulate a randomly positioned door handle and open the door in the correct direction. The reward combines handle reaching with door-opening progress.
- 1) Reaching (7-DoF arm):: The reward combines end-effector closeness to the handle with the amount the door opens in the correct direction.The two components use end-effector-to-handle distance and quaternion-based handle or door-angle information.
- 1) Reaching (7-DoF arm):: The JACO pick-and-place task requires grasping an airborne stick and moving it to randomly sampled targets.The hand begins near but not in contact with the stick, so grasping must be learned.
B. Neural Network Policy Representations
The experiments compare deep neural representations with Linear-NAF across robotic tasks. Linear models can solve simpler reaching and pick-and-place tasks but fail on the more complex door tasks.
- B. Neural Network Policy Representations: The study compares DDPG, NAF, and Linear-NAF to test whether deep representations are necessary for learning complex tasks from scratch.The comparison also examines convergence relative to simpler linear models.
- B. Neural Network Policy Representations: Linear-NAF learns good reaching and pick & place policies but converges significantly slower than NAF and DDPG.This result is reported across the corresponding experiments summarized in Figure 3 and Table 4.
- B. Neural Network Policy Representations: The authors suggest that policy expressivity may affect reinforcement-learning behavior through the interaction between data collection and model learning.They identify further investigation of this interaction as future work.
- B. Neural Network Policy Representations: Linear-NAF completely fails to learn perfect policies for the door tasks.The door tasks require hooking onto handles at different locations, turning them, and pushing or pulling.
C. Asynchronous Training
Asynchronous NAF training uses separate training and collector threads, with worker count affecting learning speed and final policy performance. In simulation and real-robot experiments, multiple workers can improve learning, although additional workers eventually provide limited benefit.
- Asynchronous mechanism: Collector threads gather data while a training thread continuously updates the network, allowing workers to pool experience asynchronously.The training frequency is constrained by network size and hardware, while each collector runs at a specified control frequency.
- Simulation findings: Increasing data collection can accelerate convergence, but final policy performance depends strongly on the ratio between collection and training speeds.Beyond a limit, collecting additional data does not further accelerate learning.
- Real-world constraints: Stricter safety measures and more aggressive real-world targets reduced performance relative to simulation, while door pushing could not be tested on the hardware.The real experiments used slower movements, tight joint limits, and a different reaching success criterion.
- Door opening: Two workers learned a door-opening policy achieving a 100% success rate across 20 consecutive trials.The result was obtained using asynchronous NAF in simulation.
- Real-robot reaching: One worker suffered in both learning speed and final policy performance during 7-DoF random-target reaching on real robots.The figure compares asynchronous NAF training with different worker counts.
A. Random Target Reaching
Real-robot reaching was evaluated with up to four workers using the same architecture and hyperparameters as simulation. Multiple workers improved learning speed over one worker, but gains beyond two workers were limited and slow collection could also harm final performance.
- Practical considerations: Real-world parallel-training benefits require accounting for reset delays, variable training speed, and physical discrepancies among robots.These factors were absent from the controlled simulation setting.
- Experimental setup: Up to four robots collected data while a training thread updated the shared network continuously at approximately 100 Hz.Robots executed policies at 20 Hz, using the simulation architecture and hyperparameters.
- Learning speed: Two or four workers significantly improved learning speed over one worker, with no substantial additional gain beyond two workers on this simple task.The comparison was conducted on real-world 7-DoF random-target reaching.
- Worker mismatch: A single worker could slow learning and hurt final policy performance when data collection was too slow relative to training.This effect was observed when the training and collection threads were not synchronized.
B. Door Opening
Asynchronous NAF learned a complex door-opening policy from scratch on physical robots, reaching reliable performance within a few hours. The section also identifies reward-shaping and multi-robot experience diversity as important boundaries for practical generalization.
- Effect of parallelism: Two workers reached 100% success in 2.5 hours, whereas one worker required significantly more than 4 hours.The comparison uses parameter updates, with 100,000 updates corresponding to approximately half an hour before periodic evaluation delays.
- Learning progression: Learning progressed from exploring free space, to sporadic handle contact, to consistent reaching and successful door opening.The reward plateau near zero corresponded to positioning the hook near the handle without pulling the door open.
- Conclusion: The paper concludes that asynchronous deep reinforcement learning can acquire complex manipulation skills from scratch on physical robots, while more learners reduce simulated training times.The approach uses an asynchronous NAF variant and practical extensions for safe, efficient training on physical systems.
- Reward limitation: The door task used shaped rewards for gripper-to-handle distance and door-pose difference, making it substantially easier than learning from a binary success signal alone.With only binary rewards, both reacher and door tasks required considerably more exploration.
- Multi-robot generalization: Future work must determine how to integrate diverse experiences from multiple robots, such as robots operating on different doors, into one policy.The paper suggests explicit exploration, separate policies with distillation, or other mechanisms to support generalization across door types.