PaperlayerView exact revision on arXiv

Source-linked AI summary

Collective Robot Reinforcement Learning with Distributed Asynchronous Guided Policy Search

Ali Yahya, Adrian Li, Mrinal Kalakrishnan, Yevgen Chebotar, Sergey Levine

arXiv:1610.00673v1cs.LGcs.AIcs.RO

TL;DR

Single robots may not collect enough diverse experience for policies that generalize across real-world conditions. The paper proposes distributed asynchronous Guided Policy Search, where multiple robots share experience while global policy training proceeds alongside local practice, and evaluates it on four-robot door opening with better generalization, utilization, and training times than a single robot.

  • Problem

    Training policies that generalize across diverse real-world conditions requires more varied experience than a single robot can practically collect.

  • Method

    The paper extends Guided Policy Search to distributed asynchronous learning, combining shared replay-memory experience with parallel local practice and global policy training.

  • Results

    The four-robot system achieves better generalization, utilization, and training times than the single-robot alternative on vision-based door opening.

  • Takeaways & Limitations

    A single global policy generalizes across unseen door appearance and mechanical properties, including varied handles, door poses, camera calibration, and robot dynamics.

  • Takeaways & Limitations

    The method assumes that robots can execute the same policy, implicitly requiring physically similar or identical robots.

Abstract

from arXiv · show

In principle, reinforcement learning and policy search methods can enable robots to learn highly complex and general skills that may allow them to function amid the complexity and diversity of the real world. However, training a policy that generalizes well across a wide range of real-world conditions requires far greater quantity and diversity of experience than is practical to collect with a single robot. Fortunately, it is possible for multiple robots to share their experience with one another, and thereby, learn a policy collectively. In this work, we explore distributed and asynchronous policy learning as a means to achieve generalization and improved training times on challenging, real-world manipulation tasks. We propose a distributed and asynchronous version of Guided Policy Search and use it to demonstrate collective policy learning on a vision-based door opening task using four robots. We show that it achieves better generalization, utilization, and training times than the single robot alternative.

I. INTRODUCTION

The paper motivates collective policy learning because single-robot experience is insufficiently diverse for generalization, while distributed asynchronous training must balance robot utilization with synchronization. It proposes a distributed asynchronous Guided Policy Search system in which multiple robots jointly train one policy and reports better generalization, utilization, and training times than a single robot.

  • Motivation: Single-robot training may not provide enough quantity and diversity of experience for policies intended to generalize across real-world conditions.The motivation is grounded in learning complex skills from sensory inputs while avoiding extensive manual engineering.
  • Motivation: Collective policy learning lets multiple robots share experience and learn a policy jointly.The paper studies distributed and asynchronous policy learning for challenging real-world manipulation tasks.
  • Challenges: Collective learning must balance robot utilization with the compute, bandwidth, and synchronization needed to assimilate shared experience.Utilization is defined as the fraction of time robots spend collecting experience for learning.
  • Approach: The proposed system uses distributed asynchronous Guided Policy Search, with multiple robots practicing distinct task instances while jointly training a centrally maintained global policy.Robots continue local practice while the global policy trains from previously collected experience.
  • Evaluation: The system is evaluated in simulation and on a vision-based door-opening task with varying door pose and appearance, outperforming the single-robot alternative in generalization, utilization, and training times.The supplied passage states the comparison at the paper level without reporting individual numerical values.

II. RELATED WORK

The related work connects robotic policy learning to deep learning, collective robotics, and distributed training, while the paper builds on Guided Policy Search to address high-dimensional control. GPS combines trajectory-centric local policy optimization with supervised learning of a global policy, and the proposed asynchronous framework separates global and local computation.

  • Related work: Robotic skill learning has often relied on manually designed representations, which can restrict the kinds of behaviors that are learned.Dynamic movement primitives are given as an example of a representation that supports goal adaptation but remains trajectory-centric.
  • Related work: Deep learning offers more expressive policy classes, but robotic manipulation requires large amounts of on-policy experience from physical platforms.The paper frames collective experience as a way to make deep learning for manipulation more feasible.
  • Related work: Collective robotic learning pools experience across robots, while this work targets distributed asynchronous data collection and policy training for real-world robotic control.The paper states that it parallelizes both data collection and neural-network policy training across multiple machines.
  • Asynchronous distributed GPS: The asynchronous framework addresses computational bottlenecks by decoupling global workers, which train from replay memory, from local workers, which execute controllers and add experience.This design is presented as asynchronous distributed GPS.
  • Guided Policy Search: Guided Policy Search learns simple local controllers with trajectory-centric reinforcement learning, then trains a high-dimensional global policy by supervised learning.The global policy can generalize across the initial conditions represented by multiple local policies.
  • Guided Policy Search: GPS methods alternate local-policy optimization and global-policy optimization, using LQR or PI2 for local learning and stochastic gradient descent for the global policy.BADMM-based GPS and MDGPS differ in how they keep local policies close to the global policy.

A. Local Policy Optimization

GPS can use either model-based LQR or model-free PI2 for local policy optimization. These methods update local policies from sampled trajectories, while BADMM-based GPS repeatedly alternates local and global optimization before new rollouts.

  • Local optimization methods: GPS supports LQR with local models and model-free PI2 for local policy optimization.LQR requires linearized dynamics, whereas PI2 can handle highly discontinuous dynamics or non-differentiable costs without fitting linear dynamics.
  • PI2 update: PI2 updates controls using soft-max probabilities based on trajectory costs, making lower-cost trajectories more probable.The feedforward-command update uses weighted maximum likelihood estimation, with temperature selected by a KL-divergence constraint.
  • Sequential GPS: In BADMM-based GPS, local and global policy optimization alternate multiple times before new rollouts are executed.This sequential organization pauses training during rollouts and pauses rollouts during training.
  • Sampling: Both LQR and PI2 require samples to improve local policies, while MDGPS can generate samples directly from the global policy.In MDGPS, local policies exist temporarily within each iteration for policy improvement.

IV. ASYNCHRONOUS DISTRIBUTED GUIDED POLICY SEARCH

ADGPS addresses synchronous GPS bottlenecks by separating local and global workers and connecting them through replay memory. Robots collect and optimize experience while global workers asynchronously train a shared policy maintained through distributed parameter updates.

  • Motivation: Synchronous GPS causes robot downtime during policy optimization and synchronization difficulties when combining data across robots.Rollout execution and policy optimization occur sequentially.
  • ADGPS architecture: ADGPS decouples rollout execution from global policy optimization through replay memory and asynchronous local and global workers.Local workers execute controllers and add data to replay memory, while global workers continuously optimize the global policy from buffered experience.
  • Worker roles: Local workers update local policies while global workers asynchronously read replay memory and update the shared global policy.Stored rollouts may come from older policies and are reweighted using importance sampling.
  • Distributed training: Multiple robots and machines concurrently collect data and train a deep neural network policy through a centralized parameter server.The parameter server stores the global policy and receives smaller model updates instead of high-bandwidth rollout data.
  • Local-worker procedure: The local-worker procedure generates trajectories from either local or global policies and appends optimized trajectories to replay memory.Local policy optimization is performed before the optimized trajectories are stored for global training.
  • Global optimization: Global workers sample mini-batches from replay memory and optimize the global policy with one stochastic-gradient-descent step.The sampled labels come from the corresponding local policies associated with each experience instance.
  • Implementation: ADGPS was implemented in TensorFlow, with rollout execution and local policy optimization remaining sequential on each local worker.The authors note that both steps could also be performed asynchronously, although local optimization is relatively cheap.

V. EXPERIMENTAL EVALUATION

The evaluation tests whether distributed asynchronous learning accelerates training of nonlinear neural policies and whether multiple robots improve policy generalization. It examines these questions in simulation and on a real-world door-opening task with varying instances.

  • Evaluation goals: The experiments assess training-time scaling with worker count and generalization from training across multiple robots.The authors state that parallelized experience collection may not automatically accelerate learning, motivating empirical evaluation.

A. Simulated Evaluation

Simulation evaluates asynchronous GPS with multiple workers on a torque-controlled reaching task. Asynchrony slightly reduces improvement per iteration but substantially improves learning progress measured in wall-clock time.

  • Setup: The simulated study varies robot and worker counts using parallelized 7-DoF arms performing Cartesian reaching in real time.The setup is designed to mimic rollout execution times observed on real robots.
  • Asynchrony trade-off: Asynchronous training slightly reduces cost improvement per iteration because workers use older policies and data.The reduced per-iteration improvement is offset by shorter iterations from parallelized global training and data collection.
  • Training efficiency: Multiple asynchronous workers substantially improve learning rate in wall-clock time compared with standard GPS.The comparison reports wall-clock speedup alongside the relative increase in sample count caused by reduced per-iteration improvement.
  • Configurations: ADGPS-4 and ADGPS-8 represent four and eight pairs of local and global workers, while AGPS uses one asynchronous pair.Figure 4 compares these settings across four test instances using iteration count and training duration.

B. Real-World Evaluation

The real-world evaluation trains a single visuomotor policy to open varied doors despite differences in handles, door pose, camera calibration, and robot mechanics. Four torque-controlled 7-DoF arms use monocular RGB images and robot state to control the task.

  • The task trains one visuomotor policy to open doors with visual and mechanical handle differences.
  • The evaluation varies door pose, camera calibration, and mechanical properties across robots.
  • Four lightweight torque-controlled 7-DoF arms use two-finger grippers and shoulder-mounted cameras with imprecisely calibrated poses.
  • The policy receives monocular RGB images and a robot state vector while sending torque commands to all seven joints at 20Hz.
  • The network uses multiscale visual features from pooled convolutional stages and skip connections.

1) Policy pre-training:

Policy pre-training uses images and pose labels to initialize the network’s visual representation before policy learning. Shared image features support pose prediction across robots despite unknown camera offsets.

  • 1) Policy pre-training:: The convolutional layers are pretrained with a proxy pose-detection objective using automatically labeled images of training doors in varied poses.
  • 1) Policy pre-training:: The architecture extracts feature points with spatial soft-argmax, predicts object and robot poses, and then combines current and initial features with robot state to produce torques.
  • 1) Policy pre-training:: Separate robot-specific pose outputs accommodate unknown camera offsets while allowing 2-D image features to be shared across robots.

2) Policy learning:

Policy learning bootstraps the network with fixed-door optimization, then trains globally with asynchronous distributed MDGPS while perturbing door poses across iterations.

  • 2) Policy learning:: Four BADMM-based ADGPS iterations with PI2 bootstrap the fully connected layers while each door pose remains fixed.
  • 2) Policy learning:: Sixteen asynchronous distributed MDGPS iterations with PI2 follow, randomly perturbing each door pose at the start of every iteration.

3) Results:

The learned global policy generalizes across unseen door and robot conditions. It succeeds on most test configurations, remains effective under shifted camera positions, and outperforms the single-robot comparison in generalization.

  • 3) Results:: All four robots open an unseen test door in most configurations using one global policy.Evaluation used 50 trials per robot over a grid of door translations and orientations.
  • 3) Results:: The policy generalizes across door-handle appearance, door pose, camera calibration, and robot-dynamics variations.
  • 3) Results:: 52% and 54% success rates were obtained after displacing a test camera 5cm toward the ground and 4cm away from the door, respectively.
  • 3) Results:: A single-robot policy trained on one robot and one door fails to generalize to an unseen door or different camera positions.

VI. DISCUSSION AND FUTURE WORK

The system enables multiple robots to collaboratively learn one generalizable motor skill through distributed asynchronous policy learning. Its current design assumes physically similar or identical robots, while heterogeneous populations remain a future extension.

  • Discussion: The method extends Guided Policy Search to asynchronous multi-robot training of a shared deep neural network policy.Robots collect experience continuously, add it to a replay buffer, and improve local policies for their task instances.
  • Discussion: Parallelizing policy training with experience collection increases robot utilization while simulated experiments reduce training times.
  • Discussion: Real-world training across different doors improves the generalization capability of a vision-based door-opening policy.
  • Future Work: The method assumes robots can execute the same policy, implicitly requiring physically similar or identical platforms.A proposed extension would separate locally learned private components from globally trained public components for heterogeneous robots.
Loading 1610.00673v1…