PaperlayerView exact revision on arXiv

Source-linked AI summary

Learning Multi-Arm Manipulation Through Collaborative Teleoperation

Albert Tung, Josiah Wong, Ajay Mandlekar, Roberto Martín-Martín, Yuke Zhu, Li Fei-Fei, Silvio Savarese

arXiv:2012.06738v1cs.ROcs.AIcs.LG

TL;DR

Multi-arm imitation learning is challenging because collecting demonstrations can burden a single operator, while many tasks require multiple arms. The paper introduces MART for remote collaborative teleoperation and evaluates a base-residual policy framework, which outperforms fully centralized and fully decentralized baselines on mixed-coordination benchmarks.

  • Problem

    Imitation learning has mostly been limited to single-arm manipulation, while multi-arm demonstration collection can burden operators asked to control multiple arms.

    Source: §I · p. 1

  • Method

    MART enables multiple remote users to teleoperate separate arms, and the paper evaluates a two-step base-residual framework for modeling centralized and decoupled policies.

    Source: §I · pp. 1–2

  • Results

    The base-residual framework outperforms fully centralized and fully decentralized baselines across the benchmark tasks requiring mixed coordination.

    Source: §I · p. 2, §VII · p. 5

  • Takeaways & Limitations

    The authors conclude that multi-arm policies for mixed-coordination tasks benefit from a residual-augmented framework rather than fully centralized or decentralized baselines alone.

    Source: §I · p. 2

Abstract

from arXiv · show

Imitation Learning (IL) is a powerful paradigm to teach robots to perform manipulation tasks by allowing them to learn from human demonstrations collected via teleoperation, but has mostly been limited to single-arm manipulation. However, many real-world tasks require multiple arms, such as lifting a heavy object or assembling a desk. Unfortunately, applying IL to multi-arm manipulation tasks has been challenging -- asking a human to control more than one robotic arm can impose significant cognitive burden and is often only possible for a maximum of two robot arms. To address these challenges, we present Multi-Arm RoboTurk (MART), a multi-user data collection platform that allows multiple remote users to simultaneously teleoperate a set of robotic arms and collect demonstrations for multi-arm tasks. Using MART, we collected demonstrations for five novel two and three-arm tasks from several geographically separated users. From our data we arrived at a critical insight: most multi-arm tasks do not require global coordination throughout its full duration, but only during specific moments. We show that learning from such data consequently presents challenges for centralized agents that directly attempt to model all robot actions simultaneously, and perform a comprehensive study of different policy architectures with varying levels of centralization on our tasks. Finally, we propose and evaluate a base-residual policy framework that allows trained policies to better adapt to the mixed coordination setting common in multi-arm manipulation, and show that a centralized policy augmented with a decentralized residual model outperforms all other models on our set of benchmark tasks. Additional results and videos at https://roboturk.stanford.edu/multiarm .

I. INTRODUCTION

The introduction presents MART for collecting multi-arm demonstrations remotely, then argues that these tasks often mix independent and coordinated stages. It proposes a base-residual policy framework that outperforms fully centralized and fully decentralized baselines on mixed-coordination benchmarks.

I. INTRODUCTION

  • MART lets multiple remote users each teleoperate one arm, addressing the cognitive burden of single operators controlling multiple arms.Users need only a smartphone and web browser; demonstrations covered five novel two- and three-arm tasks collected from users thousands of kilometers apart.Source: §I · p. 1
  • Most multi-arm tasks do not require global coordination throughout; table assembly illustrates independent column assembly followed by coordinated tabletop lifting and alignment.The authors describe the tasks as having mixed coordination across stages.Source: §I · p. 1, Figure 1 · p. 1
  • Centralized agents performed poorly on several tasks, while distributed agents can struggle when multiple arms must synchronize.The authors hypothesize that centralized agents may hallucinate incorrect arm correlations from limited demonstrations.Source: §I · pp. 1–2
  • The base-residual framework outperforms fully centralized and fully decentralized baselines across the mixed-coordination benchmark tasks.The framework is designed to combine centralized and decoupled policy behavior.Source: §I · p. 2
  • The paper introduces realistic benchmarks spanning decoupled to coordinated tasks, including a three-arm task described as the first of its kind.The authors evaluate demonstrations on simulated versions of their tasks against multiple baselines.Source: §I · p. 2

II. RELATED WORK

Related work spans multi-agent reinforcement learning, imitation learning, bimanual manipulation, and teleoperation; behavioral cloning for multi-arm manipulation remains comparatively unexplored.

II. RELATED WORK

  • Multi-agent reinforcement learning can burden exploration, while inverse-reinforcement-learning approaches to multi-agent imitation may require extensive interaction; behavioral cloning instead learns offline from demonstrations.Chernova et al. studied interactive corrections in simple 2D domains.Source: §II · p. 2
  • MART could be used as-is with real-world robots, but COVID measures prevented the authors from accessing their robots for collection.Source: §II · p. 2
  • Centralized and decentralized policy structures have been studied in multi-agent inverse reinforcement learning, but not extensively in behavioral cloning, where multi-arm manipulation remains largely unexplored.Source: §II · p. 2
  • Prior bimanual reinforcement-learning work mainly addressed short-horizon, single-stage tasks, while programming-by-demonstration approaches may rely on modeling assumptions and generalize poorly across configurations.Source: §II · p. 2
  • Existing bimanual teleoperation systems rely on specialized hardware and do not support settings with more than two arms, unlike the multi-user, smartphone-based MART system.Assistive settings have also been studied for bimanual manipulation.Source: §II · pp. 2–3, Figure 2 · p. 2
  • The compared HBC architectures vary which policy components are centralized or decentralized, with residual variants adding small action perturbations to a base policy.Source: Figure 3 · p. 3

III. PRELIMINARIES

The preliminaries formalize manipulation as an MDP and distinguish joint centralized policies from per-arm decentralized policies, then introduce behavioral cloning and its hierarchical variant.

III. PRELIMINARIES

  • Manipulation is modeled as an infinite-horizon discrete-time Markov decision process, with the objective of learning a policy that maximizes expected discounted return.Source: §III · p. 3
  • A centralized agent uses all arms' observations to choose their actions jointly, whereas a decentralized agent chooses each arm's action from that arm's observation alone.Source: §III · p. 3
  • Behavioral cloning fits a policy to demonstrated actions, while hierarchical behavioral cloning uses predicted future observations as subgoals for a subgoal-conditioned low-level policy.The high-level policy predicts future subgoals from the current observation.Source: §III · p. 3

IV. MART: MULTI-ARM ROBOTURK

MART extends RoboTurk's remote teleoperation platform to support multi-arm demonstrations, using smartphone control and browser video streams for geographically separated users.

A. RoboTurk Overview

  • RoboTurk supports remote collection of real or simulated task demonstrations through low-latency teleoperation, with users controlling a robot end effector using a smartphone as a 6-DoF motion controller.Source: A. RoboTurk Overview · p. 3
  • The platform uses WebRTC links among the user's browser, smartphone, and teleoperation server to provide low-latency video streaming and control.Source: A. RoboTurk Overview · p. 3
  • Each teleoperation server serves one user and robot simulator, receiving phone commands and sending rendered workspace frames to the user's browser.Source: A. RoboTurk Overview · pp. 3–4

IV. MART: MULTI-ARM ROBOTURK

  • MART's five multi-arm tasks are paired with agent-specific teleoperator viewpoints and span varying coordination demands.Source: Figure 4 · p. 4

B. Extending RoboTurk for Collaborative Teleoperation

Collaborative teleoperation extends the server to coordinate multiple users and robot arms, with command synchronization addressing variable network latency.

B. Extending RoboTurk for Collaborative Teleoperation

  • The extended server maintains two communication channels per user, receives each user's phone commands, and sends workspace-camera views tailored to each user's viewpoint.Source: B. Extending RoboTurk for… · p. 4
  • The server waits for new phone messages from all users before actuating the arms and advancing simulation, helping users perceive real-time operation despite network latency.Source: B. Extending RoboTurk for… · p. 4

V. LEARNING MIXED COORDINATION

Most multi-arm tasks require global coordination only during particular segments, making policies that model all arms jointly vulnerable to overfitting on non-coordinated portions. The section presents partially decentralized and residual policy architectures to address mixed coordination.

V. LEARNING MIXED COORDINATION

  • Centralized policies can overfit sections that do not require coordination, motivating architectures that combine centralized and decentralized components.The paper frames this as a challenge of learning from mixed-coordination demonstrations.Source: §V · p. 4
  • Partial decentralization factorizes either the high-level or low-level policy, between centralized HBC and fully decentralized d-HBC.dh-HBC decentralizes high-level subgoal policies while keeping the low-level policy centralized; dl-HBC does the reverse.Source: §V · p. 4
  • Residual learning adds a correction from a separate network to a pretrained policy’s action, pairing complementary architectures to mitigate base-policy pitfalls.The correction is constrained to have L2 norm smaller than ε so it does not dominate the overall policy behavior.Source: §V · p. 4
  • The two residual variants pair a decentralized HBC base with a centralized residual (r-HBC), or a centralized base with a decentralized residual (rd-HBC).Source: §V · p. 4

VI. EXPERIMENTAL SETUP

The experimental setup uses five MuJoCo multi-arm tasks spanning independent, coordinated, and mixed-coordination subtasks. Demonstrations come from experienced users and additional multi-group user studies.

VI. EXPERIMENTAL SETUP

  • All tasks use MuJoCo and robosuite, with operational-space arm control and observations of end-effector pose plus task-specific object information.For decentralized setups, state information is partitioned by relevance to each agent.Source: §VI · p. 5
  • The two pedagogical two-arm tasks contrast independent block lifting with coordinated tray lifting that must keep drinks from tipping for 1.5 seconds.Source: §VI · p. 5
  • Assembly and pick-place handover are multi-stage two-arm tasks with subtasks requiring different levels of coordination.Assembly combines independently assembled columns with coordinated tabletop alignment; handover combines object retrieval, bin clearing, and a final transfer.Source: §VI · p. 5
  • The three-arm lift-wiping task requires two arms to move a drink tray together while a third arm wipes the exposed tabletop in parallel.Source: §VI · p. 5
  • Experienced-user demonstrations cover all five tasks, with roughly 50–100 successful demonstrations per user-study dataset.The study included three user pairs for two-arm tasks and two groups of three for the three-arm task.Source: §VI · p. 5

VII. RESULTS

The results section reports that the residual framework outperforms the baseline models across the multi-arm tasks and that users can coordinate effectively using MART.

VII. RESULTS

  • Users can effectively coordinate using MART.Source: §VII · p. 5
  • Table I frames the operator analysis in terms of marginal success-rate degradation by subtask type.Source: Table I · p. 5
  • The residual framework outperforms all baseline models across the multi-arm tasks.Source: §VII · p. 5

A. System Analysis: Do operators have a hard time with coordination?

Operator completion rates show that coordination was not the main source of degradation in the two-arm tasks, while the three-arm task had slightly greater degradation on its coordinated subtask. These results support MART for collecting collaborative demonstrations across geographically separated operators.

A. System Analysis: Do operators have a hard time with coordination?

B. Data Analysis

Residual-augmented policies outperform the reported baselines across tasks, while conventional and partially decentralized policies show mixed results. The residual framework also performs well with lower-quality demonstrations, though the authors note room to improve on challenging tasks.

B. Data Analysis

  • The evaluation reports each model’s best-checkpoint rollout success rate as the mean and standard deviation across five random seeds.All models are evaluated on experienced-user demonstrations; a subset is also evaluated on user-study demonstrations.Source: B. Data Analysis · p. 5
  • Results vary by task: d-HBC leads Multi-Cube Lifting (84.5% vs. 38.5%) and Handover (24.4% vs. 16.0%), while HBC leads Three Arm Lifting Wiping (83.7% vs. 50.0%); both reach about 5% on Assembly.Neither HBC nor d-HBC consistently outperforms the other on the more challenging multi-stage tasks.Source: B. Data Analysis · pp. 5–6, Table II · p. 6
  • Naive partially decentralized variants perform at best marginally better than the weakest HBC baseline and sometimes worse than both HBC baselines, including on Drink-Tray Lifting and Pick-Place Handover.Reported performance is below 70% on Drink-Tray Lifting and below 16% on Pick-Place Handover.Source: B. Data Analysis · p. 6
  • r-HBC and rd-HBC consistently outperform all baselines across tasks; rd-HBC leads short-horizon Drink-Tray Lifting (86.7% vs. 75.3%), while r-HBC leads multi-stage Lifting Wiping (94.0% vs. 58.6%).The authors attribute the difference in best-performing base policy to the importance of inductive bias.Source: B. Data Analysis · p. 6, Table II · p. 6
  • Under lower-quality demonstrations, r-HBC matches or exceeds other baselines: 17.3% vs. 9.3% on Pick-Place Handover and 86.7% vs. 71.3% on Lifting Wiping.The authors describe this as robustness to noisy training signals and a diverse distribution of demonstrations.Source: B. Data Analysis · pp. 5–6, Table III · p. 6
  • The residual framework still has room to improve, especially on the more challenging Assembly and Pick-Place Handover tasks.The authors identify collecting further data with MART as a way to explore these multi-arm properties.Source: B. Data Analysis · p. 6

VIII. CONCLUSION

The paper introduces MART for collecting remote multi-arm demonstrations and evaluates imitation-learning policies on novel tasks with varying collaboration demands. It finds that residual-augmented policies outperform the other tested baselines on those tasks.

VIII. CONCLUSION

  • MART is a scalable teleoperation system for gathering real-time demonstrations on realistic, novel multi-arm tasks requiring varying degrees of collaboration.The paper reports that imitation-learning methods can use this data to train performant policies across those tasks.Source: §VIII · p. 6
  • A residual-augmented policy framework outperforms the other tested baselines on the paper’s multi-arm tasks.The authors present it as an approach to modeling policies for mixed coordination.Source: §VIII · p. 6
  • The authors identify MART’s potential to lower the barrier to collecting multi-arm demonstrations and support further research in this setting.The paper links the difficulty of collecting demonstrations to the prior limitations of imitation learning for multi-arm manipulation.Source: §VIII · p. 6

Related papers

Loading 2012.06738v1…