PaperlayerView exact revision on arXiv

Source-linked AI summary

Human-in-the-Loop Imitation Learning using Remote Teleoperation

Ajay Mandlekar, Danfei Xu, Roberto Martín-Martín, Yuke Zhu, Li Fei-Fei, Silvio Savarese

arXiv:2012.06733v1cs.ROcs.AIcs.LG

TL;DR

Offline imitation learning for manipulation is constrained by demonstration costs and can fail when action errors lead policies into unseen states, especially at precise bottlenecks. The paper combines remote 6-DoF human intervention with Intervention Weighted Regression, reporting better performance than full demonstrations and intervention-learning alternatives across threading and coffee tasks.

  • Problem

    Human demonstrations are costly, and offline imitation policies can encounter unseen states after action errors; human relabeling is burdensome for realistic continuous-control tasks.

    Source: §I · p. 1

  • Method

    The authors build a remote 6-DoF intervention system and train with Intervention Weighted Regression, using intervention timing to prioritize bottleneck-traversal actions.

    Source: §I · p. 2, C. Intervention Weighted Regression · p. 5

  • Results

    Across threading and coffee tasks, the method outperforms equivalent full demonstrations and intervention-learning alternatives; on Threading, performance is 87.3% vs. 76.7% for Full Demos.

    Source: D. Experiments · p. 6

  • Takeaways & Limitations

    Intervention timing identifies bottleneck locations and traversal actions, and reweighting the data lets the method leverage both intervention and non-intervention samples.

    Source: C. Intervention Weighted Regression · p. 5, §VI · p. 6

Abstract

from arXiv · show

Imitation Learning is a promising paradigm for learning complex robot manipulation skills by reproducing behavior from human demonstrations. However, manipulation tasks often contain bottleneck regions that require a sequence of precise actions to make meaningful progress, such as a robot inserting a pod into a coffee machine to make coffee. Trained policies can fail in these regions because small deviations in actions can lead the policy into states not covered by the demonstrations. Intervention-based policy learning is an alternative that can address this issue -- it allows human operators to monitor trained policies and take over control when they encounter failures. In this paper, we build a data collection system tailored to 6-DoF manipulation settings, that enables remote human operators to monitor and intervene on trained policies. We develop a simple and effective algorithm to train the policy iteratively on new data collected by the system that encourages the policy to learn how to traverse bottlenecks through the interventions. We demonstrate that agents trained on data collected by our intervention-based system and algorithm outperform agents trained on an equivalent number of samples collected by non-interventional demonstrators, and further show that our method outperforms multiple state-of-the-art baselines for learning from the human interventions on a challenging robot threading task and a coffee making task. Additional results and videos at https://sites.google.com/stanford.edu/iwr .

I. INTRODUCTION

The paper presents remote intervention-based learning for precise 6-DoF manipulation, where offline imitation policies can fail at bottlenecks. Its system and Intervention Weighted Regression use operator interventions to improve bottleneck traversal, with reported gains over full demonstrations and intervention-learning alternatives.

I. INTRODUCTION

  • Offline imitation learning is limited by human demonstration costs, and small action errors can lead policies into unseen states.Source: §I · p. 1
  • Unlike tolerant 2D driving, manipulation bottlenecks require precise action sequences, where small deviations can separate success from failure.Source: §I · p. 1
  • The authors develop a remote system for operators to monitor policies and take over 6-DoF end-effector control when needed.Source: §I · p. 2
  • On threading and coffee tasks, the method outperforms equivalent full demonstrations and intervention-learning alternatives, including across data from multiple operators.Source: §I · p. 2
  • Intervention Weighted Regression uses intervention timing to prioritize actions that demonstrate bottleneck traversal while retaining non-intervention samples.Source: §I · p. 2

II. RELATED WORK

Prior work spans offline imitation, relabeling-based learning, and intervention methods, but practical relabeling is burdensome for human continuous-control tasks. This work emphasizes remote intervention in contact-rich manipulation and retains on-policy samples while prioritizing intervention data.

II. RELATED WORK

  • Behavioral Cloning can require prohibitive expert samples, while Inverse Reinforcement Learning can require extensive agent-environment interaction.Source: §II · p. 2
  • DAGGER uses expert relabeling of agent-collected states, but human relabeling in continuous 6D control can be burdensome and error-prone.Source: §II · p. 2
  • The system is presented as the only prior intervention system with remote web-based operation and a demonstration on contact-rich manipulation tasks.Source: §II · p. 2
  • Earlier intervention approaches discard agent samples, whereas this method retains them for regularization and prioritizes interventions that show successful bottleneck traversal.Source: §II · pp. 2–3

III. PRELIMINARIES

The preliminaries formalize manipulation as an MDP and review behavioral cloning and DAGGER. They then describe intervention-based data collection and motivate retaining on-policy samples alongside interventions.

III. PRELIMINARIES

  • The task is modeled as an infinite-horizon, discrete-time MDP with states, actions, transitions, rewards, discount factor, and an initial-state distribution.Source: §III · p. 3
  • Behavioral Cloning trains a policy to match demonstration actions, while DAGGER trains on states induced by the current policy and demonstrator labels.Source: §III · p. 3
  • HG-DAGGER collects samples under a mixture of human and agent control, uses intervention samples as relabels, and discards on-policy agent samples.Source: §III · p. 3
  • The proposed alternative includes on-policy agent samples for regularization while prioritizing interventions that demonstrate successful bottleneck traversal.Source: §III · p. 3

IV. LEVERAGING REMOTE HUMAN INTERVENTIONS

The system lets remote operators monitor policy execution and intervene through 6-DoF teleoperation, collecting intervention data for learning. Its setup is described for simulated tasks, with real-robot access unavailable during the work.

A. Remote Teleoperation System

  • Remote users monitor robot video through a website and control the robot with a smartphone; the simulations run on cloud servers.The platform supports simulated and real robots, but this work focuses on simulated domains.Source: A. Remote Teleoperation System · p. 3
  • Users can pause or resume policy execution and intervene by moving their phone to issue relative pose commands to the arm.The interface uses separate smartphone buttons for pausing or resuming and for intervention.Source: A. Remote Teleoperation System · p. 3
  • Each demonstration combines human and policy samples, recorded with binary intervention labels for imitation learning.Source: A. Remote Teleoperation System · pp. 3–4
  • The system can be applied to real-world robots, but the authors could not access their robot arms because of COVID measures.Source: A. Remote Teleoperation System · p. 3
  • Threading has grasping and insertion bottlenecks, while the coffee-machine task adds a lid-closing bottleneck.Source: Figure 4 · p. 4

B. Intervention-Based Policy Learning

The section frames intervention-based learning as alternating optimization between a human-curated data distribution and a policy. Each round collects intervention data and updates the policy by behavioral cloning on that distribution.

B. Intervention-Based Policy Learning

C. Intervention Weighted Regression

Intervention Weighted Regression uses intervention timing to prioritize data around manipulation bottlenecks while retaining on-policy samples. This weighting is intended to reinforce actions that successfully traverse bottlenecks.

C. Intervention Weighted Regression

  • Unlike HG-DAGGER, which discards non-intervention samples, the method retains both intervention and on-policy samples to reduce policy shifts and new covariate-shift failures.Source: C. Intervention Weighted Regression · pp. 4–5
  • Intervention timings identify bottleneck locations and successful traversal actions, which should be reinforced over other actions in a trajectory.The method assumes offline-trained policies are most likely to err near bottlenecks.Source: C. Intervention Weighted Regression · p. 5
  • Intervention Weighted Regression re-weights the dataset distribution, assigning intervention samples a prioritization weight α relative to non-intervention samples.Source: C. Intervention Weighted Regression · p. 5
  • Choosing α = |DR|/|DI| makes q sample intervention and non-intervention state-action distributions in equal proportion.Source: C. Intervention Weighted Regression · p. 5

V. EXPERIMENTS

The experiments evaluate the system and method on challenging contact-rich manipulation tasks requiring precise control. They aim to compare intervention-collected data with an equivalent amount of full demonstrations and IWR with alternative methods.

V. EXPERIMENTS

  • The evaluation targets challenging contact-rich manipulation tasks that require precise control.Source: §V · p. 5
  • One objective is to test whether system-collected data can outperform an equivalent amount of full human demonstration trajectories.Source: §V · p. 5
  • The other objective is to compare IWR with alternatives for learning from intervention data.Source: §V · p. 5

A. Tasks

The experiments use two contact-rich manipulation tasks with bottlenecks that demand precise actions: rod insertion into a ring and pod insertion and lid closing on a coffee machine.

A. Tasks

  • Both tasks use a Sawyer arm in a table workspace, controlled with an Operational Space Controller in MuJoCo and robosuite.Source: A. Tasks · p. 5
  • Threading requires picking up a wooden rod and inserting it into a ring, with randomized object locations and bottlenecks at grasping and insertion.The ring can move if struck by the rod.Source: A. Tasks · p. 5
  • The Coffee Machine task requires picking up a pod, inserting it into a holder, and closing the lid, with randomized pod location.Its three bottlenecks are grasping, insertion, and closing; errors can make the pod slip or prevent insertion.Source: A. Tasks · p. 5

B. Baselines

The baselines vary how intervention and on-policy samples are used, while the experiments compare intervention-based collection with full demonstrations under matched human-sample budgets.

B. Baselines

  • IWR trains by behavioral cloning on equally sampled intervention and on-policy datasets; IWR-NB combines samples in one unbalanced dataset, while HG-DAGGER discards policy samples.Full Demos instead trains on full human-demonstration trajectories.Source: B. Baselines · p. 5

C. Experiment Details

  • The Threading study used one operator, while the Coffee Machine study used three operators.Source: C. Experiment Details · p. 5
  • Each operator began with 30 demonstrations and a trained base policy; intervention methods collected three rounds, targeting intervention samples equal to roughly 33% of the initial dataset per round.Full Demos collected one trajectory dataset matching the initial dataset's sample count.Source: C. Experiment Details · p. 5
  • Policies were two-layer LSTMs with hidden size 100, trained on sequences of length 10 and evaluated over 50 rollouts per checkpoint.Reported success rates are each run's maximum average across checkpoints, over three random-seed runs.Source: C. Experiment Details · pp. 5–6

D. Experiments

Across Threading and Coffee Machine experiments, the proposed method outperformed full demonstrations and intervention-learning baselines; it also performed well on datasets collected by other methods.

D. Experiments

  • 87.3% vs. 76.7%: on Threading, the proposed method significantly outperformed Full Demos, and it consistently beat HG-DAGGER and IWR-NB across rounds.The comparison is reported for the single-operator Threading task.Source: D. Experiments · p. 6
  • 87.5% vs. 69.6% vs. 64.9%: on Coffee Machine, the method's average final success exceeded HG-DAGGER and Full Demos across three operators.The reported improvements over the original base policy were over 35%, about 18%, and about 13%, respectively.Source: Table II · p. 6, D. Experiments · p. 6
  • On datasets collected by other methods, the proposed method consistently outperformed their baselines and approached performance on its own collected dataset.The authors suggest the baselines' gap reflects how they leverage data, not only data quality or base-policy quality.Source: D. Experiments · p. 6, Table IV · p. 6
  • A single large intervention-data round yielded no significant Threading difference and about 7% lower average Coffee Machine success than iterative collection.The authors suggest longer tasks with more bottlenecks may benefit more from iteration.Source: D. Experiments · p. 6

VI. CONCLUSION

The paper presents a remote intervention system and a method that prioritizes bottleneck traversal when learning from intervention data. Its results support this approach across tasks, operators, and datasets collected by other methods.

VI. CONCLUSION

  • The system lets remote operators monitor trained policies and intervene when necessary to help complete tasks.It is designed for remote users with 6-DoF control of robot end-effectors.Source: §VI · p. 6
  • The method reshapes data distribution to prioritize bottleneck traversal using the timing of human interventions.Source: §VI · p. 6
  • The method outperformed intervention-based baselines and equivalent full-demonstration data, with results across operators and other methods' datasets.The authors identify crowdsourced settings as a potential application and plan physical-arm data collection as future work.Source: §VI · p. 6
Loading 2012.06733v1…