PaperlayerView exact revision on arXiv

Source-linked AI summary

Robot Learning on the Job: Human-in-the-Loop Autonomy and Learning During Deployment

Huihan Liu, Soroush Nasiriany, Lance Zhang, Zhiyao Bao, Yuke Zhu

arXiv:2211.08416v3cs.ROcs.AIcs.LG

TL;DR

Sirius addresses selective learning from shifting, multimodal, and suboptimal human-robot deployment data. It uses human interventions to guide weighted behavioral cloning and reports higher policy performance in simulation and on real hardware.

  • Problem

    Learning from shifting, multimodal, and suboptimal deployment data requires selecting useful examples while avoiding copying actions that could lead to failure.

    Source: §I · p. 2

  • Method

    Sirius uses human interventions as trust signals to guide weighted behavioral cloning on deployment data.

    Source: §I · p. 2, §VI · p. 9

  • Results

    Sirius reports an 8% boost in policy performance in simulation and 27% on real hardware over state-of-the-art methods, and significantly outperforms evaluation baselines.

    Source: §I · p. 2, C. Experiment Results · p. 7

  • Takeaways & Limitations

    Sirius combines reliable human-robot task execution with improved autonomous policy performance over time.

    Source: §VI · p. 9

Abstract

from arXiv · show

With the rapid growth of computing powers and recent advances in deep learning, we have witnessed impressive demonstrations of novel robot capabilities in research settings. Nonetheless, these learning systems exhibit brittle generalization and require excessive training data for practical tasks. To harness the capabilities of state-of-the-art robot learning models while embracing their imperfections, we present Sirius, a principled framework for humans and robots to collaborate through a division of work. In this framework, partially autonomous robots are tasked with handling a major portion of decision-making where they work reliably; meanwhile, human operators monitor the process and intervene in challenging situations. Such a human-robot team ensures safe deployments in complex tasks. Further, we introduce a new learning algorithm to improve the policy's performance on the data collected from the task executions. The core idea is re-weighing training samples with approximated human trust and optimizing the policies with weighted behavioral cloning. We evaluate Sirius in simulation and on real hardware, showing that Sirius consistently outperforms baselines over a collection of contact-rich manipulation tasks, achieving an 8% boost in simulation and 27% on real hardware than the state-of-the-art methods in policy success rate, with twice faster convergence and 85% memory size reduction. Videos and more details are available at https://ut-austin-rpl.github.io/sirius/

I. INTRODUCTION

Sirius addresses the gap between impressive robot-learning demonstrations and robust real-world deployment by pairing monitored robot autonomy with human intervention and continual policy learning. It uses deployment feedback to improve policies, and is presented as a complement to prior shared-autonomy and human-in-the-loop learning approaches.

I. INTRODUCTION

  • Learning systems show promise in research settings but still lack the generalization and robustness needed for widespread real-world deployment.Their failures, failure circumstances, and paths to continual improvement remain opaque to potential practitioners.Source: §I · p. 1
  • Sirius combines robot execution, human monitoring and teleoperation interventions, and policy improvement from deployment data across consecutive learning rounds.The framework aims for reliable task execution while continually improving autonomy and reducing human workload.Source: Figure 1 · p. 1, §I · p. 1
  • Human interventions can reveal low-trust situations and task structure, making deployment feedback informative for policy learning.The framework uses interventions to assess the importance of training data and weighted behavioral cloning to train on differently weighted samples.Source: §I · p. 2
  • With 15% of the full memory size, Sirius’s memory strategy retains the same or better performance and enables three times faster convergence.The strategy adaptively adds and removes samples from fixed-size memory for policy updates between deployment rounds.Source: §I · p. 2
  • 8% in simulation and 27% on real hardware: Sirius reports higher policy performance than state-of-the-art methods.The comparisons include methods learning from offline data and interactive imitation learning.Source: §I · p. 2

II. RELATED WORK

  • Unlike shared-autonomy methods that do not learn from interventions, Sirius uses human control feedback in deployment to improve learning; it also derives weights from interventions rather than task rewards.Prior shared autonomy focuses on collaborative control, while advantage-weighted offline RL uses learned value estimates to set weights.Source: §II · pp. 2–3

III. BACKGROUND AND OVERVIEW

The formulation models robot actions that a human may override, with objectives to improve robot autonomy and minimize human workload. The background describes behavioral cloning and weighted variants, including intervention-based weighting as an alternative to value-based offline reinforcement learning.

A. Problem Formulation

  • The human-robot team policy selects the human policy during intervention and the robot policy otherwise, while the objectives are to maximize robot reward and minimize human interventions.The intervention indicator is binary, and the human policy is implicit.Source: A. Problem Formulation · p. 3

B. Weighted Behavioral Cloning Methods

  • Standard behavioral cloning learns a robot policy by maximizing the likelihood of dataset actions conditioned on states.Weighted behavioral cloning scales each sample’s log-likelihood by a weight that assigns sample importance.Source: B. Weighted Behavioral Cloning Methods · p. 3
  • Offline RL commonly uses advantage-based weights to prioritize high-advantage samples, but value learning can be challenging when the dataset lacks broad state-action coverage.The authors report that deployment data, consisting of successful trajectories, make current offline RL methods struggle to learn values.Source: B. Weighted Behavioral Cloning Methods · p. 3
  • Sirius builds on human-in-the-loop weighted cloning but derives sample weights from human intervention signals rather than learned values.Prior work includes Intervention-weighted Regression, which bases weights on whether a sample is a human intervention.Source: B. Weighted Behavioral Cloning Methods · p. 3

IV. SIRIUS: HUMAN-IN-THE-LOOP LEARNING AND DEPLOYMENT

Sirius couples monitored robot deployment with concurrent policy updates from accumulated human and robot data. Human intervention supports task execution, while the growing mixed-source dataset creates learning and memory challenges.

A. Human-in-the-loop Deployment Framework

  • Sirius combines robot deployment under human monitoring with policy updates that use deployment data to improve the policy for later execution.The system runs deployment and policy update as simultaneous threads.Source: A. Human-in-the-loop Deployment… · pp. 3–4, §IV · p. 3
  • Human operators can take over when needed, and human-gated control is used to support successful, trustworthy execution in real-world deployment.The initial policy is trained on a small number of human demonstrations before deployment begins.Source: A. Human-in-the-loop Deployment… · p. 4
  • Each round deploys the current best policy while training from prior data, then appends newly collected data and trains the next policy.The memory buffer aggregates data across rounds.Source: A. Human-in-the-loop Deployment… · p. 4
  • Long-term deployment produces mixed robot, intervention, and demonstration data that continually grows, creating memory burden and learning inefficiency.The section introduces weighting and memory management as responses to these challenges.Source: §IV · p. 3, A. Human-in-the-loop Deployment… · p. 4

B. Human-in-the-loop Policy Learning

Sirius uses human interventions as signals of trust and behavior quality to reweight mixed deployment data for weighted behavioral cloning. The scheme distinguishes four data classes, increasing intervention influence and reducing the impact of samples preceding interventions.

B. Human-in-the-loop Policy Learning

  • The weighting scheme treats human interventions as valuable feedback about trust and robot behavior, using them to prioritize critical states and reduce the influence of lower-quality preceding samples.Samples before intervention are treated as less desirable because intervention indicates the robot may have made mistakes or acted suboptimally.Source: B. Human-in-the-loop Policy Learning · p. 4
  • Human interventions are upweighted because they occur where the robot needs help, while preceding robot samples are downweighted as lower-quality data.The approach aims to exploit both which human samples occur and when interventions take place.Source: B. Human-in-the-loop Policy Learning · p. 4
  • Four classes distinguish demonstrations, interventions, robot actions, and pre-intervention segments, which mark robot samples preceding human correction.The pre-intervention classification uses the human partner’s implicit evaluation to distinguish ordinary robot samples from suboptimal ones.Source: B. Human-in-the-loop Policy Learning · pp. 4–5, Figure 3 · p. 4
  • The learning workflow collects rollout episodes, separates samples by class, obtains target class frequencies, and trains the policy with weighted behavioral cloning.The algorithm also invokes a memory-management strategy when the buffer exceeds its limit.Source: B. Human-in-the-loop Policy Learning · pp. 4–5
  • Weighted behavioral cloning assigns each class a target frequency and weights samples by the ratio of target to observed class frequency.The method changes the class distribution by increasing intervention samples and decreasing pre-intervention samples.Source: B. Human-in-the-loop Policy Learning · p. 5

C. Memory Management

Sirius examines fixed-size memory management because deployment data grows over time and can slow training and consume memory. It compares strategies that discard data according to intervention frequency, recency, or random selection.

C. Memory Management

  • With continued deployment, larger datasets slow training convergence and consume memory, motivating tests of whether selective forgetting preserves useful information.The stated setting assumes fixed-size storage that replaces existing samples when full.Source: C. Memory Management · p. 5
  • The memory study compares five replacement strategies: LFI, MFI, FIFO, FILO, and Uniform, each encoding a different hypothesis about which experiences to preserve.The strategies reject samples from least- or most-intervened trajectories, oldest or newest data, or uniformly at random.Source: C. Memory Management · pp. 5–6
  • LFI preserves frequently intervened trajectories, whereas MFI favors trajectories with fewer interventions; the alternatives prioritize recency, early data diversity, or uniform selection.These are the hypotheses motivating the compared rejection rules, not reported outcomes.Source: C. Memory Management · p. 6

D. Implementation Details

The implementation uses BC-RNN for vision-based control, sets intervention and pre-intervention class proportions explicitly, and chooses the pre-intervention segment length based on observed human reaction time.

D. Implementation Details

  • BC-RNN is the policy backbone, combining workspace and eye-in-hand image features with robot proprioception to produce a Gaussian Mixture Model action distribution.ResNet-18 encoders process the third-person and eye-in-hand images.Source: D. Implementation Details · p. 6
  • Interventions receive a target proportion of 50%, pre-intervention samples receive zero weight, and demonstrations retain their observed dataset proportion.The robot class proportion adjusts accordingly; the passage contrasts this with IWR’s treatment of non-intervention samples as one class.Source: D. Implementation Details · p. 6
  • The pre-intervention segment is set to 15 robot actions, corresponding roughly to the operator’s observed average reaction time of 2 seconds.The segment length is tied to human reaction time.Source: D. Implementation Details · p. 6

V. EXPERIMENTS

Experiments cover two simulated and two real-world contact-rich manipulation tasks, assessing policy performance and human workload across deployment rounds and comparing against learning baselines.

A. Tasks

  • Simulation uses Nut Assembly and Tool Hang, two contact-rich manipulation tasks selected from the robomimic benchmark.Nut Assembly requires inserting a square nut into a column; Tool Hang requires inserting a hook into a small hole and hanging a wrench.Source: A. Tasks · p. 6
  • The real-world tasks are Gear Insertion and Coffee Pod Packing, representing industrial assembly and food packaging applications.Gear Insertion places two gears onto shafts; Coffee Pod Packing involves opening a drawer, placing a pod in its holder, and closing the drawer.Source: A. Tasks · p. 6

V. EXPERIMENTS

  • Across simulated and real-world tasks, policies improve over rounds, and our model has the highest performance in all four tasks after three deployment and policy-update rounds.Fig. 5 compares against human-in-the-loop learning, imitation learning, and offline reinforcement learning baselines.Source: Figure 5 · p. 7

B. Baselines and Evaluation Protocol

  • Baselines include IWR, BC-RNN, and IQL; BC-RNN and IQL use deployment data generated by our method for comparison.IQL is tested with intervention-guided rewards and sparse rewards, and unlike our method it requires task-reward information.Source: B. Baselines and Evaluation Protocol · p. 7
  • The comparison follows three policy-learning and deployment rounds, with each deployment ending when intervention samples reach one third of the initial human-demonstration samples.Source: B. Baselines and Evaluation Protocol · p. 7
  • Policy success rate and intervention percentage measure performance and human workload; simulation uses three seeds and 100 trials per checkpoint, while real-world tests use one seed and 32 trials at the selected checkpoint.For real-world evaluation, five checkpoints are initially screened with five trials each before the best-performing checkpoint receives 32 trials.Source: B. Baselines and Evaluation Protocol · p. 7

C. Experiment Results

Across tasks, Sirius outperforms baselines; analyses also report benefits from fine-grained sample weighting, compact memory, and decreasing human intervention.

  • Sirius consistently outperforms IWR over deployment rounds and significantly outperforms BC-RNN and IQL under the same dataset distribution.Source: C. Experiment Results · p. 7
  • LFI matches or improves on Base while using less storage, and its policy converges twice as fast on both tasks, reaching 90% success in fewer epochs.Source: Figure 7 · p. 8, C. Experiment Results · p. 8
  • Removing any demo, intervention, or pre-intervention class weight hurts performance on Round 1 simulation data from the initial BC-RNN policy.Source: C. Experiment Results · p. 8
  • Policy performance peaks at P*(intv) = 0.5 and is worse when the intervention reweighting ratio is higher or lower.Source: C. Experiment Results · p. 8
  • Intervention-sample ratios decline over rounds more with Sirius than IWR; in Gear Insertion, Round 3 requires little and shorter human intervention than Round 0.Source: Figure 8 · p. 8, C. Experiment Results · pp. 8–9, Figure 9 · p. 9
  • Capping memory at 500 trajectories reduces the original dataset size to 15% for Nut Assembly and 30% for Tool Hang.Source: C. Experiment Results · p. 8
  • LFI retains the most human-intervention samples and frequently intervened trajectories, which the authors associate with broader state coverage and rare-event handling.Source: C. Experiment Results · p. 8
  • The human-in-the-loop experiments use one operator per task, require constant monitoring, and measure workload only by intervention percentage.Source: C. Experiment Results · p. 9

VI. CONCLUSION

Sirius combines human-robot teaming with intervention-based weighted behavioral cloning and continuous deployment under memory constraints. The paper identifies reducing human monitoring and teleoperation burden as future work.

VI. CONCLUSION

  • Sirius is presented as guaranteeing reliable task execution while improving autonomous policy performance over time.Its intervention-based weighted behavioral cloning uses deployment data, and its system trains and deploys models continuously under memory constraints.Source: §VI · p. 9
  • The conclusion identifies more intuitive control interfaces and faster policy learning from human feedback as future directions.It also proposes deployment monitoring that automatically detects robot errors without constant human supervision.Source: §VI · p. 9
  • Reducing the human cognitive burden of monitoring and teleoperating the system remains an open research direction.Source: §VI · p. 9

VII. APPENDIX

The appendix describes four manipulation tasks and a teaming setup in which the robot acts by default while a human monitors and can take over. Human control uses a SpaceMouse interface.

A. Task Details

  • The task suite includes Nut Assembly and Tool Hang from robomimic, plus Gear Insertion and Coffee Pod Packing designed for this work.The tasks require precise grasping, orientation, insertion, or other dexterous manipulation; the objects are initialized randomly in position and rotation.Source: A. Task Details · p. 11
  • Nut Assembly bottlenecks on orienting a square rod correctly for insertion, while Tool Hang requires precise hook insertion and wrench placement.Source: A. Task Details · p. 11
  • Gear Insertion requires sequentially placing two gears onto shafts, and Coffee Pod Packing requires placing pods into fitted holes and operating a drawer.Source: A. Task Details · p. 11

B. Human-Robot Teaming

  • The robot executes by default under human supervision; when undesirable behavior occurs, the human takes control and returns control when the robot can continue.For example, the human may intervene when a gear gets stuck.Source: B. Human-Robot Teaming · p. 11, Figure 10 · p. 11
  • A 6-DoF SpaceMouse sends position and orientation commands, with a button to pause monitoring and direct control until the robot is back in an acceptable state.Source: B. Human-Robot Teaming · p. 11

C. Observation and Action Space

The appendix specifies image and proprioceptive observations, task-specific action spaces, and common policy implementations. For evaluation, policy updates follow fixed rounds tied to the number of collected intervention samples.

C. Observation and Action Space

  • Observations comprise workspace-camera and eye-in-hand-camera images plus low-dimensional proprioception; simulation uses a 7-dimensional OSC action space, whereas real-world tasks use a 5-dimensional OSC yaw-controller action space.For Tool Hang, image size is 128×128 rather than robomimic ’s default 224×224 for training efficiency.Source: C. Observation and Action Space · p. 11

D. Method Implementations

  • Methods share robomimic-based architectures and training procedures, while the IQL baseline receives a denser intervention-informed reward.The IQL rewards are 1.0 for success, 0.25 for intervention states, −0.25 for pre-intervention states, and 0 otherwise.Source: D. Method Implementations · p. 12

E. HITL System Policy Updates

  • For consistent comparison, policies update in three rounds, each after intervention samples reach one-third of the human demonstration samples.The appendix notes that policy update rules are themselves a human-in-the-loop design dimension for future exploration.Source: E. HITL System Policy Updates · p. 12
  • Across round updates, the method reduces average intervention frequency and length more than the comparison methods.Source: Figure 11 · p. 12

F. Human Workload Reduction

In Nut Assembly, intervention frequency and length decrease over rounds, with the method reducing both faster. These measures reflect both workload and human trust in the robot.

F. Human Workload Reduction

  • Average intervention frequency counts interventions per rollout and reflects control switches, while average intervention length counts timesteps per intervention.More context switches and longer interventions correspond to greater human concentration or mental workload.Source: F. Human Workload Reduction · p. 12
  • Lower intervention frequency and shorter intervention length reflect greater human trust, because operators intervene at fewer places and return control sooner.Source: F. Human Workload Reduction · p. 12
  • The intervention ratio, average intervention frequency, and average intervention length decrease over rounds, with faster reductions in frequency and length for the method.The reported example is Nut Assembly.Source: F. Human Workload Reduction · p. 12
Loading 2211.08416v3…