PaperlayerView exact revision on arXiv

Source-linked AI summary

Model-Based Runtime Monitoring with Interactive Imitation Learning

Huihan Liu, Shivin Dass, Roberto Martín-Martín, Yuke Zhu

arXiv:2310.17552v1cs.ROcs.AIcs.LG

TL;DR

Interactive imitation learning supports continual learning during robot deployment, but existing approaches often require continuous human supervision. This paper adds a model-based runtime monitor trained with deployment and intervention data to anticipate errors, reporting higher success and more efficient human-workload utilization than state-of-the-art interactive imitation-learning baselines.

  • Problem

    Interactive imitation learning supports continual improvement during deployment, but existing approaches typically require continuous human monitoring and imminent feedback, creating high human workload.

    Source: §I · p. 1

  • Method

    A model-based runtime monitor predicts future policy outcomes and uses human interventions to continually train an error predictor without requiring explicit failure examples.

    Source: §I · pp. 1–2

  • Results

    The method achieved on average 32% higher success rates and 16% more efficient human-workload utilization than state-of-the-art interactive imitation-learning baselines across two simulated and two real-world tasks.

    Source: §I · p. 2

  • Takeaways & Limitations

    The monitor is integrated into human-in-the-loop interactive imitation learning to support trustworthy deployment and policy learning over long-term deployments.

    Source: §V · p. 6

    Experiments used one human subject per task and focused on quasi-static manipulation, so results may not represent broader human-robot teams or more dynamic settings.

    Source: §V · p. 6

Abstract

from arXiv · show

Robot learning methods have recently made great strides, but generalization and robustness challenges still hinder their widespread deployment. Failing to detect and address potential failures renders state-of-the-art learning systems not combat-ready for high-stakes tasks. Recent advances in interactive imitation learning have presented a promising framework for human-robot teaming, enabling the robots to operate safely and continually improve their performances over long-term deployments. Nonetheless, existing methods typically require constant human supervision and preemptive feedback, limiting their practicality in realistic domains. This work aims to endow a robot with the ability to monitor and detect errors during task execution. We introduce a model-based runtime monitoring algorithm that learns from deployment data to detect system anomalies and anticipate failures. Unlike prior work that cannot foresee future failures or requires failure experiences for training, our method learns a latent-space dynamics model and a failure classifier, enabling our method to simulate future action outcomes and detect out-of-distribution and high-risk states preemptively. We train our method within an interactive imitation learning framework, where it continually updates the model from the experiences of the human-robot team collected using trustworthy deployments. Consequently, our method reduces the human workload needed over time while ensuring reliable task execution. Our method outperforms the baselines across system-level and unit-test metrics, with 23% and 40% higher success rates in simulation and on physical hardware, respectively. More information at https://ut-austin-rpl.github.io/sirius-runtime-monitor/

I. INTRODUCTION

Interactive imitation learning can support continual improvement during deployment, but typically relies on constant human supervision. This work combines model-based runtime monitoring with intervention-informed learning to anticipate errors, with reported gains over interactive imitation-learning baselines.

I. INTRODUCTION

  • Interactive imitation learning uses human interventions during execution for difficult situations, but existing approaches typically require continuous monitoring and imminent feedback.The introduction identifies runtime monitoring as a step toward reducing this human workload.Source: §I · p. 1
  • The monitor predicts future policy outcomes with a dynamics model and uses intervention-informed learning to update an error predictor without requiring explicit failures.It integrates out-of-distribution detection and failure classification in a shared latent-space model.Source: §I · pp. 1–2
  • 32% higher success rates and 16% more efficient human-workload utilization were achieved on average than state-of-the-art interactive imitation-learning baselines.Evaluation covered two simulated and two real-world tasks.Source: §I · p. 2
  • The method jointly and continually updates its error predictor and policy during human-in-the-loop deployments.The evaluation compares it with baselines in simulation and on physical hardware using system-level and unit-test metrics.Source: §I · p. 2

II. RELATED WORK

Prior robot-learning runtime monitors use out-of-distribution detection, supervised failure detection, or risk estimation. This work uses a learned dynamics model to anticipate risky states and request timely human supervision.

Interactive Imitation Learning

  • Robot-gated interactive imitation learning typically solicits feedback using heuristic measures such as ensemble variance, risk, or task uncertainty.This work instead investigates automated error prediction to minimize runtime errors and support trustworthy deployment.Source: §II · p. 2

Runtime Monitoring in Robot Learning

  • Out-of-distribution detection uses distribution shift as an error proxy, supervised failure detection classifies normal and failure states, and risk methods estimate future failure likelihood.The cited prior approaches draw on different signals, including success/failure examples and agent rollouts.Source: §II · p. 2

Model-based Learning Approach

  • A learned dynamics model predicts future policy-action outcomes in latent space, enabling identification of risky states before they occur and timely requests for human supervision.The paper applies model-based prediction to runtime monitoring.Source: §II · p. 2

III. MODEL-BASED RUNTIME MONITORING

The paper formulates manipulation as a task-oriented learning problem with human intervention available during deployment. Its runtime-monitoring system is intended to solicit human feedback when it detects an error, rather than relying on continuous supervision.

A. Problem Formulation

  • The manipulation task is formulated as a discrete-time MDP with continuous states and actions, unknown transition dynamics, and an objective of maximizing reward while minimizing deployment failures.The policy maps states to actions.Source: A. Problem Formulation · pp. 2–3

III. MODEL-BASED RUNTIME MONITORING

  • The system combines a latent-state dynamics model, a policy, and a failure classifier, with the classifier using human intervention states to infer failure states.The dynamics model predicts the next latent state from the current state and action.Source: Figure 2 · p. 3

A. Problem Formulation

  • Humans provide corrective interventions during task execution, and the monitor actively solicits feedback when it detects an error rather than requiring continuous human monitoring.This contrasts with prior work in which humans continuously monitor the system and provide feedback as needed.Source: A. Problem Formulation · p. 3

B. Model-based Method Design

The method encodes observations into a shared latent space for its dynamics model, policy, and failure classifier. Deployment trajectories and intervention cues provide the training data for predicting future states and identifying risky states.

Dynamics Model and Policy

  • An observation encoder maps raw images and robot proprioception into shared latent embeddings used by the dynamics model, policy, and failure classifier.The method uses this shared space to support latent-space policy rollouts.Source: B. Model-based Method Design · p. 3
  • The dynamics model predicts future latent states rather than raw observations, and its conditional VAE can sample multiple possible futures.The model takes past latent-state embeddings and the current action as input.Source: B. Model-based Method Design · p. 3
  • The policy is jointly trained with the dynamics model using a behavior-cloning objective combined with VAE losses, including KL divergence and next-state latent reconstruction.This objective supports policy rollouts in the shared latent space.Source: B. Model-based Method Design · p. 3

Failure Classifier

  • Pre-intervention states provide human-derived risk labels for a failure classifier trained on a frozen latent space.The classifier distinguishes failure states, normal rollouts, and intervention states because interventions and failures can occur in similar states.Source: B. Model-based Method Design · p. 3

Implementation Details

  • The failure classifier is implemented as an LSTM that takes state-action history and outputs failure probability.The policy uses a BC-RNN, while the cVAE dynamics model uses a GMM latent vector and residual MLP encoder and decoder.Source: B. Model-based Method Design · p. 3

C. Runtime Monitor in Operation: Modules

The monitor simulates multiple future policy rollouts in latent space, then checks predicted states for distribution shift and likely failure before deciding whether to request human supervision.

C. Runtime Monitor in Operation: Modules

  • The cVAE dynamics model generates multiple latent-space policy rollouts up to l steps ahead, and the monitor averages evaluations across predicted futures.Rollouts are produced by iteratively running the dynamics model and policy.Source: C. Runtime Monitor in Operation: Modules · p. 3
  • Predicted futures are first checked for OOD status using distance to latent embeddings of human expert demonstrations, with threshold α.The monitor uses the mean nearest-neighbor distance across imagined futures.Source: C. Runtime Monitor in Operation: Modules · p. 4
  • For futures not classified as OOD, the failure classifier predicts future failure, and supervision is requested when K of the past P states are classified as failures.The averaged failure-classification result across rollouts is compared with threshold β to classify a state as a failure.Source: C. Runtime Monitor in Operation: Modules · p. 4

D. Runtime Monitoring in Operation: System

During iterative deployment, the robot runs its policy under runtime monitoring, records human interventions, and uses aggregated deployment data to retrain its models; experiments assess system performance and error prediction.

D. Runtime Monitoring in Operation: System

  • When monitoring detects OOD or failure, the system requests human monitoring; active human interventions are recorded as labels for training the failure classifier.The intervention label h_t is 1 when the human supervisor actively intervenes at that step.Source: D. Runtime Monitoring in Operation:… · p. 4
  • Each deployment round adds transitions and intervention labels to the dataset, and the next round retrains models on the combined data from prior rounds.The system starts from a base model initialized with the human demonstration dataset.Source: D. Runtime Monitoring in Operation:… · p. 4

IV. EXPERIMENTS

  • Table II reports that the method consistently outperforms the baseline in combined policy success rate across rounds, with Round 1 omitted because of full-human-monitoring warm-start.The experiments include system-level evaluation and unit tests of error predictors.Source: Table II · p. 4, §IV · p. 4

A. System-Level Performance

System-level evaluation uses iterative deployments to assess task success and human-input efficiency, with normalized ROHE tracking how success relates to intervention effort over time.

A. System-Level Performance

  • The deployment loop runs for three rounds, aggregates data with human interventions, retrains on accumulated data, and holds intervention counts consistent across methods.The robot starts from a behavior-cloning policy.Source: A. System-Level Performance · p. 4

1) Evaluation Protocol:

  • Collaborative policy performance is measured by overall task success, while optimized human-input use measures how effectively assistance improves outcomes within a fixed time frame.Normalized ROHE adapts Return On Human Effort for a single robot and captures success rates; its denominator penalizes excessive interventions.Source: 1) Evaluation Protocol · pp. 4–5
  • In sparse-reward settings, the ROHE numerator is the policy success rate; H counts interventions across trajectories and T counts time steps.The reward is obtained from performing trajectory τ.Source: 1) Evaluation Protocol · p. 4
  • Normalized ROHE is generally lower for the method in Round 1, then improves in later rounds as it identifies important deployment errors more effectively.The curves cover Square Nut Assembly, Threading, Coffee Pod Packing, and Gear Assembly.Source: Figure 3 · p. 5

A. System-Level Performance

  • The unit-test table reports that the method outperforms other baselines on IOU and DCI for error-predictor performance.Higher IOU indicates greater overlap with human-labeled failures, while lower DCI indicates failure events closer to human labels.Source: Table III · p. 5

2) Baselines:

The evaluation compares the method with ThriftyDAgger and also includes MoMaRT and PATO, which were not designed for continuous policy updates.

2) Baselines:

  • ThriftyDAgger queries human corrections during continuous policy updates and detects novel states through disagreement among an ensemble of policies.It estimates state risk with a Q-function learned from data and policy rollouts, requiring failure data collection that may be infeasible in safety-critical domains.Source: 2) Baselines · p. 5
  • MoMaRT uses image VAE reconstruction error to indicate OOD states, while PATO combines ensemble-policy variance with uncertainty in future latent-state prediction.MoMaRT and PATO were run on deployment data generated by the proposed method for a fair comparison on the two simulation tasks.Source: 2) Baselines · p. 5

3) Results:

Our method outperforms baselines on collaborative policy performance and Normalized ROHE, with full human monitoring in Round 1 preceding improved outcomes in later rounds. The results also describe error patterns in competing methods.

3) Results:

  • Experiments cover Nut Assembly and Threading in simulation and Coffee Pod Packing and Gear Assembly in the real world, using a Franka Emika Panda arm.The arm uses a parallel jaw gripper and operational space controller.Source: 3) Results · p. 5
  • Our method outperforms baselines in collaborative policy performance and Normalized ROHE score.The authors attribute the higher performance to future latent-state prediction enabling more timely interventions.Source: 3) Results · p. 5
  • Round 1 uses full human monitoring to obtain intervention labels; in Rounds 2–3, the method models failure states better, with higher combined success and improved ROHE.The lower Round 1 Normalized ROHE score reflects the initial full-monitoring requirement.Source: 3) Results · p. 5
  • ThriftyDAgger and PATO can produce false positives from task uncertainty, while ThriftyDAgger’s risk threshold can yield false positives early and false negatives late.MoMaRT’s reconstruction error can miss small image differences associated with fine-grained manipulation failures, producing false negatives.Source: 3) Results · p. 5

B. Unit Testing Error Predictors

The unit tests compare learned error predictions with human risk assessments from actively monitored trajectories. The method predicts human intervention points accurately and outperforms baselines across tasks.

B. Unit Testing Error Predictors

  • In active human monitoring, operators supervise learned-policy trajectories and intervene at unsafe states; predictors are then compared with human labels independently of the deployment loop.This procedure obtains failure states as judged by human observers.Source: B. Unit Testing Error Predictors · pp. 5–6
  • IOU measures timestep overlap between predicted and human failure regions, while DCI averages temporal distances between predicted failures and human interventions.DCI penalizes predictions farther in time from human interventions.Source: B. Unit Testing Error Predictors · p. 6
  • Our method accurately predicts human intervention points and outperforms baselines on all tasks.The authors attribute this to intervention-informed learning, which incorporates human judgments of risk as supervisory signals.Source: B. Unit Testing Error Predictors · p. 6

C. Ablation Study

The ablation finds higher overall performance from combining OOD and failure detection, whose complementary roles address different errors. The conclusion describes deployment-data runtime monitoring and notes limits in participant diversity and task scope.

C. Ablation Study

  • Ours (Full) achieves higher overall performance than Ours Failure and Ours OOD.The ablation uses the Round 3 setting with the same policy on Nut Assembly and Threading.Source: Figure 4 · p. 6, C. Ablation Study · p. 6
  • OOD detection flags unfamiliar states that may lead to failure, while failure detection identifies subtle changes in critical bottleneck states that OOD may overlook.The qualitative examples distinguish coarser OOD detection from finer-grained failure detection.Source: C. Ablation Study · p. 6, Figure 5 · p. 6

V. CONCLUSION AND LIMITATIONS

  • The runtime-monitoring algorithm uses deployment data to predict errors and seek human assistance within long-term human-in-the-loop policy learning.The authors report that it is effective for trustworthy deployment and facilitating policy learning over long-term deployments.Source: §V · p. 6
  • Experiments use one human subject per task and focus on quasi-static manipulation, leaving broader team representativeness and dynamic settings untested.The authors identify larger-scale studies and applications to dynamic environments, diverse robot platforms, and more complex tasks as future directions.Source: §V · p. 6
Loading 2310.17552v1…