PaperlayerView exact revision on arXiv

Source-linked AI summary

RoboMonitor: Label-Efficient Runtime Monitoring of Robot Task Execution via Predictive Representation Learning

Abhiroop Ajith, Gokul Narayanan, Kyle Coelho, Tingji Zhao, Yash Shahapurkar, Brian Zhu, Melih Erdogan, Ted Krubasik, Constantinos Chamzas, Eugen Solowjow

arXiv:2609.30715v1cs.RO

TL;DR

Robot-policy datasets contain few monitoring annotations, although execution monitoring needs to identify phase, failures, and task completion from observations. RoboMonitor pre-trains representations on robot trajectories and fine-tunes a causal monitor; with 52 labeled episodes it outperforms evaluated baselines on phase accuracy, and its integrated system completes trials in simulation and on a real robot.

  • Problem

    Robot-policy datasets often lack annotations for execution phase, failure onset, and task completion, while training monitors requires costly temporal labels.

    Source: §I · p. 1

  • Method

    RoboMonitor pre-trains visual and context representations from robot trajectories, then transfers them to a causal monitor trained with Temporal SFT.

    Source: §I · p. 1, A. Predictive Robot Pre-training · p. 3, B. Causal Monitor Fine-Tuning · p. 4

  • Results

    Across four tasks, RoboMonitor with 52 labeled episodes achieves 93.09% mean phase accuracy and 85.85% macro recall, exceeding evaluated Qwen3-VL and Robometer phase accuracy with 100 episodes.

    Source: §VII · p. 7

  • Takeaways & Limitations

    The integrated monitor–orchestrator completes 39 of 40 simulated Toolbox Sorting trials and 35 of 40 real-world Reel Packing trials, with no false recovery triggers observed.

    Source: §VII · p. 7

    The single global phase per timestep cannot represent asynchronous execution in which two manipulators independently perform different subtasks simultaneously.

    Source: §VII · p. 7

Abstract

from arXiv · show

Learned robot policies produce actions, but their outputs alone do not establish whether execution is progressing as intended. Robot execution monitoring requires identifying the current execution phase, detecting failures, and recognizing task completion from observations available during execution. Training such monitors requires annotations that are scarce in datasets collected for robot-policy learning. We present RoboMonitor, a label-efficient vision--language execution monitor that learns from these datasets before introducing monitoring supervision. We pre-train on 25 hours of multi-camera trajectories spanning 12 manipulation tasks and two robot embodiments, using action-conditioned future-feature prediction, inverse dynamics, and masked-present prediction. We then transfer the learned visual and context encoders to a causal monitor and apply temporal supervised fine-tuning (Temporal SFT), which combines supervision throughout each observation window with consistency objectives within and across overlapping windows. At deployment, RoboMonitor requires only the task instruction and camera observations. On a four-task monitoring benchmark, RoboMonitor trained with 52 labeled episodes achieves 93.1% mean phase accuracy and 85.9% macro recall over two fine-tuning seeds, exceeding Qwen3-VL and Robometer trained with the same monitoring supervision. Its phase accuracy also exceeds that of both Qwen3-VL and Robometer trained with 100 episodes. A Qwen3-VL ablation shows that Temporal SFT reduces mean spurious phase switching from 15.23% to 4.95%. In closed-loop deployment, the integrated system completes 39 of 40 simulated Toolbox Sorting trials and 35 of 40 real-world Reel Packing trials, with no false recovery triggers observed.

I. INTRODUCTION

Robot execution monitors provide feedback about task progress, failures, and completion, but monitoring labels are scarce in policy datasets. RoboMonitor learns representations from robot trajectories before fine-tuning a causal monitor; related work addresses neighboring verification, progress-estimation, and pre-training objectives.

I. INTRODUCTION

  • Execution monitoring supports task progression, recovery, replanning, or human intervention by identifying phase, failures, and completion from instructions and recent observations.Fluctuating phase estimates during the same stage can give an orchestrator inconsistent progress information.Source: §I · p. 1
  • Monitoring labels are scarce in robot-learning datasets, where trajectories often lack phase, failure-onset, and task-completion annotations.Representative failures and frame-by-frame annotation are costly, while policy datasets contain synchronized video, state, and action data.Source: §I · p. 1
  • RoboMonitor pre-trains visual and temporal representations on robot trajectories, then transfers them to a causal monitor fine-tuned with dense temporal supervision and consistency objectives.At deployment, the monitor uses the task instruction and camera observations, not robot state or commands.Source: §I · p. 1

II. RELATED WORK

  • Unlike subtask-conditioned verification and control-policy probing, RoboMonitor continuously infers phase, failure, and completion from causal visual observations without requiring the current subtask or controller access.AHA detects failures for a specified subtask, and Guardian verifies subtask execution from before-and-after observations.Source: §II · pp. 1–2
  • Robometer, SARM2 [15], and ROVER estimate reward, progress, or reasoning signals mainly for policy evaluation and improvement, whereas RoboMonitor makes execution monitoring its primary objective.RoboMonitor jointly predicts phase, failure, and task completion while explicitly considering temporal stability and online inference.Source: §II · p. 2
  • RoboMonitor applies predictive representation learning to multi-camera robot trajectories and transfers both the adapted visual encoder and its temporal context representation to runtime monitoring.This builds on reusable robot representations and joint-embedding predictive learning, which predicts target representations rather than pixels,.Source: §II · pp. 2–3

III. PROBLEM FORMULATION

The formulation considers instruction-specified manipulation observed through multiple cameras, whether execution is by an end-to-end policy or a planner selecting skills. A causal observation window supports predictions of phase, failure, and task completion, with robot state and actions used in training data but not at deployment.

III. PROBLEM FORMULATION

  • The task instruction specifies manipulation executed either by an end-to-end policy or by a high-level planner selecting low-level skills from a library.At each timestep, the robot is observed through multiple cameras.Source: §III · p. 3, A. Predictive Robot Pre-training · p. 3
  • Given the instruction and a causal window of recent multi-camera observations, the monitor predicts the current execution phase and probabilities of failure and task completion.The window contains observations through the current timestep.Source: §III · p. 3
  • Training assumes a large corpus of robot trajectories with video, state, and actions plus a smaller corpus with monitoring annotations; deployment uses only the instruction and camera observations.Source: §III · p. 3

IV. METHODOLOGY

RoboMonitor pre-trains visual and temporal representations from robot trajectories without monitoring annotations, then transfers them to a causal monitor. Predictive pre-training combines future-feature prediction with inverse-dynamics and masked-present objectives.

A. Predictive Robot Pre-training

  • Predictive pre-training learns visual and temporal representations from recorded observations, robot state, and commands without monitoring annotations.After pre-training, the visual and context encoders are retained for execution monitoring; the prediction heads and decoder are not deployed.Source: §IV · p. 3, A. Predictive Robot Pre-training · p. 4
  • The visual encoder is initialized from Qwen3-VL-4B-Instruct, while a fixed copy supplies target features for future-feature prediction rather than pixel reconstruction.The future predictor is conditioned on recorded state and intervening commands because the same scene can be followed by different commands.Source: A. Predictive Robot Pre-training · p. 3
  • The context encoder combines camera views at t − δ and t with camera, spatial-position, and relative-time information.Robot state and commands enter only the predictor, so monitoring context features can be computed from camera observations alone.Source: A. Predictive Robot Pre-training · pp. 3–4
  • Inverse dynamics predicts commands from the difference between visual context summaries, while the inverse head receives neither state nor commands as input.This gives the summary difference an explicit action-prediction target.Source: A. Predictive Robot Pre-training · p. 4
  • Masked-present prediction masks contiguous spatial regions and trains an auxiliary decoder to predict target features at masked locations from remaining visual context.The decoder receives neither robot state nor commands.Source: A. Predictive Robot Pre-training · p. 4

B. Causal Monitor Fine-Tuning

The causal monitor fine-tunes a VLM initialized with predictive visual representations, supervising labeled readouts throughout each observation window. Temporal consistency losses address phase changes within a window and disagreement across overlapping windows.

B. Causal Monitor Fine-Tuning

  • Temporal SFT trains labeled causal readouts throughout each observation window and adds consistency objectives within and across overlapping windows.The VLM’s visual encoder is initialized from the pre-trained encoder.Source: B. Causal Monitor Fine-Tuning · p. 4
  • Causal attention restricts each readout to the task instruction and observations up to its timestep.The readout is the VLM hidden state at a learned token inserted after each timestep’s image tokens.Source: B. Causal Monitor Fine-Tuning · p. 4
  • A separate frozen visual/context branch processes a recent observation pair and supplies features fused with the VLM readout through residual adapters.Phase and completion share the progression fusion feature, while failure uses separate adapters.Source: B. Causal Monitor Fine-Tuning · p. 4, Figure 3 · p. 5
  • The supervised objective combines phase cross-entropy with binary cross-entropy for failure and completion, and deployment returns predictions at the final timestep.The monitor is trained on phase, failure, and completion annotations; deployment uses the instruction and camera observations.Source: B. Causal Monitor Fine-Tuning · pp. 4–5
  • Within a window, a truncated log-probability smoothing loss discourages short-lived phase changes and caps penalties on large changes.The truncation threshold prevents the penalty from growing with the magnitude of large changes.Source: B. Causal Monitor Fine-Tuning · pp. 4–5
  • Across overlapping windows, the monitor penalizes prediction disagreement at shared timestamps using symmetric KL for phase distributions and squared error for failure probabilities.The failure term is weighted by β.Source: B. Causal Monitor Fine-Tuning · p. 5

V. EXPERIMENTS

The experiments examine whether RoboMonitor improves monitoring with limited labels, whether Temporal SFT improves temporal stability, and whether the monitor supports closed-loop execution and recovery.

V. EXPERIMENTS

  • The label-efficiency question is whether RoboMonitor improves execution monitoring when labeled data are limited.Source: §V · p. 5
  • The temporal-stability question is whether Temporal SFT reduces spurious phase switching compared with final-readout-only supervision.Source: §V · p. 5
  • The closed-loop question is whether RoboMonitor supports task execution and failure recovery within a task-orchestration system.Source: §V · p. 5

A. Data and Tasks

RoboMonitor is pre-trained on unannotated robot trajectories and fine-tuned for monitoring across four manipulation tasks, including tasks absent from pre-training.

A. Data and Tasks

  • Pre-training uses 25 hours of multi-camera trajectories across 12 manipulation tasks and two robot embodiments, without phase, failure, or completion annotations.The trajectories include synchronized RGB observations, robot state, and commanded actions.Source: A. Data and Tasks · p. 5
  • The supervised monitor covers Reel Packing, Electronic Component Insertion, Toolbox Sorting, and Humanoid Grasping, with task-specific annotated phase labels.The phases are PICK, HANDOVER, PLACE for Reel Packing; PICK, TRANSPORT, INSERT for Electronic Component Insertion; NONE, PICK, TRANSPORT, PLACE for Toolbox Sorting; and PUSH, PICK, PLACE for Humanoid Grasping.Source: A. Data and Tasks · p. 5
  • Reel Packing and Electronic Component Insertion appear in the pre-training corpus, whereas Toolbox Sorting and Humanoid Grasping do not; all four are used in supervised fine-tuning.The monitor uses a shared seven-class phase vocabulary and separate binary outputs for failure and task completion.Source: A. Data and Tasks · p. 5

B. Offline Evaluation

Offline evaluation compares monitoring systems at two labeled-episode budgets on a shared held-out benchmark, with results reported across two fine-tuning seeds.

B. Offline Evaluation

  • The supervised training budgets are 52 and 100 labeled episodes, with the smaller set nested within the larger one.The 52-episode set contains 13 episodes per task; the 100-episode set contains 27 Reel Packing, 26 Humanoid Grasping, 25 Toolbox Sorting, and 22 Electronic Component Insertion episodes.Source: B. Offline Evaluation · p. 5
  • Table I reports offline execution-monitoring performance in percentages, with higher values better except for spurious-switch rate.The supplied caption does not provide individual table values.Source: Table I · p. 6
  • Pointwise SFT is evaluated only at 52 episodes, while Gemini uses few-shot examples without parameter updates.Source: B. Offline Evaluation · p. 6
  • All methods are evaluated on the same held-out benchmark of 66 episodes and 7,022 observation windows, and fine-tuned configurations are reported as means and standard deviations over two seeds.Each evaluation query contains the task instruction and an eight-timestep causal window from two or three camera streams; checkpoints are validation-selected and scoring is shared.Source: B. Offline Evaluation · p. 6

C. Baselines

The baselines compare matched fine-tuning configurations with different supervision or architectures, alongside a few-shot model that receives labeled examples without parameter updates.

C. Baselines

  • Qwen3-VL-4B + Pointwise SFT uses eight-timestep causal windows but supervises only the final readout, omitting intermediate supervision and both temporal consistency objectives.This is the 52-episode ablation baseline.Source: C. Baselines · p. 6
  • Qwen3-VL-4B + Temporal SFT starts from the same foundation model, allowing comparison with Pointwise SFT of dense causal supervision and temporal consistency together.The comparison measures their combined contribution.Source: C. Baselines · p. 6
  • Robometer-4B + Temporal SFT provides a robot-specific reward-model baseline under the same downstream supervision.Source: C. Baselines · p. 6
  • Gemini Robotics ER 2 is evaluated without parameter updates, using labeled in-context examples, the task instruction, and causal camera observations to predict the same monitoring outputs.Its few-shot result is reported separately from supervised fine-tuning budgets.Source: C. Baselines · p. 6
  • RoboMonitor transfers visual and context encoders learned during predictive pre-training and applies Temporal SFT for downstream monitoring.Source: C. Baselines · p. 6

D. Closed-loop Evaluation

The closed-loop evaluation embeds RoboMonitor in a task orchestrator that selects skills and uses monitor predictions to track progress and trigger predefined recovery. Deployment is tested in simulation and on a real bimanual workcell, with only task instructions and camera observations required at runtime.

D. Closed-loop Evaluation

  • The orchestrator uses monitor predictions to track execution progress and trigger predefined recovery behaviors while selecting low-level skills.Source: D. Closed-loop Evaluation · p. 6
  • Evaluation covers simulated bimanual Toolbox Sorting in Isaac Sim and real-world Reel Packing on a bimanual UR7e workcell, with 40 trials in each setting.Source: D. Closed-loop Evaluation · p. 6

E. Implementation Details

F. Metrics

RoboMonitor performs strongly on phase recognition and task completion with limited labels, while failure detection shows a different budget pattern. Temporal SFT reduces spurious switching, but phase accuracy and switching rate do not always favor the same method.

F. Metrics

  • Phase accuracy, macro recall, task-completion AP, failure AP, and spurious-switch rate are reported offline; closed-loop metrics include failure detection, false recovery triggers, recovery success, task completion, and latency.Spurious-switch rate counts adjacent predicted phase changes while the ground-truth phase remains unchanged; lower is better. Table II defines recovery success over attempted recoveries.Source: F. Metrics · p. 6, Table II · p. 7

A. Label-Efficient Monitoring (Q1)

  • 93.09% mean phase accuracy and 85.85% macro recall are achieved by RoboMonitor with 52 episodes, exceeding both Qwen3-VL and Robometer with Temporal SFT on these measures.At this budget, task-completion AP is also highest for RoboMonitor at 87.60%, versus 85.42% for Robometer and 74.01% for Qwen3-VL.Source: A. Label-Efficient Monitoring (Q1) · p. 7
  • 93.09% mean phase accuracy with 52 episodes exceeds Robometer’s 91.65% with 100 episodes, while macro recall is 85.85% versus 86.22%.Against Qwen3-VL with 100 episodes, RoboMonitor’s 52-episode model has 93.09% versus 91.19% phase accuracy and 85.85% versus 82.73% macro recall.Source: A. Label-Efficient Monitoring (Q1) · p. 7
  • 45.87% failure AP for RoboMonitor with 52 episodes is below Robometer’s 59.39% and close to Qwen3-VL’s 46.37%; with 100 episodes, RoboMonitor leads both baselines on all four reported means.At 100 episodes, RoboMonitor records 93.63% phase accuracy, 88.55% macro recall, 92.29% task-completion AP, and 72.01% failure AP; failure AP is higher by 4.33 and 11.05 percentage points than the respective baselines.Source: A. Label-Efficient Monitoring (Q1) · p. 7
  • 74.67% phase accuracy and 61.19% macro recall for Gemini Robotics ER 2 are below all fine-tuned configurations.This few-shot reference uses labeled in-context examples without parameter updates.Source: A. Label-Efficient Monitoring (Q1) · p. 7

B. Temporal Stability (Q2)

  • 15.23% to 4.95% is the reduction in mean spurious-switch rate for Qwen3-VL with Temporal SFT versus Pointwise SFT at 52 episodes.Failure AP rises from 30.34% to 46.37%, while phase accuracy rises from 85.42% to 86.76%; this ablation evaluates the combined dense-supervision and temporal-consistency recipe.Source: B. Temporal Stability (Q2) · p. 7
  • 5.58% is RoboMonitor’s spurious-switch rate with 52 episodes, compared with 5.27% for Robometer and 4.95% for Qwen3-VL, although RoboMonitor has the highest phase accuracy at that budget.With 100 episodes, RoboMonitor has the lowest switching rate at 3.17%, versus 3.76% for Robometer and 3.58% for Qwen3-VL.Source: B. Temporal Stability (Q2) · p. 7

C. Closed-loop Execution and Recovery (Q3)

In closed-loop deployment, RoboMonitor supports task progression and predefined recovery in simulation and on a real robot. The conclusion reports successful trial completion in both settings, while identifying global per-timestep phase assignment as a limitation for asynchronous subtasks.

C. Closed-loop Execution and Recovery (Q3)

  • Inference takes approximately 180–200 ms per observation window, and the two deployments complement a four-task offline stress test.The deployments assess failure-event detection, recovery, and task completion within the integrated monitor–orchestrator system.Source: C. Closed-loop Execution and Recovery… · p. 7
  • 39 of 40 simulated Toolbox Sorting trials and 35 of 40 real-world Reel Packing trials are completed, with no false recovery triggers observed in either setting.In simulation, 11 of 12 failure events are detected and all 11 attempted recoveries succeed; in real-world trials, 15 of 18 are detected and 13 of 15 attempted recoveries succeed.Source: §VII · p. 7, C. Closed-loop Execution and Recovery… · p. 7

VII. CONCLUSION

  • One global execution phase per timestep cannot express asynchronous execution in which two manipulators independently perform different subtasks at once.The formulation represents coordinated bimanual behaviors such as handover; factorized or multilabel states are proposed for future work.Source: §VII · p. 7
  • Visually grounded language reasoning and conformal methods are proposed extensions to categorical outputs, with calibrated uncertainty intended to support abstention, additional observations, or human involvement.Source: §VII · p. 8

Related papers

Loading 2609.30715v1…