PaperlayerView exact revision on arXiv

Source-linked AI summary

Robot-WATCHDOG : failure detection through object-centric graph representation

Quentin Rolland, Fabrice Mayran de Chamisso, Jean-Baptiste Mouret

arXiv:2610.13403v1cs.RO

TL;DR

Reliable robotic failure detection is difficult when failures cannot be exhaustively specified and existing monitors struggle with benign variation and relational errors. Robot-WATCHDOG models object-centric spatio-temporal graphs and scores them with geometric-relational or predictive methods, calibrated using nominal demonstrations. TPC leads the reported dataset averages, while the framework's discrete object representation and upstream tracking impose scope limits.

  • Problem

    Open-world failures cannot be exhaustively enumerated, and existing monitors struggle to balance robustness to benign variation with detection of object-interaction errors.

    Source: §1 · p. 2, §2.2 · p. 3

  • Method

    Robot-WATCHDOG represents manipulation episodes as object-centric spatio-temporal graphs and scores deviations with TPC or a predictive GnnT, using conformal calibration from nominal demonstrations.

    Source: §3 · p. 4, §3.4 · p. 6

  • Results

    TPC leads both dataset averages, with AUPR/maximum MCC of 0.811/0.751 on BotFails and 0.810/0.719 on Watchdog.

    Source: §4.4 · p. 7

  • Takeaways & Limitations

    TPC is strongest overall in the tested data regimes without training or high inference cost, while GnnT approaches it in the continuously dynamic conveyor task.

    Source: §5 · p. 8

    The discrete object-centric representation limits applicability to deformable materials or domains without clear object decomposition, and centroid-only modeling misses orientation and fine-grained pose failures.

    Source: §6 · p. 8

Abstract

from arXiv · show

Reliable real-time failure detection is critical for deploying learned robotic policies, yet defining failures explicitly is infeasible in open-world settings. This motivates failure detection from nominal demonstrations alone. Existing methods, however, struggle to balance robustness to benign variations with the ability to capture semantic and interaction-level errors. We introduce Robot-WATCHDOG, a self-supervised framework that models manipulation scenes as object-centric spatio-temporal graphs. Building on this representation, we investigate two complementary failure scores: Trajectory Projection and Correlation (TPC), a non-parametric measure of geometric deviation and relational consistency, and GnnT, a Graph Attention Network with a Transformer that predicts future object states. We evaluate both on a public benchmark and on a new dataset of semantically complex manipulation tasks, across six tasks and ten random seeds. Both scores outperform image-based failure detectors, particularly on semantic and interaction-level failures. TPC provides the strongest performance in AUPR and MCC on both dataset averages, with low variance across seeds. Qualitative results are made available at https://cea-list.github.io/Robot-WATCHDOG/.

1 INTRODUCTION

Robot-WATCHDOG targets failures that nominal demonstrations can define more practically than an exhaustive open-world failure list, especially errors in object interactions. It represents object interactions over time and evaluates two graph-based scores, which outperform image-based detectors in the reported averages.

Motivation

  • Image-based detectors are sensitive to benign visual changes, while kinematic monitors can miss wrong-object interactions and mismatches between the robot and its environment [14] [17].The one-class setting trains monitors on nominal demonstrations because enumerating all possible failures is infeasible [12].Source: §1 · p. 2
  • Relational failures include grasping the wrong item, violating task order, or placing an object in an incorrect relation despite otherwise familiar objects.Such errors are difficult to capture using only image appearance or motion trajectories.Source: §1 · p. 2

Approach

  • Robot-WATCHDOG models object states as graph nodes and pairwise spatial relationships as edges, jointly representing visual, kinematic, and relational signals over time.This object-centric representation is intended to capture task-relevant interaction failures alongside signals monitored by prior approaches.Source: §1 · p. 2
  • TPC measures geometric deviation and pairwise-correlation changes, while GnnT predicts future object dynamics; conformal prediction calibrates both using held-out expert demonstrations.The framework is self-supervised and scores deviations from expert demonstrations.Source: §1 · p. 2

Results

  • TPC achieves AUPR/maximum MCC of 0.81/0.75 on BotFails and 0.81/0.72 on Watchdog, outperforming GnnT and the strongest baselines in both dataset averages.Results are averaged over ten seeds.Source: §1 · pp. 2–3

Contributions

  • The paper identifies relational anomalies, introduces TPC and GnnT, and releases a benchmark of manipulation tasks with annotated relational failures.The framework is evaluated on four BotFails tasks and two newly introduced relationally challenging tasks.Source: §1 · pp. 2–3

2 RELATED WORK

Prior work uses object-centric graphs for representation, dynamics, policy learning, and planning, but has given limited attention to online failure detection. Existing monitors also leave gaps in robustness to benign visual variation and detection of relational errors.

2.1 Graph oriented object representation

  • Object-centric graph representations encode task-relevant entities and their relations, supporting work on dynamic scenes, manipulation, relational policies, planning, dynamics, and policy adaptation [21] [22] [24] [25] [26] [28].The paper identifies online failure detection as a comparatively underexplored use of these representations.Source: §2.1 · p. 3

2.2 Failure detection in autonomous robotics

  • Existing monitors include one-class methods and multimodal approaches, but visual clutter, environment-only failures, latency, and brittle modality combinations remain concerns [12] [29] [30] [31] [32].The passage notes that Sentinel's VLM fallback has severe inference latency for reactive continuous control.Source: §2.2 · p. 3
  • Failures involving object identity, spatial arrangement, or task progression can retain nominal motion and subtle pixel changes, making them difficult for action-consistency metrics and standard visual detectors.Robot-WATCHDOG represents tracked entities as a spatio-temporal graph and detects relational changes through TPC or a predictive GNN-Transformer without failure demonstrations.Source: §2.2 · pp. 3–4

3 Method

Robot-WATCHDOG turns RGB observations and robot proprioception into object-centric graph sequences, then scores deviations from nominal demonstrations. It uses geometric-relational scoring or learned future-state prediction, with conformal calibration converting scores into alerts.

3 Method

  • The framework applies graph extraction, per-timestep failure scoring, and conformal prediction to produce a binary failure decision.Both scorers use nominal demonstrations; the scalar score sums per-object scores.Source: §3 · p. 4, Figure 2 · p. 4

3.1 Problem Formulation and Graph Extraction

  • Each timestep's graph encodes object positions and velocities at nodes and pairwise distances and orientations on edges, supporting translation-invariant relations and temporal deviations.The episode is represented as a sequence of these graphs.Source: §3.1 · pp. 4–5

3.2 TPC: Geometric and Relational Consistency

  • TPC compares each object's trajectory with the closest expert trajectory of the same label, then adds an L1 penalty for changes in pairwise correlations between objects.Exponentially increasing temporal weights emphasize recent deviations, and streaming scores can be updated at constant cost per timestep.Source: §3.2 · p. 5

3.3 GnnT: Predictive Spatio-Temporal Dynamics

  • GnnT predicts future object positions from sliding-window graph embeddings, using attention over neighbors and a Transformer to model dependencies across objects and time.It trains on nominal demonstrations with trajectory MSE plus an edge-consistency loss; this learned dynamics approach can represent action-reaction delays but needs enough demonstrations.Source: §3.3 · pp. 5–6

3.4 Uncertainty Calibration via Conformal Prediction

  • Conformal prediction uses held-out nominal scores to set a time-varying threshold for user-specified tolerance α, avoiding manual task-specific tuning and failure examples.The empirical quantile is approximate; under exchangeability, the corresponding finite-sample construction would control trajectory-level error rate.Source: §3.4 · p. 6

4 Evaluation

The evaluation compares Robot-WATCHDOG on four BotFails tasks and two newly introduced relationally and dynamically challenging tasks, against trajectory, learned-representation, and visual-semantic baselines. TPC leads threshold-independent and calibrated performance on both datasets.

4.1 Dataset

  • The evaluation uses four rigid, distinguishable-object BotFails tasks and two new tasks: structured table setting and waste sorting on a moving conveyor.BotFails tasks involving deformable or fluid interactions are excluded because they violate the method’s discrete-object abstraction; the new tasks add relational and dynamic complexity.Source: §4.1 · p. 6

4.2 Baselines

  • Baselines span trajectory matching, nominal-data density and reconstruction methods, and visual-semantic alignment; Robot-WATCHDOG contributes TPC and GnnT variants.Trajectory methods include AP-RMS, AP-Fréchet, and Soft-DTW; learned baselines include normalizing flows and an autoencoder, alongside FIDeL.Source: §4.2 · pp. 6–7

4.4 Results and discussion

  • TPC leads on both datasets, reaching AUPR 0.811 / MCC 0.751 on BotFails and 0.810 / 0.719 on Watchdog, ahead of GnnT and the strongest baselines.In the misplaced-cup example, the graph captures relational inconsistency and triggers a score spike although absolute kinematics cannot detect the error.Source: Figure 3 · p. 7, §4.4 · p. 7, Figure 4 · p. 8

4.3 Evaluation Protocol

  • Step-level evaluation reports threshold-independent AUPR and maximum MCC, plus calibrated MCC after conformal prediction, averaged over ten seeds.Seeds redraw the expert train/calibration partition, network initialization, and conformal split; calibrated results average over α ∈[0.80, 0.95].Source: §4.3 · p. 7

4.4.1 End-to-End Failure Detection

  • After conformal calibration, TPC leads in MCC on both datasets: 0.675 on BotFails and 0.641 on Watchdog, ahead of GnnT and the best online baselines.Across calibration settings, baseline rankings change; TPC peaks at low tolerance, while GnnT becomes competitive at high tolerance.Source: Figure 5 · p. 8, §4.4.1 · p. 8

5 Conclusion

Robot-WATCHDOG monitors robotic manipulation using object-centric spatio-temporal graphs and evaluates non-parametric TPC alongside predictive GnnT. TPC is strongest overall across the tested data regimes, while GnnT only approaches it on the continuously dynamic conveyor task.

5 Conclusion

  • Robot-WATCHDOG is a self-supervised robotic failure detector based on object-centric spatio-temporal graphs.The framework studies TPC, a non-parametric geometric and relational deviation score, and GnnT, a predictive graph-transformer.Source: §5 · p. 8
  • Both scores outperform image- and density-based detectors, particularly on failures involving object interactions and task structure.TPC is strongest overall in both tested data regimes and requires no training and the lowest inference cost; GnnT only approaches it on the continuously dynamic conveyor task.Source: §5 · p. 8
  • The results identify object-centric relational representations as a promising direction for scalable real-time monitoring of learned robotic policies.Source: §5 · p. 8

6 Limitations

The ablation examines when object-centric graph structure helps failure detection, while the limitations identify settings the current abstraction and perception pipeline do not cover reliably.

6 Limitations

  • The discrete object-centric abstraction limits use on deformable materials, fluids, or domains without clear object decomposition; tracking errors and centroid-only states further constrain detection.Severe occlusions, missed detections, or identity switches can degrade the graph, while centroid-only modeling cannot capture orientation or fine-grained pose failures.Source: §6 · p. 8

A Ablation Study

  • The ablation study tests input representation, relational modeling, spatio-temporal architecture, and predictive formulation.The supplied passages describe the vision-feature, trajectory-only, and graph-based input configurations.Source: A Ablation Study · p. 14, §A.1 · p. 14

A.1 Impact of Object-Centric Graph Representation

  • On Table-setting, graphs beat trajectory-only by 29.1 TPR, 25.5 AUPR, and 23.8 MCC points; on Waste-sorting, trajectories retain higher AUPR and MCC.Waste-sorting results are 54.7% vs. 49.7% AUPR and 56.5% vs. 53.1% MCC for trajectory-only versus graph-based inputs.Source: §A.1 · p. 14, Table 1 · p. 15
  • Explicit object trajectories improve on raw vision features, while the graph's added value is greatest when failures depend on inter-object configurations.Waste-sorting failures often involve individual-object deviations detectable from trajectories, leaving less room for relational reasoning.Source: §A.1 · pp. 14–15

A.2 Impact of Relational Modeling

These ablations show how relational structure, spatio-temporal modeling, and prediction affect failure detection, and describe the trajectory-extraction pipeline that supplies object states.

A.2 Impact of Relational Modeling

  • For TPC, combining projection and correlation yields Table-setting's best AUPR (96.3%) and MCC (88.8%), while on Waste-sorting it has the highest TNR (96.8%) but lower TPR (47.7%).The two signals capture complementary geometric deviations and relational inconsistencies, with their relative value depending on task.Source: Table 2 · p. 16, §A.2 · p. 16
  • Removing GnnT's edge model degrades both tasks, most sharply on Table-setting (AUPR −20.7 points, MCC −11.2 points); it also increases Waste-sorting TNR variability across seeds.Waste-sorting TNR standard deviation is 8.3 without the edge model versus 3.5 with it.Source: §A.2 · p. 16

A.3 Impact of Spatio-Temporal Modeling

  • GnnT achieves the highest MCC on both tasks and highest AUPR on Table-setting; on Table-setting, adding temporal attention to the GNN raises AUPR to 80.0% and MCC to 73.1%.The MLP baseline has the lowest AUPR and MCC on both tasks; differences among non-MLP variants on Waste-sorting are small.Source: Table 3 · p. 17, §A.3 · p. 17

A.4 Prediction vs Density Estimation

  • On Waste-sorting, GnnT gains 8.0 MCC points over density estimation at equivalent false-positive rates; on Table-setting, metrics are comparable, with lower variance for GnnT.On Table-setting, AUPR is 80.0% vs. 78.2% and MCC is 73.1% vs. 72.9% for GnnT versus density estimation.Source: §A.4.1 · p. 18, Table 4 · p. 18

B.1 Trajectory Extraction

  • The extraction pipeline uses a task-prompted vision-language model to initialize object detections, then SAM2 tracks mask centroids through the episode.A degenerate mask carries forward the last valid centroid and marks the object invalid; VLM initialization is offline, while SAM2 runs at approximately 44 fps on an A100 with five objects.Source: §B.1 · p. 19

B.2.1 VLM Initialization Robustness

This section evaluates object detection and tracking components, describes the GnnT and TPC scoring methods, and sets out how nominal demonstrations calibrate sequential decision bands. The VLM standardization loop improves object-detection success, while the later methods compare robot behavior with nominal patterns.

B.2.1 VLM Initialization Robustness

  • 94.4% overall detection success with Qwen WithStd, compared with 92.4% for base Qwen, after the loop re-prompts when the expected object count is not found.The missing-object rate falls from 5.3% to 2.1%; base Qwen also has 1.8% identity confusion and 0.6% localization inaccuracies.Source: Table 6 · p. 22, §B.2.1 · p. 22

B.2.2 Tracking Robustness

  • SAM2 tracking outcomes are categorized as Perfect, Recovered, Missing, or Fusion, with evaluation initialized from ground-truth bounding boxes.Recovered means an object lost temporarily through occlusion is tracked again before the task ends; Missing means it is definitively lost, and Fusion means it is confused with another nearby object.Source: §B.2.2 · pp. 22–23, Table 7 · p. 23

B.3 GnnT - Model Architecture

  • GnnT combines an edge-aware spatial graph encoder with a Transformer sequence model to capture object geometry and inter-frame dynamics.It predicts future object trajectories from the latest object tokens using a shared two-layer MLP.Source: §B.3 · pp. 23–24

B.4 Trajectory Projection and Correlation (TPC) Formulation

  • TPC scores kinematic deviation from the nearest nominal expert trajectory and relational deviation from expert pairwise correlations, with the combined score summing both components.Trajectories are filtered to remove static segments; projection distances are normalized using expert-set statistics and temporally weighted.Source: §B.4 · pp. 24–25

B.5.1 Calibration Data

  • Each seed partitions expert demonstrations into a detector or scorer reference set and a disjoint nominal calibration set.No demonstration is used both to fit the detector and to calibrate its threshold.Source: §B.5.1 · p. 25

B.5.2 Sequential Band Construction

  • The sequential band estimates its center from DA and calibrates its width on DB, using the empirical (1 − α)-quantile of each trajectory’s maximum normalized deviation.The constant modulation gives a uniform offset from the mean expert profile; the alternative widens the band where expert demonstrations disagree.Source: §B.5.2 · pp. 25–26

B.5.3 Online Deployment and Evaluation

The framework flags a failure when the online score first breaches a time-varying band, with the final band value held for longer rollouts. The reported component latencies support real-time use, while the nominal-level guarantee is asymptotic and indicative at the calibration sizes used.

B.5.3 Online Deployment and Evaluation

  • At test time, a failure is flagged at the earliest timestep when the score exceeds the one-sided time-varying threshold; beyond the calibration horizon, the final band value is held constant.Headline results average operating points α ∈ [0.80, 0.95], and all results average 10 seeds with redrawn partitions, calibration splits, and network initialization.Source: §B.5.3 · p. 26
  • The nominal-trajectory flag rate is controlled at approximately α only asymptotically in the exchangeable regime, so the nominal level is indicative for calibration sets of a few tens of demonstrations.The method uses the plain empirical quantile rather than the finite-sample-corrected order statistic, and the difference is not negligible at the calibration sizes used.Source: §B.5.3 · p. 26

B.6 Pipeline Latency and Real-Time Capability

  • 29.72 ms per frame (33.6 FPS) for TPC exceeds a 30 FPS control-loop requirement, while GnnT runs at 23.5 FPS for standard 15–20 FPS setups.On an NVIDIA RTX 3090, SAM2 takes approximately 22.72 ms per frame, GnnT inference 19.87 ms, and TPC sliding-window inference 7.00 ms; VLM initialization takes approximately 7.0 seconds but runs only once.Source: §B.6 · p. 26, Table 9 · p. 27

C.1 Datasets

The evaluation combines BotFails with a new dataset of structured table-setting and dynamic waste-sorting tasks, complementing controlled scenarios with more variable manipulation conditions. The section also specifies trajectory-based baselines for comparison.

C.1 Datasets

  • BotFails tasks involving rigid, non-deformable, visually distinguishable objects are retained, while tasks such as pouring are excluded because they violate persistent object identity assumptions.The new dataset complements BotFails with more expert demonstrations and sorting scenarios where object motion induces continuous distribution shift.Source: §C.1 · pp. 27–28
  • The new dataset adds table-setting and moving-conveyor waste sorting to increase relational and dynamic task complexity.It includes expert demonstrations and evaluation trajectories containing both nominal behavior and failures; table-setting uses an ALOHA robot and waste sorting a SO-100 platform.Source: §C.1 · p. 27

C.2 Baselines

  • The comparison includes classical and learning-based detectors, including the proposed few-shot TPC and predictive-graph GnnT approaches.The baseline overview is presented in Table 11.Source: §C.2 · p. 28, Table 11 · p. 29

C.2.1 Arc-Length Prefix RMS (AP-RMS)

  • AP-RMS aligns trajectories by arc length and scores each prefix by its RMS distance to the closest expert prefix, averaging minimal deviations over the trajectory.Arc-length parameterization is used to make the comparison invariant to execution speed and sampling rates.Source: §C.2.1 · pp. 28–29

C.2.2 Arc-Length Prefix Fr´echet (AP-Fr´echet) Formulation

  • AP-Fréchet aligns trajectories by arc length and measures each prefix’s discrete Fréchet distance to the closest expert prefix, standardized using expert-set distances.The distance considers order-preserving couplings and is computed with dynamic programming.Source: §C.2.2 · pp. 29–30

C.2.3 Soft Dynamic Time Warping (Soft-DTW) Formulation

  • Soft-DTW scores the minimum soft dynamic-time-warping distance from a full observed trajectory to the nominal expert set, then standardizes that distance using expert-set statistics.The soft-minimum makes the distance less brittle to localized noise than the standard DTW path; trajectories are downsampled or zero-padded to a fixed maximum length.Source: §C.2.3 · pp. 30–31

C.3 Evaluation Metrics

The evaluation measures continuous failure scores separately from thresholded end-to-end decisions, using step-level labels. It reports metrics for both discrimination under class imbalance and operational failure detection.

C.3 Evaluation Metrics

  • Predictions are evaluated at the timestep level, with labels distinguishing nominal behavior from task failure.The evaluation separates raw-score discrimination from binary detection after applying conformal prediction thresholds.Source: §C.3 · p. 31
  • Raw failure scores are assessed with AUPR and maximum MCC across thresholds to measure precision–recall performance and threshold-independent classification balance.AUPR is favored under strong class imbalance; MCC ranges from −1 to 1, with 0 indicating random correlation and 1 perfect correlation.Source: §C.3 · p. 31
  • The end-to-end pipeline reports TPR, TNR, and threshold-calibrated MCC to assess failure interception and rejection of benign noise.TPR measures the proportion of failures intercepted, while TNR measures the ability to avoid false alarms on benign noise.Source: §C.3 · p. 31

D.1 Threshold-independent results per task

TPC leads on nearly every task in AUPR and maximum MCC, with an exception for AUPR on Dish storing. The comparisons also show task-dependent baseline performance and a distinct challenge on Waste sorting.

D.1 Threshold-independent results per task

  • TPC is best on every task in AUPR and maximum MCC except Dish storing AUPR, where logpZO scores 0.670 versus 0.639.Table 12 reports per-task AUPR, maximum MCC, and AUROC.Source: §D.1 · p. 31, Table 12 · p. 32
  • TPC outperforms GnnT on all six tasks in both metrics; on Waste sorting, its maximum MCC is 0.551 versus 0.531 and its AUPR is 0.658 versus 0.497.For Watchdog table-setting, TPC's AUPR is 0.963 versus GnnT's 0.800.Source: §D.1 · pp. 31–32
  • GnnT exceeds every baseline on dataset averages but not every task: logpZO leads Dish storing AUPR at 0.670, and AP-RMS leads Vegetable sorting at 0.711.AP-Fréchet leads Groceries sorting maximum MCC at 0.527 versus GnnT's 0.497; no baseline is strong across tasks.Source: §D.1 · p. 32
  • Trajectory-matching baselines fall below chance on Waste sorting AUPR (0.139–0.166 versus failure-step prevalence 0.22) and show seed variability up to 0.373 AUPR.The paper attributes the Waste sorting difficulty for stored-trajectory comparisons to continuously changing belt initial states.Source: §D.1 · p. 32

D.2 End-to-end results per task

After conformal calibration, TPC has the highest MCC on five of six tasks, while AP-Fréchet leads Waste sorting. Calibration particularly benefits trajectory matching, whose time-varying band absorbs score drift over an episode.

D.2 End-to-end results per task

  • TPC has the best calibrated MCC on five of six tasks, ranging from 0.519 on Dish storing to 0.819 on Vegetable sorting.On Waste sorting, AP-Fréchet scores 0.553, ahead of TPC at 0.470 and GnnT at 0.449.Source: §D.2 · p. 32, Table 13 · p. 33
  • A time-varying calibration band can raise trajectory-matching MCC by absorbing episode-long score drift; AP-Fréchet on Waste sorting reaches 0.553 calibrated MCC versus 0.192 maximum MCC.GnnT's calibrated MCC is below AP-Fréchet's on both dataset averages: 0.347 versus 0.452 and 0.571 versus 0.630.Source: §D.2 · p. 33
Loading 2610.13403v1…