PaperlayerView exact revision on arXiv

Source-linked AI summary

RAFAIL: Relationship-Aware Failure Detection for Robotic Manipulation

Loris Schneider, Edgar Welte, Rania Rayyes

arXiv:2609.18324v1cs.RO

TL;DR

Robotic failure detectors must distinguish deviations that threaten task success from harmless scene changes, while continuous VLM inference adds runtime cost. RAFAIL uses offline VLM supervision to detect anomalies in task-relevant object relationships, achieving 73.4% balanced accuracy across three real-world tasks versus 65.2% for the best evaluated baseline.

  • Problem

    Existing OOD and uncertainty detectors can flag harmless scene changes or miss consequential interaction errors, while continuous VLM inference adds runtime cost.

    Source: §I · p. 1

  • Method

    RAFAIL learns point-cloud relationship representations using offline VLM supervision, then applies relationship-specific OOD detection with predicted task progress and relationship importance at runtime.

    Source: §I · p. 1, F. Relationship Representation Learning · p. 5

  • Results

    73.4% balanced accuracy across three real-world robotic manipulation tasks, versus 65.2% for the best evaluated baseline.

    Source: §I · p. 2

  • Takeaways & Limitations

    Relationship-specific detections frequently agreed with the relationships involved in observed failures.

    Source: §V · p. 8

    Subtle failures caused by small pose deviations may remain within the distribution of successful relationship states.

    Source: §V · p. 8

Abstract

from arXiv · show

Detecting failures during execution is essential for reliable robotic manipulation. Vision-language models (VLMs) can assess task outcomes semantically but add runtime computation, whereas out-of-distribution (OOD) detectors may respond to harmless scene variations rather than failure-relevant deviations. We introduce RAFAIL, a framework for detecting execution failures during robotic manipulation. RAFAIL identifies failures by detecting anomalies in task-relevant relationships between entities, such as a gripper and an object or an object and its target. By focusing OOD detection on relevant parts of the observation, RAFAIL reduces sensitivity to task-irrelevant scene variation. Offline, a VLM annotates successful demonstrations with task progress and relationship importance, which are used to learn point-cloud-based relationship representations without relying on policy-internal features. At runtime, relationship-specific OOD detectors evaluate these representations while relationship importance and task progress are predicted without VLM inference. RAFAIL requires no failure data and achieves 73.4% balanced accuracy across three real-world robotic manipulation tasks, outperforming the strongest evaluated OOD- and uncertainty-based baselines.

I. INTRODUCTION

RAFAIL addresses the challenge of distinguishing failure-relevant deviations from harmless differences during robotic execution. It monitors task-relevant object relationships, using offline VLM supervision and runtime relationship-specific OOD detection; it outperforms the strongest evaluated baseline across three real-world tasks.

A. Foundation Models for Failure Detection

  • VLMs can assess semantic task outcomes, but continuous inference adds runtime cost; OOD and uncertainty detectors are efficient but may flag harmless changes or miss consequential interaction errors.Some such detectors also rely on policy-internal features that may not be available across policy architectures.Source: §I · p. 1

I. INTRODUCTION

  • RAFAIL focuses detection on relationships between task-relevant entities, such as a gripper and object, rather than global observations or policy representations.Relationship-specific OOD scores indicate which interaction deviates from successful execution.Source: §I · p. 1
  • RAFAIL decomposes global observations into dedicated representations of individual task-relevant relationships for failure detection.The detector is task-progress-conditioned and integrates relationship importance to localize anomalous interactions.Source: §I · p. 1
  • RAFAIL uses a VLM offline to annotate successful demonstrations with task progress and relationship importance, enabling runtime estimates without VLM inference.Its relationship OOD detection is independent of policy-internal features.Source: §I · p. 1
  • RAFAIL was evaluated on three real-world robotic manipulation tasks, achieving 73.4% balanced accuracy versus 65.2% for the best evaluated baseline.Source: §I · p. 2

III. METHOD

The method formulation describes a 6-DoF arm controlled through observations and predicted action sequences. A separate failure predictor combines scene imagery with auxiliary relationship-importance and task-progress estimates to produce a failure signal.

A. Problem Formulation

  • A 6-DoF arm with a parallel-jaw gripper acts over discrete time steps, receiving workspace point-cloud and proprioceptive observations.Its end-effector command specifies a desired pose and an open-or-closed gripper state.Source: A. Problem Formulation · p. 2
  • Given the most recent K observations, the policy predicts a sequence of H future actions.The commanded pose is tracked by a low-level Cartesian impedance controller.Source: A. Problem Formulation · p. 2
  • The policy also exposes auxiliary estimates of task progress and object-relationship importance for use by the failure predictor.The online overview shows the policy predicting actions and auxiliary scores from point-cloud and proprioceptive observations.Source: A. Problem Formulation · p. 2, Figure 2 · p. 3
  • At each time step, an independent failure predictor receives the scene RGB and depth image together with relationship-importance and task-progress estimates, then outputs a failure signal.The online detector uses the image and auxiliary scores to segment and score each relationship through an OOD detector.Source: A. Problem Formulation · p. 2, Figure 2 · p. 3

B. Method Overview

RAFAIL uses VLM-generated task understanding to identify relevant object relationships and supervise relationship-importance and task-progress labelers offline. At deployment, the policy predicts these signals without running the VLM or labelers.

B. Method Overview

  • The offline VLM-generated pairwise preferences supervise importance and progress labelers, whose frame-wise outputs become targets for policy auxiliary heads.At deployment, the policy predicts relationship importance and task progress jointly with its action output; neither the VLM nor the labelers run online.Source: B. Method Overview · p. 3

C. Object-Relationship Representation

  • Task-relevant objects are identified from VLM-generated task descriptions, and all unique object pairs are treated as task-relevant relationships.Descriptions are refined across demonstrations to capture alternative object descriptors.Source: C. Object-Relationship Representation · p. 3

D. Relationship Importance Estimation

  • A VLM compares frame sequences to label relationship importance and task progress, including equal-importance and equal-progress comparisons.The importance labeler predicts a score for each relationship, and frame-wise labels are assigned using a moving average window.Source: D. Relationship Importance Estimation · p. 3

E. Task Progress Estimation

The task-progress labeler predicts frame-wise progress from complete demonstration sequences. Its construction produces continuous, increasing labels and its training comparisons are filtered to emphasize informative examples.

E. Task Progress Estimation

  • The progress labeler uses ResNet18 frame features, a BiLSTM for temporal features, and an MLP head to predict positive progress increments.The frame-wise predictions are converted into continuous progress labels in [0, 1] for identifying corresponding task stages across demonstrations.Source: E. Task Progress Estimation · pp. 3–4
  • The normalized temporal-weight construction yields continuously increasing progress labels from 0 to 1, independent of demonstration length.Predicted progress is averaged over the same four-frame windows used for VLM comparisons.Source: E. Task Progress Estimation · p. 4
  • Training filters equal-progress comparisons for cross-demonstration consistency and selects strict comparisons near retained equal-progress comparisons.A ranking loss handles selected strict comparisons, while equal-progress comparisons use a margin-based penalty; a quadratic jump penalty discourages progress increments concentrated in few frames.Source: E. Task Progress Estimation · p. 4

F. Relationship Representation Learning

RAFAIL learns a low-dimensional representation for each relationship by encoding the point cloud of its object pair. An importance-weighted contrastive objective separates matched relationship states from selected mismatches.

F. Relationship Representation Learning

  • A DGCNN encodes each object-pair point cloud into a low-dimensional latent vector, trained directly with an importance-weighted ranking contrastive loss.The loss encourages the anchor latent to be closer to a positive sample and farther from negative samples.Source: F. Relationship Representation Learning · p. 5
  • The contrastive loss uses cosine distance and a margin hyperparameter to compare the anchor, positive, and negative latent vectors.The selected negatives are intended to capture distinct, comparatively difficult relationship mismatches.Source: F. Relationship Representation Learning · p. 5
  • The positive sample is the same relationship at the same progress in another demonstration, while negatives represent three relationship or progress mismatches.The three negative cases include a different relationship at the same progress, or the same relationship at a different progress within or across demonstrations.Source: F. Relationship Representation Learning · p. 5

G. Relationship-Aware Failure Prediction

RAFAIL applies OOD detection to learned relationship latents, using independent detectors trained on successful demonstrations. Predicted relationship importance gates anomalous scores according to task stage.

G. Relationship-Aware Failure Prediction

  • OOD detection is applied to learned relationship latents, with an independent detector for each relationship trained on successful demonstrations.The framework uses RND, conditioned on current task progress with FiLM, for fast inference and comparatively high performance.Source: G. Relationship-Aware Failure Prediction · p. 5, Figure 4 · p. 5
  • An OOD score is combined with predicted relationship importance so that an anomaly is considered in light of its relevance at the current task stage.The importance threshold is set to 0.5, and an execution is classified as failed if the failure signal is triggered at any timestep.Source: G. Relationship-Aware Failure Prediction · p. 5
  • During deployment, Cutie object segmentation is initialized with GroundedSAM2 for real-time tracking.The paper states that Cutie offers improved real-time tracking capabilities compared with SAM3.Source: G. Relationship-Aware Failure Prediction · p. 5

A. Experimental Setup

The experiments evaluate RAFAIL on three manipulation tasks using successful teleoperated demonstrations for training and autonomous executions for evaluation.

  • The tasks are PickPlace, Uprighting, and Pouring, involving object placement, cup uprighting, and pouring a ball into a bowl.Source: A. Experimental Setup · pp. 5–6
  • Twelve successful demonstrations per task were collected on a UR10e with a Robotiq gripper and Azure Kinect; demonstrations were used only for training and calibration.Source: A. Experimental Setup · p. 6
  • ManiFlow was trained on successful task demonstrations and used in 178 task executions, with OOD situations induced by changing object placement and adding task-unrelated background objects.The executions comprised 60 each for Uprighting and Pouring and 58 for PickPlace.Source: A. Experimental Setup · p. 6

B. Baselines

The evaluation compares RAFAIL with real-time-capable OOD and uncertainty detectors, including pooled global observation representations.

  • Baselines include ACE and FIPER for uncertainty-based detection, plus lopO, logpZO, PCA-kmeans, and RND as OOD detectors.Global observation latents for the OOD baselines use mean- or max-pooling across observation tokens, with separate CP thresholds.Source: B. Baselines · p. 6

C. Failure Detection Performance

RAFAIL achieves the strongest overall balanced accuracy and maintains a favorable true-positive/false-positive trade-off, while its detections tend to align more closely with observed failures.

  • 73.4% overall balanced accuracy gives RAFAIL the highest Acc., ahead of ACE at 65.2%; RAFAIL’s overall TPR is 58.5% and FPR is 11.6%.Source: C. Failure Detection Performance · p. 6, Table II · p. 7
  • RAFAIL has the highest Acc. for Pouring and Uprighting, but ranks third in PickPlace at 66.7%, behind RND at 72.8% and logpZO with max-pooling at 68.0%.Source: C. Failure Detection Performance · p. 6
  • Figure 5 aligns first-detection times to manually annotated failure times; RAFAIL detections tend to be closer, especially for PickPlace and Pouring, while Uprighting has a broader distribution.The annotated failure timestep is a reference, and the timing distribution is normalized by the maximum duration of successful executions.Source: C. Failure Detection Performance · pp. 6–7, Figure 5 · p. 7
  • Among true-positive detections, RAFAIL’s triggering relationship agrees with manually annotated failure relationships in 93.3% of PickPlace, 69.3% of Pouring, and 68.0% of Uprighting cases.The authors note that these subjective annotations are not an independent ground truth.Source: C. Failure Detection Performance · p. 7

D. Ablations

The ablations indicate that relationship-wise encoding and importance gating contribute to RAFAIL’s performance; the conclusion bounds its scope to failures representable by pairwise relations among segmentable entities.

D. Ablations

  • Whole-scene encoding lowers balanced accuracy across all tasks; removing the importance gate raises overall FPR from 11.6% to 69.8% and reduces Acc. from 73.4% to 58.1%.Continuous importance weighting yields 68.9% overall Acc. and 37.8% TPR; the gate provides the more favorable evaluated trade-off.Source: D. Ablations · pp. 7–8, Table III · p. 8

V. CONCLUSION

  • RAFAIL assumes task-critical failures can be represented by pairwise relations among segmentable entities identified from successful demonstrations.Unmodeled entities, global scene properties, geometrically invisible contact forces, and segmentation or tracking breakdown may leave failures undetected.Source: §V · p. 8

Related papers

Loading 2609.18324v1…