PaperlayerView exact revision on arXiv

Source-linked AI summary

FailureSpot: Label-Efficient Timestamp-Level Failure Detection for Vision-Language-Action Models

Jie Ma, Zongxi Liu, Yi Zhu

arXiv:2609.04277v1cs.ROcs.CV

TL;DR

Trajectory-wide labels can misclassify normal pre-failure behavior, while direct timestamp annotation is costly. FailureSpot uses action-derived weak supervision and uncertainty-based active learning; experiments report improved trajectory-level and timestamp-level detection across three VLA policies.

  • Problem

    Trajectory-wide supervision can label normal pre-failure behavior as failure, while accurate timestamp-level labels require costly dense inspection.

    Source: Introduction · p. 2, Related Work · p. 3

  • Method

    FailureSpot pretrains a detector with weak signals from unlabeled action chunks, then selects uncertain trajectories for timestamp annotation and fine-tuning.

    Source: Introduction · p. 2

  • Results

    Across three VLA policies, FailureSpot consistently improves both trajectory-level and timestamp-level failure detection over existing baselines.

    Source: Introduction · p. 2

  • Takeaways & Limitations

    The authors identify timely failure intervention and recovery as future uses of the timestamp-level detector.

    Source: Conclusion · p. 7

    Hide-and-Seek’s primary detection accuracy is evaluated at the trajectory level rather than through fine-grained timestamp classification.

    Source: Related Work · p. 3

Abstract

from arXiv · show

Vision-language-action (VLA) policies have shown strong potential for general-purpose robotic manipulation, but they can still fail unpredictably during long-horizon execution, making reliable failure detection essential for safe deployment. Existing methods either rely on visual models that typically detect failures only after erroneous actions have occurred, or use lightweight proactive detectors trained on VLA internal representations. However, these proactive methods are often supervised with trajectory-level labels, causing normal pre-failure behavior in unsuccessful trajectories to be incorrectly labeled as failure. This supervision mismatch introduces label noise and limits both trajectory-level detection accuracy and precise timestamp-level failure localization. In this work, we study fine-grained timestamp-level VLA failure detection while addressing the cost of dense annotation. We propose a data-efficient framework that first leverages unlabeled VLA action chunks to construct action-derived weak supervision signals, capturing abnormal patterns such as inconsistent consecutive chunks, frozen or idle actions, and aggressive random motions. We then use active learning to select only the most uncertain trajectories for timestamp-level annotation and fine-tune the detector with these informative labels. Experiments across multiple VLA policies show that our method improves both timestamp-level and trajectory-level failure detection performance.

Introduction

VLA policies can fail unpredictably during long-horizon execution, while trajectory-wide failure labels can mislabel normal pre-failure behavior and impair timestamp-level localization. FailureSpot combines action-derived weak supervision and active learning to reduce dense annotation needs; experiments report improved detection across three VLA policies.

Introduction

  • Reliable failure detection matters because VLA policies can miss targets, become stuck or idle, or produce unstable actions during long-horizon execution.The stated motivation is safe and reliable deployment, including cases requiring intervention, recovery, or stopping execution.Source: Introduction · p. 1
  • Trajectory-level labels mark normal pre-failure timestamps as failures, introducing label noise that can degrade trajectory detection and obscure failure onset.SAFE may therefore raise alarms too early or too late relative to ground-truth onset.Source: Introduction · pp. 1–2
  • Fine-grained timestamp labels provide a more accurate learning signal, but require labor-intensive, costly inspection of robot trajectories.The challenge is minimizing human labeling while retaining accurate timestamp-level detection.Source: Introduction · p. 2
  • FailureSpot pretrains from unlabeled action chunks using weak signals for chunk inconsistency, idle or frozen actions, and aggressive random motions, then actively selects uncertain trajectories for timestamp annotation.The detector is fine-tuned on the selected labeled subset to learn more accurate failure boundaries while reducing annotation cost.Source: Introduction · p. 2
  • Across three VLA policies, FailureSpot consistently improves both trajectory-level and timestamp-level failure detection over existing baselines.Ablations attribute complementary benefits to action-derived weak supervision and uncertainty-based active learning.Source: Introduction · p. 2

Related Work

Related work spans reactive vision-based detection and proactive methods that use information available before action execution. The paper positions its approach as addressing the limited direct timestamp-level supervision and evaluation in prior work.

Vision-language-action policies

  • VLA policies map language instructions and visual observations to actions, using either autoregressive action tokens or chunks of future actions.Examples include OpenVLA, π0, and π0-FAST, which differ in how they represent or generate actions.Source: Related Work · p. 2

Failure detection for VLAs

  • Reactive vision-based methods assess execution frames, but generally detect failures only after behavioral consequences appear and can incur high per-step model-query costs.The cited examples include methods for judging task progress (Du et al. 2023; Grislain et al. 2025; Lin et al. 2025) and explaining failures (Duan et al. 2024).Source: Related Work · p. 2
  • Proactive approaches include costly VLM predictors and lightweight detectors using action chunks or learned representations for alarms before execution.Examples include STAC (Agia et al. 2024), ActProbe (Huang et al. 2026), FAIL-Detect (Xu et al. 2025), SAFE (Gu et al. 2026), and Foresight (Zhang et al. 2026b).Source: Related Work · pp. 2–3
  • Many prior detectors train timestamp scores using trajectory outcomes at every step, mislabeling normal pre-failure behavior; Hide-and-Seek (Park et al. 2026) localizes signals but primarily evaluates trajectory-level accuracy.The paper’s method instead explicitly trains and evaluates at the timestamp level, using action-derived weak supervision and active annotation of uncertain trajectories.Source: Related Work · p. 3

Method

The method frames VLA failure detection as timestamp-level classification using policy representations available before actions execute, with failure onset marking the first timestamp labeled as failure. It reduces the cost of dense annotation by combining action-derived weak supervision, targeted trajectory annotation, and conformal thresholding.

Problem Setting

  • The detector maps each pre-action VLA representation to a timestamp-level failure probability, treating failure detection as binary classification.Both MLP and LSTM are explored as lightweight detectors.Source: Problem Setting · p. 3

Timestamp-level failure annotation

  • Failure labels mark timestamps where the robot stops making meaningful task progress, and onset is the first timestamp labeled as failure.Examples include missing the target, becoming stuck or idle, unpredictable movement, and actions inconsistent with the task.Source: Timestamp-level failure annotation · p. 3

Framework overview

  • A two-stage framework first pretrains on action-derived weak signals without human timestamp labels, then annotates uncertain trajectories and fine-tunes with their timestamp labels.The selected labels are used for supervised binary classification while limiting human labeling effort.Source: Framework overview · p. 3, Fine-tuning with active learning · p. 4

Pretraining with weak supervision

  • Action-chunk inconsistency and action magnitude provide weak signals: mismatched overlapping actions suggest instability, while unusually small or large magnitudes may indicate idle or aggressive behavior.The method smooths each signal over time and normalizes the combined score into a soft timestamp-level pseudo-label.Source: Pretraining with weak supervision · pp. 3–4

Fine-tuning with active learning

  • The detector estimates timestamp uncertainty with prediction entropy, averages it across each trajectory, and requests labels for the highest-uncertainty trajectories.This selects complete trajectories because timestamp-level annotation is more practical at trajectory granularity.Source: Fine-tuning with active learning · p. 4

Conformal prediction

  • A one-sided, time-varying conformal band calibrated on held-out successful trajectories from seen tasks sets the alarm threshold; scores at or above it indicate failure.The band adapts to temporal variation in scores during normal execution.Source: Conformal prediction · p. 4

Evaluation

With a 15% timestamp-level annotation budget, the method achieves the highest timestamp-level AUROC across three VLA policies. Evaluation also examines onset timing, ablations, data scaling, qualitative detections, and recurring failure modes.

Experimental settings

  • Evaluation uses 500 LIBERO-10 trajectories from three VLA policies, with timestamp labels marking deviations and recovery intervals across seen and unseen tasks.The timestamp-annotation budget is 15% of seen-task trajectories; metrics include timestamp- and trajectory-level AUROC and timestamp-level balanced accuracy.Source: Experimental settings · pp. 4–5

Overall Performance

  • With only 15% timestamp-level annotation, the method achieves the highest overall timestamp-level AUROC across all three VLA policies.It also achieves the best trajectory-level AUROC on π0 and π0-FAST, while performing slightly worse on OpenVLA.Source: Overall Performance · p. 5, Table 1 · p. 6

Accuracy and Failure-Onset Time

  • At the α yielding highest balanced accuracy, alarms precede the average ground-truth failure onset while balanced accuracy remains high across α values.Figure 2 plots timestamp-level balanced accuracy against average normalized detected onset time on failed trajectories.Source: Accuracy and Failure-Onset Time · p. 5, Figure 2 · p. 5

Ablation Study

  • Action-derived pretraining alone is insufficient; timestamp-level supervision substantially improves performance, and active learning further improves the limited annotation budget's effectiveness.The full framework has the best timestamp-level AUROC across all three policies and strongest trajectory-level performance on π0 and π0-FAST.Source: Ablation Study · p. 6, Table 2 · p. 7

Impact of training data amount

  • Increasing either pretraining data or timestamp-level annotation improves performance, with substantially larger gains from timestamp-level annotations.The main setting—60% of trajectories for pretraining and 15% timestamp-level annotation—is reported as a favorable labeling-cost and performance trade-off.Source: Impact of training data amount · p. 6, Figure 3 · p. 6

Visualization of timestamp-level failure detection

  • In two π0 examples, failure scores stay low during normal actions and rise above threshold near detected onsets shortly before annotated onsets.The corresponding frames show progress toward the task before onset and behavior leading to failure afterward.Source: Visualization of timestamp-level… · p. 6, Figure 4 · p. 7

Failure mode discovery

  • Recurring failure modes are repeated ineffective actions, near-zero-magnitude stagnation, and large motions with less-consistent consecutive predictions.π0 commonly repeats unsuccessful grasp attempts, π0-FAST more often swings excessively, and OpenVLA frequently stagnates; the modes motivate magnitude and temporal-consistency signals.Source: Failure mode discovery · pp. 6–7

Conclusion

The study targets detecting both whether VLA trajectories fail and when failure begins. FailureSpot improves timestamp-level and trajectory-level detection, with more accurate and timely alarms.

Conclusion

  • The study targets identifying both whether a VLA trajectory fails and when its failure begins.Source: Conclusion · p. 7
  • Figure 5 marks normal and failure phases and presents representative failure modes for π0, π0-FAST, and OpenVLA.Source: Figure 5 · p. 7
  • Experiments show FailureSpot improves timestamp-level and trajectory-level detection, with more accurate and timely failure alarms.Source: Conclusion · p. 7
  • Future work will explore using timestamp-level failure detection for timely intervention and recovery.Source: Conclusion · p. 7
Loading 2609.04277v1…