PaperlayerView exact revision on arXiv

Source-linked AI summary

Hide-and-Seek in Trajectories: Discovering Failure Signals for VLA Runtime Monitoring

Seongheon Park, Wendi Li, Changdae Oh, Samuel Yeh, Zsolt Kira, Michael Hagenow, Sharon Li

arXiv:2605.30834v1cs.ROcs.AI

TL;DR

VLA failure detection must identify per-step failure signals from coarse trajectory-level outcomes, without step-level annotations. Hide-and-Seek uses inter- and intra-trajectory contrastive learning to localize failure-indicative actions, consistently outperforming baselines across simulated and real-world evaluations with a practical accuracy–timeliness trade-off.

  • Problem

    Per-step VLA failure detection lacks step-level supervision, creating a granularity mismatch with available trajectory-level outcome labels.

  • Method

    Hide-and-Seek combines inter-trajectory and intra-trajectory contrastive objectives to discover failure-indicative actions from trajectory-level supervision alone.

  • Results

    Up to +11.7% bACC over the strongest classifier baseline, +13.1% accuracy over a VLM monitor, and over 2,000× higher speed were achieved across seen and unseen tasks.

  • Takeaways & Limitations

    Hide-and-Seek provides a lightweight runtime monitor with state-of-the-art multi-task detection and a practical accuracy–timeliness trade-off.

  • Takeaways & Limitations

    Averaging action embeddings across degrees of freedom may discard kinematic structure, while action-level embeddings may miss inherently perceptual failures.

Abstract

from arXiv · show

Vision-Language-Action (VLA) models enable robots to follow natural language instructions and generalize across diverse tasks, but they remain vulnerable to execution failures that compromise reliability in real-world deployment. Detecting such failures during execution is therefore critical for the robust deployment of embodied systems. Existing failure detection methods either rely on expensive action resampling or external models, while alternatives propagate trajectory-level labels uniformly across every timestep, obscuring localized failure signals. In this paper, we propose \textbf{Hide-and-Seek}, a framework that formulates VLA failure detection as a coarsely supervised learning problem. By combining inter-trajectory and intra-trajectory contrastive objectives, Hide-and-Seek localizes failure-indicative actions and induces temporally structured failure signals from trajectory-level supervision alone, without any step-level annotation. We evaluate Hide-and-Seek on LIBERO, VLABench, and a real-world robotic platform across three representative VLA policies: OpenVLA, $π_0$, and $π_{0.5}$.Our method achieves state-of-the-art multi-task failure detection performance with a practical accuracy--timeliness trade-off under conformal prediction, and generalizes well to both seen and unseen tasks.

1 Introduction

Hide-and-Seek formulates VLA failure detection as coarsely supervised learning, using trajectory-level outcomes to localize failure-indicative actions and induce temporally structured signals. It addresses costly step-level annotation and uniform label propagation, achieving strong performance across simulation and real-world evaluations.

  • Motivation: VLA execution failures undermine robot reliability and increase the cost of real-world deployment.
  • Challenge: Existing methods face expensive step-level annotation, while uniformly propagating trajectory labels mislabels normal actions before failure onset.
  • Method: Hide-and-Seek discovers failure-indicative actions from trajectory-level supervision alone instead of uniformly labeling every timestep.
  • Evaluation: +11.7% bACC over the strongest classifier-based baseline is achieved across LIBERO, VLABench, and a real-world platform while maintaining a practical accuracy–timeliness trade-off.The evaluation uses OpenVLA, π0, and π0.5 and covers seen and unseen tasks.
  • Method: Its contrastive objective distinguishes failure-indicative from non-failure actions across and within trajectories, converting coarse labels into temporally structured failure signals.

2 Related Works

Related work spans VLA policy-generation paradigms, robot-manipulation failure detection, and coarse-supervision methods for localizing fine-grained signals. Existing approaches include autoregressive and generative-process policy learning, OOD-based failure detection, and weakly supervised localization paradigms.

  • VLA Policy Learning: VLA models follow two broad paradigms: autoregressive action generation and policy learning framed as a generative process.Autoregressive models produce actions token-by-token, analogously to language modeling.
  • Robot-Manipulation Failure Detection: Failure detection is critical because minor robot-manipulation errors can cascade into catastrophic outcomes during real-world deployment.Existing approaches are grouped into four categories, including OOD detection-based methods.
  • Robot-Manipulation Failure Detection: OOD detection-based methods treat successful executions as in-distribution and flag deviations as failures, but struggle to generalize to unseen distributions.The passage identifies OOD detection as the first of four categories of existing failure-detection approaches.
  • Coarse Supervision: Coarse supervision uses coarse-grained annotations to localize fine-grained signals and has been applied to object detection, video temporal action localization, and anomaly detection.Multiple instance learning assigns one label to a bag of instances and learns to identify responsible instances from bag-level supervision.

3 Problem Setup

Section 3 formulates VLA failure detection from trajectory-level success or failure labels into fine-grained, per-step decisions over action embeddings. The central problem is bridging coarse supervision and localized runtime detection.

  • Problem Setup: Each VLA timestep maps an observation to an action, with an internal action embedding h_t forming trajectories τ = (h_1, h_2, ..., h_T).Observations include RGB images, a natural-language instruction, and the current robot state.
  • Coarsely Supervised Failure Detection: Given successful and failed trajectories labeled y(τ) ∈ {0, 1}, the detector learns to identify failure-indicative steps from trajectory-level supervision.At runtime, it evaluates trajectory prefixes τ_≤t to produce a binary failure decision.
  • Problem Setup: The central challenge is the mismatch between trajectory-level supervision and per-step detection granularity.The framework is designed to bridge this supervision-detection gap.

4 Hide-and-Seek Failure Detection

Hide-and-Seek detects localized VLA execution failures from trajectory-level supervision by contrasting failure-indicative actions across and within trajectories. It converts coarse labels into temporally structured failure scores and deploys them with time-varying conformal thresholds.

  • Overview: Hide-and-Seek addresses failures hidden among mostly normal behavior by seeking failure-indicative actions using trajectory-level supervision instead of uniformly labeling every timestep.A trajectory may succeed until a localized stage, such as placement after grasping.
  • Inter-trajectory contrastive loss: The inter-trajectory loss contrasts the most salient failure-indicative action in a failure trajectory against the hardest false-positive action in a successful trajectory.It adaptively discovers the salient signal without assuming where failure occurs and imposes a margin between the compared scores.
  • Intra-trajectory contrastive loss: The intra-trajectory loss separates normal and failure phases by defining tonset as the sharpest increase in failure score and encouraging higher average post-onset than pre-onset scores.The sharpest score transition closely approximates annotated failure onset and performs comparably to an oracle onset reference.
  • Overall objective: A weighted combination of inter-trajectory and intra-trajectory losses converts coarse trajectory-level supervision into temporally structured failure signals without per-step annotation.The inter-trajectory term seeks hidden signals across trajectories, while the intra-trajectory term sharpens the temporal boundary between normal execution and failure.
  • Inference-time monitoring: At inference, the detector raises an alarm when the prefix failure score st exceeds a threshold ζt, with functional conformal prediction providing a time-varying threshold from successful calibration trajectories.The calibration approach addresses temporal score evolution and the risks of excessive false alarms or missed detections.

5 Experiments

Hide-and-Seek achieves state-of-the-art failure detection across simulated and real-world benchmarks, generalizing to unseen tasks while balancing detection accuracy and timeliness. Ablations show that contrastive objectives, temporal context, and latent action embeddings are central to its performance and efficiency.

  • Simulation environment: Hide-and-Seek consistently outperforms all baselines across LIBERO-10 and VLABench, achieving state-of-the-art bACC, wACC, and TWA on seen and unseen tasks.The evaluation covers OpenVLA, π0, and π0.5, with results averaged over 3 random seeds.
  • Real-world robot: Hide-and-Seek achieves the best TWA across all four real-world settings and gains +11.7% bACC and +15.0% TWA on unseen CUBE over SAFE-MLP.The real-world evaluation uses a UFactory xArm 6 with π0.5 across CUBE and KITCHEN task suites.
  • Detection accuracy–timeliness trade-off: Stricter α produces delayed detection, whereas looser α enables earlier detection at the cost of more false alarms, tracing an accuracy–timeliness operating curve.The sweep spans α ∈[0.01, 1.0] on unseen LIBERO-10 splits.
  • Loss component analysis: Inter-trajectory contrastive loss alone surpasses SAFE-MLP by +2.6% on OpenVLA and +2.1% on π0, whereas intra-trajectory loss alone yields degraded performance.The result indicates that inter-trajectory contrastive learning extracts failure-indicative signals under weak trajectory-level supervision.
  • Comparison with VLM-based runtime monitor: Hide-and-Seek exceeds the VLM-based monitor by +13.1% bACC while running at over 2,000× higher speed, at 0.001s versus 2.343s per step.The comparison uses LIBERO-10 with OpenVLA on a single NVIDIA A6000 GPU.
  • Comparison with uniform trajectory-level labeling: Hide-and-Seek outperforms uniform trajectory-level labeling by +19.4% bACC on OpenVLA and +18.6% on π0.The comparison uses otherwise identical settings, including the window size.

6 Conclusion … A.1 Failure Score Dynamics Across Splits

Hide-and-Seek is a lightweight VLA runtime failure detector that learns failure-indicative actions from trajectory-level supervision alone. Across training, seen, and unseen LIBERO-10 splits, its scores stay low for successful trajectories and rise clearly for failed ones.

  • 6 Conclusion: Hide-and-Seek detects VLA execution failures from trajectory-level supervision alone through coarse supervision.It is presented as a lightweight runtime failure detector.
  • 6 Conclusion: The inter-trajectory contrastive objective identifies critical failure-indicative actions.This objective is combined with an intra-trajectory objective in the proposed framework.
  • 6 Conclusion: The intra-trajectory objective sharpens score separation around the inferred failure onset.Together, the two objectives induce temporally structured failure signals from coarse supervision.
  • A.1 Failure Score Dynamics Across Splits: Average per-timestep failure scores were evaluated for OpenVLA and π0 on LIBERO-10 across training, seen, and unseen evaluation splits.OpenVLA results are shown in Figure 5 and π0 results in Figure 6.
  • A.1 Failure Score Dynamics Across Splits: Across all splits, successful trajectories maintain consistently low failure scores, whereas failed trajectories exhibit a clear upward trend.The visualization averages per-timestep scores across episodes and shows standard deviation in the plotted trends.
  • A.1 Failure Score Dynamics Across Splits: Although gradients propagate only through each successful trajectory’s maximum-scoring timestep, changing argmax identities distribute supervision across different timesteps during training.The passage states that this effectively distributes supervision across most timesteps of successful trajectories.

A.2 Qualitative Results

Hide-and-Seek’s failure scores align with observable physical failure events despite training without step-level supervision. Qualitative examples show conformal prediction declaring failure when the score first exceeds an adaptive, time-varying threshold.

  • Failure-score interpretation: Failure scores correlate with observable visual indicators and meaningful transitions in the robot’s physical state without step-level supervision.The onset point can align with subtle early indicators, such as a book beginning to slip from the gripper.
  • Failure-score interpretation: The onset point tonset captures subtle early failure indicators, whereas the peak point tmax aligns with the most salient critical event.Examples include a book starting to slip at onset and the robot dropping the object at the critical event.
  • Conformal prediction: Conformal prediction declares failure at the earliest timestep t where failure score st exceeds the time-varying threshold ζt.The threshold adapts to calibration-set variability, rising when successful trajectories vary more and falling when they are more consistent.

B Implementation Details … C.2 Thresholding Strategies

Hide-and-Seek uses a lightweight LSTM-based detector trained with staged contrastive optimization and calibrated through functional conformal prediction. Ablations show that the learned onset proxy and time-varying threshold improve detection robustness and timeliness across policies.

  • B Implementation Details: The detector is a single-layer LSTM with hidden size 256, trained for 300 epochs using AdamW at learning rate 5 × 10−4 with L2 regularization weight 0.1.Training uses Python 3.10.19, PyTorch 2.10.0, and a single 48GB NVIDIA A6000 GPU; training typically takes approximately 5 minutes.
  • B Implementation Details: Linter is trained alone until tonset stabilizes, after which Lintra is introduced; stopgradient ensures Lintra gradients flow through score values rather than onset-index selection.The onset index is computed as tonset = arg max_t sg(st − st−1), implemented with detach().
  • B.1 Hyperparameters: Hyperparameters are tuned in two stages: global settings are selected on LIBERO-10 with OpenVLA, followed by benchmark- and policy-specific tuning.Real-robot experiments use a shared hyperparameter set across tasks.
  • B.2 Conformal Prediction: Functional conformal prediction calibrates a time-varying threshold from successful trajectories and guarantees a trajectory-level false-alarm rate bounded by α under exchangeability.Calibration splits successful trajectories into profile-estimation and band-width-calibration subsets, comprising 30% and 70% of the calibration set, respectively.
  • B.3 Evaluation Metrics: Runtime evaluation treats failure trajectories as positives and successful trajectories as negatives, with alarms triggered when scores exceed the conformal threshold; TWA rewards earlier detections and penalizes delays.The reported metrics include balanced accuracy, weighted accuracy, and time-weighted accuracy.
  • C.1 Failure Onset Proxy: +0.016 for OpenVLA and −0.008 for π0 are tonset’s normalized deviations from the oracle onset, versus +0.107 and +0.068 for tmax.Using tonset as the Lintra anchor performs on par with the oracle and substantially better than tmax because the sharpest score transition aligns with failure onset.
  • C.2 Thresholding Strategies: Functional CP provides the most stable thresholding across OpenVLA and π0, while fixed and Split CP thresholds degrade noticeably under π0.OpenVLA is relatively robust across thresholding strategies, whereas π0 is more sensitive to threshold selection.

C.3 Layer-wise Ablation … D.2 Detection Timeliness Relative to Annotated Failure Onset

Hide-and-Seek’s analysis shows that layer-averaged or mid-to-late OpenVLA representations support robust failure detection, while uniform trajectory labels obscure localized failure signals. Across LIBERO-10 timeliness analysis, Hide-and-Seek alarms slightly before annotated onset and outperforms delayed or less accurate baselines despite coarse supervision.

  • C.3 Layer-wise Ablation: Mid-to-late OpenVLA layers, especially 20–31, generally provide stronger and more stable failure-detection performance than early layers.Performance does not have a sharp single-layer peak.
  • C.3 Layer-wise Ablation: The performance gap among well-performing layers remains within ∼3% bACC, while layer-averaged representations match or exceed the best individual layer.The results indicate that averaging across layers is a strong default rather than relying on one selected layer.
  • C.3 Layer-wise Ablation: The layer study finds a mild mid-to-late-layer advantage, leaving cross-architecture and cross-policy failure-discriminative layer analysis as future work.This limitation is framed alongside related interpretability findings about semantically rich intermediate representations.
  • D.1 VLA Latent Visualization: Isolating the failure phase produces more separable representations and supports automatically discovering failure-indicative actions without step-level annotations.The analysis motivates Hide-and-Seek’s coarse-supervision objective by showing that label propagation dilutes discriminative signals.
  • D.2 Detection Timeliness Relative to Annotated Failure Onset: Hide-and-Seek consistently raises alarms slightly before GPT-5.2-annotated failure onset across settings, enabling timely intervention without step-level supervision.The normalized lag is evaluated over true-positive failure trajectories and averaged across α ∈{0.15, 0.20, 0.25} and three random seeds.
  • D.2 Detection Timeliness Relative to Annotated Failure Onset: SAFE-MLP shows negative lag in three of four settings, including −0.272 on OpenVLA unseen, whereas SAFE-LSTM is timelier but has higher variance and lower accuracy.SAFE-MLP’s cumulative score and increasing target weight bias detection toward later timesteps.

D.3 Detector Architecture

The detector’s backbone ablation shows that temporal context is important for failure detection, while the method remains compatible with multiple sequence architectures. LSTM performs best among the tested context-aware backbones, especially on unseen tasks.

  • Architecture ablation: All architecture variants share the same input features, training objective, and schedule, differing only in backbone architecture.The comparison includes MLP, GRU, single-layer Transformer, and LSTM backbones.
  • Architecture ablation: The MLP performs noticeably worse than context-aware architectures, highlighting the importance of temporal context for VLA failure detection.GRU, Transformer, and LSTM all incorporate sequence context.
  • Architecture ablation: LSTM achieves TWA gains of 2.1 and 3.3 points over GRU and Transformer on seen OpenVLA tasks.These gains compare the LSTM against GRU and Transformer under the same ablation setting.
  • Architecture ablation: LSTM achieves TWA gains of 4.0 and 4.2 points over GRU and Transformer on unseen OpenVLA tasks.The larger gains on unseen tasks support LSTM as the strongest tested backbone in this ablation.

E Benchmark Details … F.1 Annotation Protocol

The experiments use held-out evaluation across LIBERO, VLABench, and real-robot tasks, with conformal thresholds calibrated from successful trajectories. Failure-onset annotations are generated offline by GPT-5.2 using a strict frame-level protocol for ablation analysis.

  • E Benchmark Details: Training uses the training set, while evaluation uses a held-out set for both simulation and real-robot experiments.Conformal thresholds use successful held-out trajectories from seen tasks for seen-task evaluation and from the seen-task evaluation split for unseen-task evaluation.
  • E.1 Simulation Setup: LIBERO-10 evaluates fine-tuned OpenVLA and π0 checkpoints on long-horizon tasks with diverse objects, layouts, and language instructions.VLABench evaluates fine-tuned π0.5 on diverse manipulation tasks.
  • E.2 Real-Robot Setup: Real-robot experiments use a UFactory xArm 6 across 8 CUBE and KITCHEN tasks, with 25 demonstrations per task for LoRA fine-tuning of π0.5.Rollouts randomize robot initial states and object configurations, and failures arise naturally from policy execution without human intervention.
  • F GPT-5.2-based Failure Onset Annotation: GPT-5.2 annotates failure onset for each failed rollout from the complete recorded video in an offline video question-answering setup.These annotations are used solely for ablation analysis, and the setup is less challenging than streaming-prefix online monitoring.
  • F.1 Annotation Protocol: The annotation goal is to pinpoint exactly one FAILURE START frame index.The protocol defines failure start as the earliest frame showing an irreversible mistake or clearly unrecoverable behavior that is not corrected.
  • F.1 Annotation Protocol: Annotators must provide 1–2 frame-referenced visual evidence cues and assign low confidence when no decisive change is visible.In that case, they select the earliest index where the failure signal is most apparent.
  • F.1 Annotation Protocol: The strict JSON output records confidence, evidence cues, failure type, recoverability before onset, and optional notes.Failure types include drop, wrong-object, collision, stuck/off-track, goal-violation, and other.

F.2 Human Agreement Analysis … G.2 Multi-Sampling-based Methods

The paper validates GPT-5.2’s failure-onset annotations through strong human agreement and compares diverse timestep-level baselines, including embedding-based, learned OOD, and multi-sampling methods. These baselines derive failure scores from action embeddings, sampled actions, or token likelihoods using distributional distance, uncertainty, and temporal-consistency measures.

  • F.2 Human Agreement Analysis: 90.7% of 304 ratings agreed with GPT-5.2’s predicted failure onset, while all four raters unanimously agreed on 75.0% of the 76 failed rollouts.Raters reviewed videos with GPT-5.2’s onset marked and allowed small temporal offsets.
  • G Baselines: Baseline methods operate on VLA action embeddings, sampled actions, or per-token likelihoods and produce a per-timestep failure score at evaluation.Embedding-level methods use successful and failed rollout embeddings from the training set.
  • G.1.1 Embedding Distance: Embedding-distance baselines compare each query embedding’s distances to successful and failed embedding sets, using their difference as the failure score.This extends standard OOD detection, which considers only successful embeddings, to use trajectory-level failure supervision.
  • G.1.1 Embedding Distance: Mahalanobis, cosine k-NN, and PCA-KMeans instantiate embedding-distance detection through directional deviation, local neighborhood structure, and nearest-cluster-centroid distance.Cosine k-NN is non-parametric, whereas Mahalanobis models each embedding set with a Gaussian.
  • G.1.2 Learned OOD Detector: Learned OOD detectors train separate models on successful and failed embedding sets, with RND using predictor–target mismatch and LogpZO using a flow-matching energy-style residual.Higher RND mismatch or LogpZO residual indicates that an embedding lies outside the modeled distribution or carries greater out-of-distribution risk.
  • G.2 Multi-Sampling-based Methods: Multi-sampling baselines draw N = 10 policy action samples per timestep and estimate uncertainty from cluster entropy, covariance spectral dispersion, temporal distribution drift, or action-chunk entropy.These methods operate directly on sampled policy outputs rather than intermediate action embeddings.
  • G.2 Multi-Sampling-based Methods: EigenScore computes log det(Σt + ϵI) in action space, where larger determinant values indicate greater variance across action directions.The action-space computation keeps EigenScore consistent with other multi-sampling baselines and directly reflects predicted-action variability.
  • G.2 Multi-Sampling-based Methods: STAC measures temporal action-distribution inconsistency with MMD, while ACE sums per-step histogram entropies; OpenVLA uses adjacent single-step MMD for STAC and single-step entropy for ACE.STAC is adapted for OpenVLA because it produces immediate actions without overlapping action chunks.

G.3 Classifier-based Methods … J Broader Impacts

The paper compares classifier-, token-uncertainty-, and VLM-based runtime monitors, while situating Hide-and-Seek among internal-state probing methods and discussing limitations, future applications, and deployment safeguards. These sections emphasize temporal localization, uncertainty modeling, perceptual and kinematic coverage, early intervention, and the risks of false alarms or missed failures.

  • G.3 Classifier-based Methods: Classifier-based methods predict per-timestep failure likelihood from VLA embeddings but uniformly propagate trajectory labels, obscuring temporal localization and introducing pre-onset label noise.SAFE variants differ in backbone and loss while sharing this coarse-supervision limitation.
  • G.3 Classifier-based Methods: SAFE-MLP accumulates independently predicted timestep scores, whereas SAFE-LSTM sequentially processes embeddings; both use class balancing and regularization, but SAFE-MLP’s temporal weighting is failure-agnostic.SAFE-MLP uses asymmetric L1 loss, while SAFE-LSTM uses uniformly applied binary cross-entropy.
  • G.4 Token Uncertainty-based Methods: Token uncertainty monitors aggregate action-token probabilities or entropies, with max scores emphasizing worst-case uncertainty and mean scores providing smoother overall estimates.Log-likelihood scores depend on sampled tokens, whereas entropy scores reflect the full predictive distribution.
  • G.5 VLM-based Method: The Qwen3-VL-8B-Instruct monitor performs online VQA over prior frames and classifies a trajectory as failed when any timestep is predicted as a failure.Its prompt asks for a JSON decision about whether the current frame shows failure.
  • H Extended Related Work: Hide-and-Seek extends internal-state probing by treating VLA action embeddings as carriers of task-success and impending-failure signals beyond raw actions or visual inputs.This positions the method within broader uncertainty- and probing-based hallucination-detection research.
  • I Limitations and Future Work: Averaging embeddings across degrees of freedom may discard joint-specific kinematic signals, while action embeddings alone may miss perceptual failures; combining visual observations is proposed.The limitation concerns failures expressed through specific joints, end-effector configurations, or inherently perceptual cues.
  • I Limitations and Future Work: Early warning signals could support corrective intervention, recovery after failure onset, and other downstream mechanisms that close the loop beyond detection.The passage specifically identifies intervention before irreversible errors and restoration to a viable state.
  • J Broader Impacts: Runtime failure detection may reduce risks of physical harm, property damage, and unintended interactions, but false negatives and false positives require complementary safeguards.The paper frames Hide-and-Seek as a monitoring component alongside hardware constraints or higher-level supervision, not a standalone reliability guarantee.
Loading 2605.30834v1…