Foresight: Failure Detection for Long-Horizon Robotic Manipulation with Action-Conditioned World Model Latents

Haoran Zhang; Yifu Lu; Boyang Wang; Xuhui Kang; Yen-Ling Kuo; Zezhou Cheng; Mengdi Wang; Odest Chadwicke Jenkins

Source-linked AI summary

Foresight: Failure Detection for Long-Horizon Robotic Manipulation with Action-Conditioned World Model Latents

Haoran Zhang, Yifu Lu, Boyang Wang, Xuhui Kang, Yen-Ling Kuo, Zezhou Cheng, Mengdi Wang, Odest Chadwicke Jenkins

arXiv:2606.23085v1Jun 22, 2026cs.RO

TL;DR

Failure detection for long-horizon robotic manipulation is underexplored because failures can emerge ambiguously over extended, action-dependent trajectories without dense temporal labels. Foresight uses action-conditioned world-model latents and conformal calibration to monitor rollouts, achieving the best calibrated balanced accuracy across three simulated benchmarks and showing effective signals in real-robot experiments.

Problem
Failure detection remains underexplored for long-horizon robotic tasks, where trajectory meaning depends on action history and dense temporal annotations are typically unavailable.
Method
Foresight predicts action-conditioned world-model latents, feeds them to a causal Transformer detector, and calibrates time-varying thresholds using functional conformal prediction.
Results
Across all three simulated benchmarks, Foresight-Transformer achieved the best calibrated balanced accuracy: 0.94 ± 0.06 on LIBERO-Long, 0.80 ± 0.10 on ManiSkill-Long, and 0.78 ± 0.02 on BEHAVIOR-1K.
Takeaways & Limitations
Action-grounded world-model embeddings provide effective signals for scalable and policy-adaptable failure monitoring, particularly on longer-horizon manipulation tasks.
Takeaways & Limitations
Pretrained world-model computation and latency make on-device deployment challenging, while conformal guarantees depend on calibration and deployment distributions matching.

Abstract

from arXiv · show

Long-horizon tasks are common in real-world robotic deployments, yet failure detection for such tasks remains underexplored. Detecting failures in long-horizon robotic tasks is particularly challenging because failure onset is often ambiguous and dense temporal annotations are typically unavailable. We present Foresight, a failure detection framework that monitors manipulation trajectories using latent representations from an action-conditioned world model. Foresight is trained using only final task-level success or failure labels. By leveraging predictive world-model embeddings, our method provides a unified framework for failure detection across different policies. We further use functional conformal prediction (FCP) to calibrate detection thresholds adaptively. We evaluate Foresight with state-of-the-art vision-language-action policies in simulation on LIBERO-Long, ManiSkill-Long, and BEHAVIOR-1K, compare it against state-of-the-artfailure detection methods, and validate it on real robots with three long-horizon tasks on a ReactorX-200 arm and one task on a Franka arm. Our results suggest that action-conditioned world-model embeddings provide a scalable representation for reliable failure monitoring in long-horizon manipulation.

1 Introduction

Foresight detects both current and emerging failures in long-horizon robotic manipulation by using action-conditioned world-model latents that capture action- and stage-dependent task state. The framework is policy-interface-agnostic, trained with final task-level labels, and evaluated across simulation benchmarks and real-robot rollouts.

Motivation: Long-horizon failure detection must identify executions drifting toward failure, because visual states have different meanings depending on action history and task stage.Small deviations can accumulate over hundreds or thousands of steps and later become irreversible.
Method: Foresight uses latent representations from an action-conditioned world model to monitor task-relevant state cues, including spatial relationships, motion, interaction dynamics, and action-conditioned scene changes.The representations condense these cues into a small set of informative tokens for long-horizon monitoring.
Method: Foresight is a policy-interface-agnostic detector that feeds predicted action-conditioned latent features into a causal transformer failure detector.Its visual encoder is frozen, while the action-conditioned predictor is trained from scratch.
Evaluation: Foresight is evaluated on LIBERO-Long, ManiSkill-Long, and BEHAVIOR-1K simulations plus real-robot rollouts spanning tabletop manipulation and mobile household tasks.The simulated horizons range from hundreds of steps to BEHAVIOR-1K rollouts averaging more than 8,000 steps.
Training: The framework enables failure detection across vision-language-action policies using only final task success or failure labels.The contribution explicitly targets supervision without dense temporal failure annotations.

2 Related Work

Prior failure-detection methods monitor partial or complete robot rollouts using vision-language reasoning, confidence functions, or world-model latents. Related work also includes conformal calibration and foundation world models for predicting future physical observations.

Failure Detection: Failure detection identifies unsuccessful robot executions from partial or complete rollout observations using diverse monitoring signals.Vision-language methods detect manipulation failures and provide natural-language explanations, while ReDiffuser learns an RND-based confidence function for sampled decisions.
Failure Detection: Gauge uses compressed video world-model latents with conformal prediction thresholds to classify executions as success, known failure, or out-of-distribution anomaly.
Foundation World Models: Foundation world models learn general-purpose physical-dynamics representations or simulators from large-scale video data through generative future-observation prediction.Cosmos World Foundation Models support future video or world-state prediction conditioned on text, image, or video.

3 Problem Formulation

The problem is to detect whether a long-horizon manipulation rollout will eventually fail using only trajectory-level outcomes and information available before the next action chunk. Foresight formulates this as predicting a failure score and comparing it with a time-varying threshold during execution.

Task and observations: Long-horizon tasks require multiple subgoals, typically involving symbolic actions such as pick, place, open, and close.At timestep t, the robot receives image observation I_t and observation context c_t, consisting of the current image or a short recent-image history.
Policy execution: A policy π maps observation context c_t to an action chunk, executes H′ ≤ H actions, and then replans.H is the prediction horizon, while a_t+k|t denotes the action predicted for timestep t+k at replanning timestep t.
Supervision: Rollouts receive binary trajectory-level labels, with failure marked 1 and successful completion marked 0, without precise failure-timestep annotations.The setup assumes access only to final success or failure labels.
Online failure prediction: Failure detection learns a score from c_t and the policy’s next action chunk, triggering an alarm when the score exceeds a time-varying threshold δ_t.Foresight encodes observation context and action-conditioned future latents, then uses a causal sequence model for per-timestep scores and conformal calibration for δ_t.

4 Methodology

Foresight detects failures by combining action-conditioned world-model latents with a causal sequence detector that produces per-timestep scores from available trajectory history. Functional conformal prediction calibrates these scores into time-varying binary alarms with false-positive control under exchangeability assumptions.

Framework: Foresight combines the current observation context and policy-predicted action chunk through an action-conditioned world model to produce execution-aware latent features and per-timestep failure scores.A calibrated time-varying threshold converts each score into a binary failure alarm.
Framework: Unlike methods using policy-internal features, Foresight uses action-conditioned video-world-model features to capture execution-level cues and support cross-policy generalization.This design aims to reduce reliance on policy-specific artifacts.
World-model representations: V-JEPA 2-AC receives observation context and the proposed action chunk, producing observed and action-conditioned predicted latents that represent current observations and expected future states.The predicted latent is combined with a fixed sinusoidal positional encoding to form the timestep token.
Failure detector: A causal Transformer with positional encodings and masked self-attention predicts failure scores using only latent tokens available up to the current timestep.MLP and LSTM detector variants are compared experimentally.
Conformal calibration: FCP calibrates scores into a one-sided time-varying upper band δt = µt + ht using successful calibration rollouts, and controls the false positive rate at level α under mild exchangeability assumptions.A failure is declared when the score first exceeds the calibrated band.

5 Experiment

Foresight is evaluated on three long-horizon simulation benchmarks and real-robot manipulation settings using rollout-level ROC-AUC and balanced accuracy. It achieves strong simulation and real-world results, while cross-policy transfer is feasible but policy-dependent and asymmetric.

Evaluation settings: Experiments span LIBERO-Long, ManiSkill-Long, and BEHAVIOR-1K, covering varied horizons, environment complexity, robot embodiments, and policy sources.BEHAVIOR-1K additionally requires navigation and manipulation in larger household environments.
Evaluation protocol: Evaluation uses rollout-level ROC-AUC and balanced accuracy, with 3-fold cross-validation and threshold calibration across swept significance levels α.ROC-AUC aggregates the maximum per-timestep failure score over each trajectory, while balanced accuracy weights successful and failed rollouts equally.
Simulation failure detection: 0.94 ± 0.06 balanced accuracy on LIBERO-Long, 0.80 ± 0.10 on ManiSkill-Long, and 0.78 ± 0.02 on BEHAVIOR-1K are achieved by Foresight-Transformer.It also obtains ROC-AUC values of 0.84 ± 0.03 on ManiSkill-Long and 0.76 ± 0.02 on BEHAVIOR-1K.
Real-world rollout monitoring: Foresight-Transformer achieves the best ROC-AUC in three real-world settings: ReactorX / ACT at 0.93 ± 0.01, ReactorX / π0.5 at 0.87 ± 0.03, and Franka / GR00T N1.5 at 0.89 ± 0.10.Foresight-LSTM is consistently strong, whereas Foresight-MLP remains near chance at 0.50–0.59.
Cross-policy generalization: Cross-policy transfer is feasible, but asymmetric: training on π0.5 transfers well to ACT, whereas training on ACT transfers poorly to π0.5.π0.5 rollouts may include recovery behaviors that are absent from ACT-like rollouts, causing detectors trained only on ACT-like data to misclassify recovery as failure.

6 Conclusion

Foresight detects failures in long-horizon robotic manipulation by monitoring rollouts with action-conditioned world-model representations, using only trajectory-level success/failure labels. Its deployment is limited by world-model computation and latency, while conformal guarantees depend on calibration and deployment distributions matching.

Contributions: Foresight monitors long-horizon manipulation rollouts with action-conditioned world-model representations and causal failure detectors.The framework combines V-JEPA-style latent prediction with functional conformal calibration.
Contributions: The method requires only trajectory-level success/failure labels and no policy-internal states or uncertainty estimates.
Limitations: Pretrained world models impose computational cost and latency that challenge on-device deployment and highly reactive or agile tasks.
Limitations: Conformal calibration controls false alarms under held-out successful rollouts only when calibration and deployment distributions match.

Appendix · 7 More Implementation Details

The appendix specifies Foresight’s world-model feature extraction, predictor training setup, and three failure-detector architectures. The detectors operate on 1408-dimensional V-JEPA 2-AC latents using shared optimization settings while differing in temporal processing.

7 More Implementation Details: V-JEPA 2-AC provides the action-conditioned world-model features, with a frozen ViT-Giant visual encoder and a trainable predictor initialized from vjepa2-ac-vitg.pt.Images are resized to 256×256 and normalized with ImageNet statistics; the encoder uses 16×16 patches and tubelets of size 2.
7 More Implementation Details: The predictor combines teacher-forcing and n=2-step autoregressive rollout training, summing L1 losses on LayerNorm-normalized representations as L = LTF + LAR.Training uses AdamW with β1=0.9, β2=0.999, and weight decay 0.04.
7 More Implementation Details: For BEHAVIOR-1K and ManiSkill-Long, training uses 2×H200 GPUs with effective batch size 512 and the same learning-rate schedule.Real-world benchmarks use 2×H200 GPUs, effective batch size 32, and peak learning rate 5×10−5.
7 More Implementation Details: MLP, LSTM, and causal Transformer detectors share 1408-dimensional inputs, 2 layers, hidden dimension 256, learning rate 10−4, Adam, ℓ2 regularization λ=10−2, dropout 0.1, and 300 epochs.They are trained with batch size 512 on a single H200 GPU.
7 More Implementation Details: The MLP applies Linear→ReLU→Linear→Sigmoid independently at each timestep, whereas the LSTM processes the full episode sequence with a 2-layer recurrent core.The LSTM uses hidden dimension 256 and a Linear→Sigmoid output head, with dropout between layers and on the final hidden state.
7 More Implementation Details: The causal Transformer projects features to 256 dimensions, adds sinusoidal positional encodings, and uses 2 pre-norm encoder layers with 4 heads and feedforward dimension 1024.A causal mask restricts timestep t to features through t, and a final Linear→Sigmoid head produces per-step failure probabilities.

8 Data Splits and Calibration Protocol

The protocol randomly partitions rollouts into three equal-sized folds for 3-fold cross-validation, holding out one fold per round for testing. The remaining data are divided for detector training, validation, and conformal calibration, while the AC predictor uses all available non-test data.

Three equal-sized folds produce three experimental rounds under standard 3-fold cross-validation, with one fold held out for testing in each round.
The two non-test folds are split into training, validation, and calibration sets in a 6:1:1 ratio.Training fits the downstream detector, validation supports model selection and hyperparameter tuning, and calibration constructs time-varying conformal thresholds.
The AC predictor uses all available non-test data.

9 Conformal Prediction Thresholding

FCP sets a one-sided, time-varying threshold from the mean score trajectory of successful calibration rollouts plus a calibrated bandwidth. It declares failure when the score first exceeds this band and selects α per method and benchmark using held-out calibration and cross-validation.

Threshold construction: FCP defines the threshold as δ_t = μ_t + h_t, combining the mean score trajectory from successful calibration rollouts with a calibrated bandwidth term.The mean trajectory is estimated from successful rollouts in a held-out calibration set.
Threshold construction: A time-varying modulation term σ_t captures score deviations from the calibration mean, supporting the bandwidth construction through normalized nonconformity scores.Normalized nonconformity scores are computed for each calibration rollout.
Alarm rule: FCP declares a failure alarm at the first step where the score exceeds the time-varying threshold band.The alarm rule compares the score trajectory against the calibrated band.
Selection of α: The operating α is selected from 15 fixed candidates by maximizing balanced accuracy across three cross-validation folds for each method and benchmark.The selected value is computed from a dedicated calibration split and fixed before test reporting.

10 Baseline Implementation Details … Cost Decomposition and Deployment Implications

The paper supplements its evaluation with baseline implementation details, benchmark and ablation specifications, qualitative failure-monitoring examples, and runtime measurements. These details show that Foresight’s world-model backbone dominates monitoring cost, while its absolute latency fits the replan interval.

10 Baseline Implementation Details: Baselines use common train, calibration, and test splits where applicable, with thresholding methods calibrated on the same held-out rollouts as Foresight.Gauge uses released code and default hyperparameters but only its success model, trained and calibrated on successful rollouts; the best CP scoring variant is reported.
11 Additional Benchmark Details: LIBERO-Long evaluates 10 multi-stage tabletop tasks with 50 rollouts per task, while ManiSkill-Long evaluates four longer-horizon tasks and collects 319 valid rollouts.BEHAVIOR-1K evaluates four mobile-manipulation tasks with 400 total rollouts; successful π0-FAST rollouts average 93 policy calls and 1,484 simulation control steps in ManiSkill-Long.
11.4 Real-World Benchmarks: The real-robot evaluation spans ReactorX and Franka platforms, with SmolVLA reaching 65% success on arrange versus 22% for π0.5.ACT uses 40 episodes per task with 12 policy calls, while GR00T N1.5 uses 44 episodes for pick 3 toys and achieves 48% success.
12 Ablation Studies: Ablations compare world-model backbones and hidden versus action-conditioned predicted latents, with results indicating stronger failure-detection features from V-JEPA-style latent prediction.The motivation is that predicted latents expose action-conditioned deviations from expected dynamics rather than only visual anomalies.
13 Qualitative Results: Qualitative examples show calibrated thresholds remaining below failure scores for successful episodes and alarms crossing at t∗= inf{t: s_t ≥ δ_t} in failures.Examples span LIBERO-Long, ManiSkill-Long, BEHAVIOR-1K, and real-world ReactorX episodes, including dropped, misplaced, or ungrasped objects.
14 Runtime and Deployment: Runtime measurements use CUDA timing events on one NVIDIA H200 GPU, averaging 100 forward passes after 10 warm-up iterations.Foresight is invoked once per action-chunk boundary, so latency is evaluated per replan interval rather than per control step.