PaperlayerView exact revision on arXiv

Source-linked AI summary

Learning Actionable Manipulation Recovery via Counterfactual Failure Synthesis

Dayou Li, Jiuzhou Lei, Hao Wang, Lulin Liu, Yunhao Yang, Zihan Wang, Bangya Liu, Minghui Zheng, Zhiwen Fan

arXiv:2603.13528v1cs.ROcs.CV

TL;DR

Robotic foundation models still lack reliable, executable recovery from manipulation failures, while existing data sources are costly, hazardous, or separated from reality by simulation gaps. Dream2Fix generates and verifies photorealistic counterfactual failures from successful demonstrations, then trains a VLM to predict failure types and trajectory corrections. Correction accuracy improves from 19.7% to 81.3%, with 46% zero-shot closed-loop recovery in physical deployment.

  • Problem

    Robotic models struggle to convert failure detection into precise recovery because paired real-world failure-correction data are scarce, costly, hazardous, and difficult to obtain across diverse tasks.

    Source: §I · p. 1

  • Method

    Dream2Fix perturbs actions in a generative world model to create photorealistic failure-correction rollouts, verifies them for validity and safety, and fine-tunes a VLM for structured recovery prediction.

    Source: §I · p. 2, 1) Failure Type Definition and Rollout… · p. 3

  • Results

    Correction accuracy improves from 19.7% to 81.3%, and zero-shot real-robot deployment achieves a 46% recovery rate.

    Source: §I · p. 2, C. Quantitative Results · p. 7

  • Takeaways & Limitations

    Dream2Fix enables failure reasoning that produces actionable trajectory corrections for real-world recovery rather than only binary failure judgments.

    Source: §I · p. 2, C. Quantitative Results · p. 7

Abstract

from arXiv · show

While recent foundation models have significantly advanced robotic manipulation, these systems still struggle to autonomously recover from execution errors. Current failure-learning paradigms rely on either costly and unsafe real-world data collection or simulator-based perturbations, which introduce a severe sim-to-real gap. Furthermore, existing visual analyzers predominantly output coarse, binary diagnoses rather than the executable, trajectory-level corrections required for actual recovery. To bridge the gap between failure diagnosis and actionable recovery, we introduce Dream2Fix, a framework that synthesizes photorealistic, counterfactual failure rollouts directly from successful real-world demonstrations. By perturbing actions within a generative world model, Dream2Fix creates paired failure-correction data without relying on simulators. To ensure the generated data is physically viable for robot learning, we implement a structured verification mechanism that strictly filters rollouts for task validity, visual coherence, and kinematic safety. This engine produces a high-fidelity dataset of over 120k paired samples. Using this dataset, we fine-tune a vision-language model to jointly predict failure types and precise recovery trajectories, mapping visual anomalies directly to corrective actions. Extensive real-world robotic experiments show our approach achieves state-of-the-art correction accuracy, improving from 19.7% to 81.3% over prior baselines, and successfully enables zero-shot closed-loop failure recovery in physical deployments.

I. INTRODUCTION

Dream2Fix addresses the gap between coarse failure diagnosis and executable recovery by generating verified counterfactual failures from successful real-world demonstrations. It trains a VLM on paired data to predict failure causes and trajectory-level corrections, improving correction accuracy and enabling zero-shot recovery.

Motivation

  • Existing approaches are limited by costly and hazardous real-world failure collection, low-fidelity feedback, simulation-to-real gaps, and coarse diagnoses.These limitations leave a persistent gap between failure detection and actionable recovery.Source: §I · p. 1

Method

  • Dream2Fix synthesizes photorealistic failure rollouts from successful demonstrations and filters them for task validity, visual coherence, and kinematic safety.Targeted action perturbations in a generative world model produce paired failures and deterministic trajectory-level corrections.Source: §I · p. 1, Figure 1 · p. 1

Results

  • 81.3% correction accuracy versus 19.7% for GPT-4o demonstrates Dream2Fix-VLM’s substantially stronger recovery outputs.The system also achieves a 46% zero-shot recovery rate during real-robot closed-loop deployment.Source: §I · p. 2

Contribution

  • The framework produces over 120,000 paired failure-correction samples for learning actionable recovery.The dataset is intended to mitigate the sim-to-real gap and provide denser supervision than prior datasets.Source: §I · pp. 1–2
  • Dream2Fix jointly predicts structured failure causes and precise trajectory-level corrections rather than only detecting anomalies.This failure-aware VLM directly maps visual anomalies to recovery actions.Source: §I · p. 2

II. RELATED WORKS

Prior work detects or reasons about manipulation failures, scales data through simulation, or improves general visual grounding, but generally lacks diverse, realistic, structured trajectory-level recovery supervision. Dream2Fix frames failure recovery as joint result classification, failure typing, and actionable correction from visual observations.

A. Failure Detection in Robotic Manipulation

  • Prior failure-reasoning systems provide coarse diagnoses while remaining constrained by unrealistic simulations, weak supervision, or small datasets, limiting generalization across manipulation settings.Several cited systems detect failure likelihood or reason about causes without supplying corrective actions.Source: A. Failure Detection in Robotic… · p. 2

B. Data Generation in Robotics

  • Simulation pipelines scale robotic data collection and provide fine-grained task control, but their lack of visual realism creates a large sim-to-real gap.World models motivate realistic action-conditioned generation, while physical plausibility remains challenging.Source: B. Data Generation in Robotics · p. 2

C. Robotic Vision-Language Models

  • Robotic VLMs improve spatial grounding, referential reasoning, or affordance prediction but provide limited failure-driven, trajectory-level correction.The cited systems are designed primarily for perception and planning rather than structured recovery from execution errors.Source: C. Robotic Vision-Language Models · p. 2

A. Problem Statement

  • The task takes an RGB video sequence and predicts manipulation outcome, failure type, and a structured actionable correction.Failure types include translation, weak close, force open, and delay close, while correction directives specify gripper actions or spatial adjustments.Source: A. Problem Statement · p. 2

B. Photorealistic Failure Generation

Dream2Fix defines reproducible manipulation failure modes and induces them through localized keyframe-level action perturbations in an action-conditioned world model. Each generated rollout receives structured labels for its outcome, failure type, and corrective action.

1) Failure Type Definition and Rollout Collection:

  • Four representative failure modes are defined: delay close, weak close, force open, and translation.They capture common gripper-control and spatial-accuracy errors that current robotic world models can stably induce and reproduce.Source: 1) Failure Type Definition and Rollout… · pp. 2–3
  • Dream2Fix uses Ctrl-World, an action-conditioned generator, to synthesize failures by perturbing robot actions rather than only task descriptions.This conditioning supports direct manipulation of execution behavior during rollout generation.Source: 1) Failure Type Definition and Rollout… · p. 3, Figure 2 · p. 3
  • Keyframe-local perturbations create continuous, visible deviations while increasing the probability of the intended failure.Temporal offsets induce delay close, reduced closure magnitude induces weak close, inverted gripper actions induce force open, and Gaussian x-y noise induces translation.Source: 1) Failure Type Definition and Rollout… · p. 3
  • Every generated video is annotated with a result label, failure type, and structured correction, enabling scalable recovery supervision without real-world failure collection.The structured annotations support subsequent prediction of actionable corrections.Source: 1) Failure Type Definition and Rollout… · p. 3, Figure 2 · p. 3

2) Structured Failure Validation:

Dream2Fix validates synthesized failures with complementary checks for task validity, visual quality, dynamics consistency, temporal coherence, and kinematic safety. Samples are retained only when they satisfy the required validation criteria.

2) Structured Failure Validation:

  • Four verifiers screen generated samples for task-failure validity, visual quality, dynamics consistency, and kinematic safety.A sample is retained only if it satisfies all verifier conditions.Source: 2) Structured Failure Validation · p. 3
  • The VLM verifier compares generated clips with reference successes to judge whether the intended failure occurs and whether severe visual artifacts are present.Qwen3-VL-8B produces task-failure and visual-quality judgments.Source: 2) Structured Failure Validation · p. 3
  • The IDM verifier compares visually inferred end-effector transitions with conditioning state differences and rejects samples whose discrepancy exceeds a calibrated threshold.The IDM is trained on successful demonstrations and evaluates generated rollout transitions.Source: 2) Structured Failure Validation · p. 3
  • The joint-pose verifier rejects clips with predicted joint-limit violations or excessive joint velocity or acceleration.Thresholds are calibrated on successful reference trajectories, removing visually plausible but unsafe rollouts.Source: 2) Structured Failure Validation · pp. 3–4
  • Point tracking evaluates temporal coherence using motion smoothness, visibility stability, local topology stability, and global continuity scores.The metrics are combined into a total score for dataset curation, and clips with insufficient tracking confidence are discarded.Source: 2) Structured Failure Validation · p. 4

3) Fix Suggestion Generation:

Dream2Fix converts each verified failure into deterministic, structured recovery supervision. The schema specifies failure outcomes, failure modes, and corrections that downstream controllers can parse.

3) Fix Suggestion Generation:

  • Each verified failure receives an actionable recovery generated deterministically from the injected perturbation parameters.The recovery uses a fixed key–value schema followed by a brief failure-and-correction summary.Source: 3) Fix Suggestion Generation · p. 4
  • Gripper recoveries specify the corrected command and temporal anchor, while translation recoveries specify axis-aligned directions and discrete step counts.The temporal anchor handles cases where a gripper failure makes the failed keyframe ambiguous.Source: 3) Fix Suggestion Generation · p. 5
  • Every sample contains a success/fail label, failure mode, and structured actionable correction mapped to robot control commands.A predefined schema keeps the output space stable and unambiguous, with an optional natural-language summary.Source: 3) Fix Suggestion Generation · p. 5

1) VLM Finetuning:

Dream2Fix-VLM is fine-tuned to jointly recognize manipulation failures and produce structured corrections, then maps those predictions into executable closed-loop robot actions. The deployment pipeline selects invocation times, converts corrections into control primitives, and can replay corrected trajectories.

1) VLM Finetuning:

  • Dream2Fix-VLM is fine-tuned on single-view manipulation videos to predict success, failure type, and key–value recovery corrections.The model adapts Qwen3-VL-8B-Instruct while updating language-model and multimodal-projector parameters and freezing the remaining components.Source: 1) VLM Finetuning · p. 5

2) Closed-Loop Recovery for VLA:

  • The closed-loop system processes recorded VLA trajectories, parses structured VLM outputs, and executes deterministic robot-control primitives to correct failures.This grounds visual failure reasoning in physical recovery actions.Source: 2) Closed-Loop Recovery for VLA · p. 5
  • An event-aligned rule with a fixed action budget determines when to invoke the VLM, including cases where the base policy never initiates a gripper-closing attempt.Translation corrections become robot-space deltas, while gripper corrections are issued at predicted times.Source: 2) Closed-Loop Recovery for VLA · p. 5
  • The physical evaluation workspace uses a Franka Research 3 arm, Franka Hand gripper, and fixed-view Intel RealSense D435i RGB-D camera.These components support real-world evaluation of the closed-loop recovery system.Source: Figure 3 · p. 5
  • After recovery, the system replays the edited trajectory, records states and actions, and adds recovered rollouts to the training database.For gripper failures, recovery may issue, repeat, or strengthen a closing command.Source: 2) Closed-Loop Recovery for VLA · p. 5

IV. EXPERIMENT

The experiments evaluate both structured VLM failure reasoning and physical closed-loop correction across generated and real-world benchmarks. Metrics cover recognition, semantic correctness, and recovery directives, while robot tests measure correction and end-to-end recovery.

A. Experiment Setup

  • Experiments assess offline VLM failure understanding and correction prediction alongside real-robot closed-loop recovery.The physical evaluation includes direct execution of predicted corrections and OpenVLA recovery during failures.Source: A. Experiment Setup · pp. 5–6
  • The real-world failure test set contains 100 trajectories across 10 manipulation tasks collected with a Franka Panda, RealSense D435i, and Franka Hand.A separate Dream2Fix test split contains generated failures with structured reasoning and recovery labels.Source: A. Experiment Setup · p. 5
  • VLMs receive a task instruction and execution clip, then predict success or failure, failure stage and type, and a structured correction.All models are evaluated under a unified prompt and structured output format.Source: A. Experiment Setup · pp. 5–6
  • The evaluation reports ROUGE-L, cosine similarity, binary success accuracy, LLM-based fuzzy match, and correction accuracy.Fuzzy match uses Qwen3-30B-Instruct ratings mapped to scores of 1.0, 0.5, or 0.0.Source: A. Experiment Setup · p. 6
  • Translation correction accuracy tolerates up to three-bin deviation in predicted magnitude, while gripper failures use a separate correction-accuracy definition.The translation metric represents corrections by axis direction and discretized magnitude.Source: A. Experiment Setup · p. 6

B. Data Quality Verification.

Dream2Fix verifies synthesized failures against visual, temporal, physical, and kinematic criteria, retaining rollouts whose behavior remains comparable to successful demonstrations.

  • The IDM comparison isolates generation-induced inconsistency beyond the learned model’s baseline prediction error.Successful demonstrations establish the reference error distribution before synthesized failures are evaluated.Source: B. Data Quality Verification · p. 6
  • The joint pose verifier flags synthesized cases when any joint exceeds p95-calibrated velocity or acceleration thresholds.Thresholds are calibrated from pooled successful-demonstration joint motions.Source: B. Data Quality Verification · p. 6
  • Synthesized failures remain close to successful demonstrations across smoothness, visual consistency, stopping, global consistency, IDM discrepancy, and joint-motion metrics.These comparisons indicate plausible end-effector transitions and joint motions under the verifier.Source: B. Data Quality Verification · p. 6
  • The verification suite therefore supports using the generated failures for recovery learning and evaluation by checking physical plausibility against real execution statistics.The reported verifier metrics remain in moderate and controlled ranges relative to successful demonstrations.Source: B. Data Quality Verification · p. 6

C. Quantitative Results

Dream2Fix-VLM is evaluated on a dedicated test set and a zero-shot real-world benchmark using shared structured outputs and multiple recovery metrics. It achieves the strongest correction performance in both settings, including substantial gains over GPT-4o and robust zero-shot accuracy on real-world data.

  • The evaluation covers Dream2Fix test-set and real-world zero-shot settings with text-similarity, match-based success, and correction-accuracy metrics.All models use the same structured output format matching the annotation schema.Source: C. Quantitative Results · pp. 6–7, Table III · p. 7
  • 0.813 correction accuracy surpasses GPT-4o’s 0.197 on the Dream2Fix benchmark, while Dream2Fix-VLM also achieves the highest binary success rate.Table III compares open- and closed-source baselines under the same evaluation protocol.Source: C. Quantitative Results · p. 7, Table III · p. 7
  • Several baselines attain high binary success but low correction accuracy, whereas Dream2Fix-VLM provides stronger recovery outputs rather than only failure judgments.This contrast separates recognizing success or failure from producing an executable correction.Source: C. Quantitative Results · p. 7
  • 0.472 correction accuracy is achieved on the real-world benchmark in strict zero-shot evaluation without real-world fine-tuning.The benchmark includes more varied appearances and execution conditions than the Dream2Fix test set.Source: C. Quantitative Results · p. 7

D. Downstream Robotic Experiments

Downstream experiments test whether Dream2Fix-VLM corrections execute successfully on real-robot failures, including failures from both a real-world benchmark and OpenVLA deployment. The conclusion reports that synthetic training supports zero-shot correction while identifying rollout realism as an improvement target.

D. Downstream Robotic Experiments

  • The downstream pipeline records a failed VLA trajectory, processes it with Dream2Fix-VLM, parses the predicted recovery, and executes robot control primitives.This deterministic mapping grounds structured model outputs into actionable corrections.Source: D. Downstream Robotic Experiments · p. 7
  • Dream2Fix-VLM corrections are executed in 50-run real-robot evaluations on failures from the Real-World Benchmark and OpenVLA deployment.Recovery is counted when executing the predicted correction leads to task success, and both settings use strict zero-shot application to real-world data.Source: D. Downstream Robotic Experiments · pp. 7–8, Table IV · p. 8

V. CONCLUSIONS

  • 41.9% of synthesized rollouts are removed and 58.1% retained, producing over 120k paired failure–correction samples for experiments.The retention ratio indicates that a substantial fraction still deviates from realistic visual and physical behavior.Source: §V · p. 8
  • Real-robot trials confirm that predicted corrections can recover from failures, while future work targets more physically consistent world-model rollouts.The conclusion identifies action-conditioned dynamics as the main route toward higher-fidelity data augmentation.Source: D. Downstream Robotic Experiments · pp. 7–8, §V · p. 8
Loading 2603.13528v1…