Source-linked AI summary
FLARE: A Failure-Aware Framework for Autonomous Correction and Recovery in Visual-Language Robotic Manipulation
Ganlong Zhao, Zijia Tang, Xingping Chen, Zhanghui Kuang, Ye Tian, Guanbin Li
TL;DR
VLAs trained on trajectory-monotonic, failure-free demonstrations remain brittle and lack recovery across common execution errors. FLARE addresses this with perturbation-and-bridging Retry augmentation for ID deviations and MLLM-guided Reset skills for OOD states, achieving an 84.0% average success rate across diverse manipulation tasks while remaining bounded by a predefined reset library and current hardware limitations.
Problem
VLAs trained on success-biased, trajectory-monotonic demonstrations lack robust recovery from execution errors and catastrophic states.
Method
FLARE combines Retry augmentation that decouples robot pose from environment state with MLLM-guided Reset skills for OOD failures.
Results
84.0% average success rate across diverse manipulation tasks is reported for FLARE, surpassing existing self-reflection methods and VLA backbones.
Takeaways & Limitations
FLARE treats failure recovery as a distinct learned capability for open-world robotic deployment.
Takeaways & Limitations
The current Reset mechanism relies on a predefined recovery-skill library, limiting handling of unseen complex failures requiring non-standard or multi-step actions.
Abstract
from arXiv · showhide
Vision-Language-Action Models~(VLAs) have demonstrated significant promise in generalizing to complex, long-horizon robotic manipulation tasks. However, their performance remains brittle, as they are typically trained on trajectory-monotonic, failure-free demonstrations. This reliance on ``perfect" data leaves them unable to recover from common execution errors, such as a missed grasp, a dropped object, or an unexpected collision. In this paper, we propose FLARE, a novel framework that endows VLAs with robust error recovery capabilities through a ``Retry" and ``Reset" paradigm. First, we introduce a ``Retry" mechanism by injecting perturbation and bridging segments that decouple robot pose from environment state into demonstrations, enabling the policy to autonomously handle execution deviations. Second, to address critical, state-breaking (OOD) failures, we introduce a ``Reset" pipeline. We leverage an MLLM for offline failure analysis to automatically identify OOD states from execution videos. This analysis enables the efficient, targeted collection of a small library of object-centric ``Reset" skills, which are trained to restore the environment to a task-valid state. Our full framework integrates these learned policies. At inference, an online MLLM monitor arbitrates between task execution and ``Reset" skills. Experiments on challenging, contact-rich manipulation tasks show our approach significantly improves task success and robustness.
1. Introduction
FLARE reframes brittle VLA execution as failure-aware resilience, using an ID/OOD taxonomy and complementary Retry and Reset mechanisms. Its MLLM-driven dual-loop system supports both learned retry robustness and closed-loop recovery.
- Motivation: VLAs remain brittle because failure-free, trajectory-monotonic demonstrations leave them unable to recover from perturbations, contacts, or execution deviations.The paper identifies success-biased, sparse demonstrations and narrow pose-progress correlations as central data-regime problems.
- FLARE Framework: FLARE introduces a unified Retry/Reset paradigm grounded in an ID/OOD error taxonomy.Retry targets minor, in-distribution deviations, while Reset targets catastrophic, out-of-distribution states.
- FLARE Framework: Retry uses perturbation and bridging augmentation to decouple robot pose from environment state and improve recovery from execution deviations.The augmented demonstrations expose VLAs to varied poses while reconnecting them to valid task progress.
- FLARE Framework: Reset uses MLLM-based offline failure analysis to acquire object-centric recovery skills for state-breaking failures.These skills restore the environment to a task-valid state before task execution resumes.
- FLARE Framework: An online MLLM monitor arbitrates between task execution and recovery modes in the integrated system.The framework combines original, retry-augmented, and reset-augmented data into a unified skill repertoire.
- Significance: FLARE advances VLA autonomy from imitation-driven execution toward resilience-driven agents that can act and recover.This is the paper’s stated overall consequence of treating failure recovery as part of robotic autonomy.
2. Related Works
Related work spans MLLM-based failure analysis, reinforcement learning recovery, and data-driven approaches within the broader evolution of VLA systems. Existing methods remain limited by fixed skill libraries, sample-efficiency demands, or incomplete coverage of recovery regimes.
- Robotic Self-Correction: MLLM-based approaches analyze failures and support replanning, but fixed skill libraries restrict low-level adaptability.REFLECT is cited as an example of this approach.
- Robotic Self-Correction: Reinforcement learning methods explicitly learn recovery behaviors but can require prohibitive amounts of real-world data.The paper cites SeRO as handling OOD states while retaining sample-efficiency challenges.
- Vision-Language-Action Models: VLA research unified perception, language, and control, progressing from RT-1 and RT-2 to OpenVLA, π0, and π0-FAST.These systems provide the embodied-AI foundation on which FLARE builds.
3. Methodology
FLARE models VLA failures through an ID/OOD taxonomy and trains complementary recovery capabilities. Perturbation-and-bridging augmentation builds Retry robustness for novel robot poses, while MLLM-guided reset-skill learning addresses catastrophic environment states.
- 3.1. Problem Formulation: ID errors preserve a valid environment state but place the robot in a low-probability pose, requiring Retry rather than environmental restoration.The taxonomy separates recoverable pose deviations from failures that invalidate the environment state.
- 3.1. Problem Formulation: A VLA policy predicts action chunks from the current visual observation and language instruction using a Markovian policy πθ.The policy is trained by imitation learning on successful human demonstrations.
- 3.1. Problem Formulation: Trajectory-monotonic demonstrations create spurious correlations between task progress and specific robot configurations, causing failures after valid-state perturbations.The paper attributes this brittleness to sparse, success-biased demonstrations and low-variance pose distributions.
- 3.1. Problem Formulation: OOD errors make the environment non-recoverable by the standard task policy, such as when an object topples, requiring a distinct Reset skill.Reset transforms the failed environment back into a valid task state.
- 3.2. Enabling Robust Retry via Perturbation & Bridging: Perturbation-bridging augmentation inserts random out-of-distribution motions followed by bridging actions that return the robot to the next valid task pose.This breaks pose-state coupling and creates diverse retry examples.
- 3.2. Enabling Robust Retry via Perturbation & Bridging: Trajectory composition stitches demonstrated subtask segments into baseline task trajectories for new randomized scenes.The composed trajectory is then enhanced with perturbation-bridging segments.
- 3.2. Enabling Robust Retry via Perturbation & Bridging: Bridging-to-task sequences, rather than perturbation actions themselves, are used as training targets to build the Retry capability.The augmented dataset combines original successful subsequences with bridging-to-task subsequences.
- 3.3. Learning Reset Skills via MLLM-driven Failure Mining: Retry cannot solve OOD failures because success-centric demonstrations omit catastrophic states and their recovery procedures.FLARE therefore mines failures with an MLLM and learns object-centric Reset skills.
4. Experiment
Experiments evaluate FLARE against several policy baselines on nine contact-rich RoboMimic tasks and in two real-world manipulation settings. FLARE achieves state-of-the-art performance on eight of nine simulated tasks and reaches an 84.0% average success rate, while also being validated on real hardware.
- Simulation experiments: Experiments cover 9 contact-rich RoboMimic manipulation tasks and compare FLARE with open-source VLA, diffusion-policy, and self-reflection baselines.The VLA backbone is π0.5.
- Simulation experiments: FLARE achieves state-of-the-art performance on 8 out of 9 tasks and surpasses Phoenix on the remaining Threading D0 task.On Threading D0, it also substantially improves over the π0.5 backbone.
- Simulation experiments: 84.0% average success surpasses Phoenix’s 57.8% and improves over π0.5 alone by 11.8%.FLARE also outperforms Phoenix-Human, which receives correct human guidance.
- Simulation experiments: Performance gains are more pronounced on D1 than D0 task variants, consistent with improved robustness to environmental variation.The D suffix denotes object-randomization range and therefore task difficulty.
- Real-world validation: Real-world validation uses a Piper arm and RealSense D435i on Stack Three Blocks and Insert U-shaped Block, with success rates reported in Table 2.Only 10 human demonstrations were collected and augmented to 50 per task or reset skill; Any6D supplied object pose estimates for augmentation.
5. Ablation and Analysis
The analyses show that perturbation settings trade off task performance against demonstration-generation efficiency, while reset skills and MLLM-based failure analysis materially support recovery.
- Perturbation & Bridging: The best perturbation and bridging performance occurs at r = 30◦ and t = 0.7.Larger perturbations increase variance and task success but reduce generation efficiency; excessively large perturbations degrade performance.
- Reset Ablation: Removing reset skills reduces average success rate by 3.5%, while human-generated reset instructions improve performance by an additional 7%.The comparison includes Ours, Ours w/o Reset, Ours Reset-Only, and Ours-Oracle on Coffee and ThreePieceAssembly.
- Reset Skill Learning: Resetting the coffee machine lid reaches 84% success and is easier than resetting the coffee pod; the same difficulty pattern holds for T-shaped versus U-shaped blocks.Coffee-pod resets often require grasping a toppled pod, adjusting its pose, and placing it upright.
- Reset Skill Learning: Reset-task demonstration-generation efficiency decreases as reset difficulty increases, matching the success-rate trend.The results indicate room for further augmentation and model training.
- Failure Identification: Gemini-2.5-Pro reaches 88% and 96% accuracy for retry/reset classification on Coffee and ThreePieceAssembly, respectively.Reset-object identification achieves 88% and 78% accuracy across the two tasks.
6. Conclusion
FLARE treats failure recovery as a learned capability by combining perturbation-based retry with MLLM-guided OOD reset skills. It achieves an 84.0% average success rate, while hardware limits correction of highly complex object poses.
- Conclusion: FLARE combines perturbation-based augmentation for ID retry capabilities with an MLLM-guided pipeline for OOD reset skills.The framework uses a dual Retry/Reset paradigm.
- Conclusion: FLARE achieves an 84.0% average success rate across diverse manipulation tasks, surpassing existing self-reflection methods and VLA backbones.
- Conclusion: Current hardware limits correction of highly complex object poses.Future work targets larger reset-skill libraries and more dexterous recovery maneuvers.
1. Implementation Details
The implementation expands small human demonstration sets with perturbation and reset augmentations, trains π0.5 with LoRA, and evaluates success over repeated trials.
- Data Generation and Augmentation: Each task starts from 10 human demonstrations and is expanded to 500 augmented demonstrations using MimicGen.
- Data Generation and Augmentation: Retry augmentation injects perturbation and bridging sequences with maximum 45◦ rotation and 0.5 meters translation.
- Reset Skill Collection: Reset skills are object-centric, with the number of skills determined by the manipulable objects involved.The implementation excludes unrecoverable cases such as objects too large to reorient or outside the robot’s reachable workspace.
- Training Details: LoRA fine-tunes the language model and action expert while the remaining model parameters stay frozen.Training uses Adam with a constant learning rate of 2.5 × 10^-4.
- Evaluation Protocol: Each task is evaluated over 50 trials, with the average success rate reported.
2. Manipulation Tasks
The evaluation covers nine manipulation tasks, with reset skills visualized for failure recovery; a ThreePieceAssembly example shows reset followed by successful retry.
- Manipulation Tasks: The evaluation includes nine manipulation tasks and identifies reset objects for demonstration collection.
- Manipulation Tasks: Figure 6 visualizes reset cases across tasks, excluding Stack because reset is not applicable.
- Manipulation Tasks: In ThreePieceAssembly, an incorrectly placed T-shaped block triggers a reset before the robot retries and completes the task.The robot places the T-shaped block aside to restore its pose before retrying.
3. Prompt for Video Analysis and Failure Detection
Tables 6 and 7 provide the prompts used for video analysis and closed-loop failure detection during inference.
- The section documents prompts for video analysis and failure detection during closed-loop inference.
4. Broader Impact and Future Work
FLARE is positioned as a route toward more autonomous, reliable robotic deployment, while its broader use requires safety oversight and further validation of scalability and generalization.
- Broader Impact: FLARE aims to improve robotic autonomy and reliability for dependable real-world deployment.
- Broader Impact: The framework could reduce human intervention and improve efficiency and safety in industrial and service robotics.
- Safety and Ethics: Recovery actions must follow predefined safety constraints and retain human oversight through interpretability and emergency override controls.
- Future Work: Current Reset relies on a predefined recovery-skill library, limiting handling of OOD failures requiring non-standard or multi-step collaborative actions.
- Future Work: Future validation targets dynamic environments, deformable objects, and long-horizon tasks involving human collaboration.