Source-linked AI summary
AHA: A Vision-Language-Model for Detecting and Reasoning Over Failures in Robotic Manipulation
Jiafei Duan, Wilbert Pumacay, Nishanth Kumar, Yi Ru Wang, Shulin Tian, Wentao Yuan, Ranjay Krishna, Dieter Fox, Ajay Mandlekar, Yijie Guo
TL;DR
Robotic systems need to detect and reason about failures, but VLMs and LLMs often struggle with this capability in dynamic manipulation settings. AHA frames failure detection as free-form reasoning, using FailGen-generated data to produce explanations and generalize across robots, environments, and tasks. It outperforms existing VLMs across diverse evaluation datasets and improves downstream manipulation policies.
Problem
Robotic VLMs and LLMs support spatial reasoning and task execution but remain limited in detecting and reasoning about failures in dynamic manipulation environments.
Method
AHA is an open-source VLM trained with FailGen-generated robotic failure data to predict sub-task success and provide free-form natural-language explanations for failures.
Results
AHA outperformed GPT-4o by more than 20.0% on average across datasets and metrics, while generalizing across embodiments, unseen environments, and novel tasks.
Takeaways & Limitations
AHA’s natural-language failure feedback improves error recovery and policy performance when integrated into manipulation frameworks.
Takeaways & Limitations
AHA’s language reasoning remains closely aligned with failure scenarios represented in its fine-tuning data, limiting coverage of failures outside the taxonomy.
Abstract
from arXiv · showhide
Robotic manipulation in open-world settings requires not only task execution but also the ability to detect and learn from failures. While recent advances in vision-language models (VLMs) and large language models (LLMs) have improved robots' spatial reasoning and problem-solving abilities, they still struggle with failure recognition, limiting their real-world applicability. We introduce AHA, an open-source VLM designed to detect and reason about failures in robotic manipulation using natural language. By framing failure detection as a free-form reasoning task, AHA identifies failures and provides detailed, adaptable explanations across different robots, tasks, and environments. We fine-tuned AHA using FailGen, a scalable framework that generates the first large-scale dataset of robotic failure trajectories, the AHA dataset. FailGen achieves this by procedurally perturbing successful demonstrations from simulation. Despite being trained solely on the AHA dataset, AHA generalizes effectively to real-world failure datasets, robotic systems, and unseen tasks. It surpasses the second-best model (GPT-4o in-context learning) by 10.3% and exceeds the average performance of six compared models including five state-of-the-art VLMs by 35.3% across multiple metrics and datasets. We integrate AHA into three manipulation frameworks that utilize LLMs/VLMs for reinforcement learning, task and motion planning, and zero-shot trajectory generation. AHA's failure feedback enhances these policies' performances by refining dense reward functions, optimizing task planning, and improving sub-task verification, boosting task success rates by an average of 21.4% across all three tasks compared to GPT-4 models.
1 Introduction
Robotic foundation models can execute open-world tasks but still struggle to detect and reason about failures. AHA addresses this gap with free-form failure reasoning, trained on procedurally generated data and evaluated across diverse settings.
- Motivation: The paper targets autonomous failure detection and reasoning because robots must understand why they failed before they can learn from mistakes.The motivation connects failure reasoning in robotics with broader learning-from-feedback paradigms.
- Motivation: VLMs and LLMs support spatial reasoning, reward generation, task planning, and zero-shot trajectory generation, but often struggle with failure recognition.This limitation is especially consequential for robotic manipulation in dynamic and complex environments.
- AHA: AHA frames failure detection as free-form reasoning, identifying failures and generating detailed natural-language explanations.The approach is designed to adapt across robots, camera viewpoints, tasks, and simulated or real-world environments.
- Data generation: FailGen procedurally generates failure demonstrations and produces the AHA dataset with over 49K image-query pairs across 79 simulated tasks.The pipeline perturbs successful demonstrations and integrates with different simulators for scalable data generation.
- Evaluation: AHA generalizes across embodiments, unseen environments, and novel tasks while outperforming open-source and proprietary VLMs.The evaluation covers three out-of-distribution datasets and four metrics.
3. We show that AHA enhances downstream robotic applications by providing failure reasoning
AHA supplies natural-language failure feedback to downstream robotic systems that use VLMs or LLMs for reward generation, planning, and sub-task verification. Across three downstream tasks, this feedback improves task performance over GPT-4 models.
- Downstream applications: AHA improves reward functions through Eureka reflection, enhances task and motion planning, and verifies sub-task success in zero-shot manipulation.These integrations use AHA's failure feedback within existing VLM- and LLM-based reasoning pipelines.
- Downstream applications: AHA is designed to enhance robotic applications using VLMs for reward generation, task planning, or sub-task verification.Figure 1 presents these as downstream uses of the model's failure reasoning.
- Results: 21.4% higher average success rate than GPT-4 models was achieved across three downstream tasks.The reported gain is attributed to accurate natural-language failure feedback supporting error correction.
2 Related Work
Related work establishes failure detection, scalable robotic data generation, and foundation models as connected directions in robotic manipulation. AHA combines these lines by using simulation-generated failure data to train a foundation model for failure reasoning.
- Overview: The paper positions AHA as language-based failure reasoning for downstream robotic applications.Its related-work framing covers failure detection, data generation, and foundation models for manipulation.
- Failure detection in robotic manipulation: Failure detection has been studied in HRI and task-and-motion planning, gaining renewed importance as LLMs and VLMs enter robot manipulation systems.These models increasingly generate rewards and synthesize robot trajectories.
- Data generation in robotics: Robotic data-generation methods automate task demonstrations for behavior cloning, VLM instruction-tuning, and simulated-policy benchmarks.Examples include trajectory adaptation and simulation-based representation or benchmark generation.
- Data generation in robotics: AHA combines its robotic failure dataset with VQA and object-detection data during instruction-tuning.The mixed dataset supports failure reasoning across domains, embodiments, and tasks.
- Foundation models for robotic manipulation: Foundation-model approaches for robotic manipulation leverage LLMs and VLMs to interpret open-world semantics and generalize across tasks.Related systems use visual prompts for low-level action generation or other instruction-based manipulation capabilities.
3 The AHA Dataset
The AHA dataset is built by classifying common robotic manipulation failures and procedurally perturbing successful simulated trajectories across tasks and keyframes. FailGen operationalizes these perturbations in simulation environments to generate instruction-tuning failure data.
- Dataset generation: FailGen generates the AHA dataset by perturbing successful RLBench trajectories across 79 tasks and seven categorized failure modes.The seven modes include incomplete grasp, grip slippage, keyframe translation and rotation errors, missing rotation, wrong action sequence, and wrong target object.
- Failure taxonomy: The taxonomy distinguishes object-centric failures from action-centric failures, including grasping, retention, pose, sequencing, and object-selection errors.Object-centric examples include No_Grasp, Slip, and Wrong_object; action-centric examples include Translation, Rotation, No_Rotation, and Wrong_action.
- Procedural synthesis: FailGen sweeps keyframes and failure-mode configurations, using simulation success checking to generate parameterized failure demonstrations.Configurations encode perturbations such as offsets, release timing, altered rotations, reordered activations, and reassigned objects.
- Instruction-tuning data: The pipeline generates corresponding query and answer prompts for each synthesized task and failure mode, supporting instruction tuning.The overview pairs procedural trajectory perturbation with prompt generation, while the dataset construction applies the process across simulator tasks.
- Cross-environment generation: FailGen also produces failure data from ManiSkill and adapts RoboFail, demonstrating use across different simulation environments.This extends the generation pipeline beyond RLBench-based dataset construction.
4 Method
AHA formulates failure reasoning as sub-task evaluation followed by free-form explanation, using temporally organized multi-view inputs and instruction tuning. Its training data comes from procedurally perturbed trajectories, while tuning updates the projector and language model but freezes the vision encoder and tokenizer.
- Failure reasoning formulation: AHA first predicts whether a sub-task succeeds, then generates a concise natural-language explanation when the answer is negative.The formulation uses a language specification and image prompt to reason about failures at the current sub-task.
- Failure reasoning formulation: The model represents a manipulation trajectory as consecutive-keyframe sub-tasks and prompts it to explain what led to failure.This shifts failure reasoning beyond binary success classification toward explanations tied to the current action segment.
- Input representation: Inputs concatenate keyframes up to the current sub-task in temporal order, add remaining viewpoints to mitigate occlusion, and include a detailed task description.The image representation combines temporal trajectory context with available camera views.
- Training data: FailGen perturbs successful trajectories across 79 RLBench tasks to create 49k failure image-text pairs for instruction tuning.The resulting training data alternates across tasks while synthesizing multiple failure modes.
- Model training: Instruction tuning updates the projection layers and transformer language model while keeping the image encoder and tokenizer frozen.The architecture combines image tokens, projected into the language-token space, with a transformer-based language model.
5 Experimental Results
AHA is evaluated across failure-reasoning benchmarks, general VQA, scaling experiments, and downstream robotic applications. It generalizes across tasks, environments, and embodiments while improving several LLM/VLM-assisted robotics systems.
- Evaluation setup: AHA was evaluated across three diverse datasets, four metrics, and multiple VLM baselines for failure detection and reasoning.The evaluation includes out-of-domain tasks, simulation environments, and cross-embodiment scenarios.
- Failure-reasoning results: AHA-13B outperformed GPT-4o-ICL by 12.6% on AHA Test, 13.4% on ManiSkill-Fail, and 4.9% on RoboFail across evaluation metrics.These datasets test task, simulation-domain, and real-world embodiment generalization, respectively.
- General knowledge retention: AHA-13B remained comparable to LLaVA-v1.5-13B on standard VQA benchmarks, with only a 1.5% margin difference.This evaluation examines whether domain-specific fine-tuning preserves general-purpose visual-language knowledge.
- Scaling: AHA performance increased with fine-tuning data size, with an average quadratic fit gradient of 0.0022 across four ManiSkill-Fail metrics.The tested instruction-tuning data sizes ranged from 3k to 60k examples.
- Downstream robotics tasks: AHA failure feedback supports reward synthesis, task and motion planning, and sub-task verification across LLM/VLM-assisted robotics applications.The reported downstream integrations include Eureka, PRoC3S, and Manipulate-Anything.
- Downstream robotics tasks: AHA-13B improved downstream task success over GPT-4o by 36.7% in task-plan generation within the PRoC3S system.PRoC3S used failure feedback during up to 100 plan samples and three feedback cycles.
6 Conclusion
AHA improves failure detection and reasoning for robotic manipulation, with natural-language explanations that adapt across robots, tasks, and environments. Its main scope boundary is that its reasoning remains closely tied to the failure scenarios represented in fine-tuning data.
- AHA’s language reasoning remains closely aligned with failure scenarios represented in its fine-tuning data.The authors identify more open-ended failures outside the included taxonomy as an opportunity for future improvement.
- AHA identifies manipulation failures and explains them in natural language across varied robots, tasks, and environments.
- AHA outperforms existing models across multiple metrics and datasets.
- AHA’s natural-language feedback improves error recovery and policy performance when integrated into manipulation frameworks.