Source-linked AI summary
FailBench: How Reliable are VLMs at Judging Robot Task Success?
Zaruhi Navasardyan, Tatul Danielyan, Hrant Davtyan
TL;DR
Existing robot-failure benchmarks provide limited evidence that VLM judges generalize across independently collected sources. FailBench evaluates thirteen detectors on 2,197 attempts from fourteen public sources and finds that cross-source reliability remains limited, while outcome-focused cropping improves the strongest detector without retraining.
Problem
Existing evaluations provide limited evidence of whether robot-failure judgments remain reliable beyond the data on which detectors were developed.
Method
FailBench evaluates thirteen detectors on a common benchmark of 2,197 labeled manipulation attempts from fourteen public sources, using balanced-accuracy comparisons across subsets.
Results
The strongest detector reaches only 0.77 mean balanced accuracy; performance approaches chance on contact-dependent outcomes, while cropping outcome-relevant regions improves it by 2.3 points without retraining.
Takeaways & Limitations
General-purpose VLMs remain unreliable for cross-source robot failure detection, and the visual evidence presented to them is an actionable part of performance.
Takeaways & Limitations
FailBench measures only execution-time failures as one binary outcome per attempt, not bad plans, anticipated failures, failure types, or progress.
Abstract
from arXiv · showhide
Vision-Language Models (VLMs) are increasingly used to evaluate robot manipulation outcomes, but existing benchmarks offer limited evidence of cross-domain generalization. We introduce FailBench, a benchmark for robot failure detection comprising 2,197 manipulation attempts across 14 public sources (12 real-world, 2 simulated). In FailBench, 75% of failures occur naturally, and six real-world sources come from non-failure-detection datasets. Evaluating 13 VLM-based detectors, we find the best model achieves only 0.77 mean balanced accuracy. Notably, models fine-tuned for failure detection consistently underperform general-purpose VLMs and their own pretrained baselines. Performance depends heavily on required visual evidence: models approach saturation when outcomes depend on observable object motion, but degrade to near-chance (<0.60 balanced accuracy) on contact-intensive assembly tasks. Error analysis reveals a systematic bias toward predicting success under ambiguous evidence, which persists even with increased reasoning effort. Finally, we show that input-level intervention--spatially localizing and cropping outcome-relevant regions--improves the top detector by 2.4 percentage points without extra training.
1 INTRODUCTION
FailBench addresses limited evidence about whether VLM-based robot failure judgments transfer beyond the data used to develop detectors. It evaluates cross-source detection and analyzes how visual evidence affects reliability.
- VLM judgments of robot success can become evaluation labels, data filters, rewards, or recovery signals, making reliability beyond development data important.
- Existing benchmarks often use single collection efforts and deliberately constructed failures, which can make detection depend on recognizing the construction procedure.
- FailBench contains 2,197 manipulation attempts from fourteen public sources, including twelve real-world and two simulated sources.Six real-world sources were originally collected for purposes other than failure detection.
- The benchmark asks whether a completed attempt succeeded from its instruction and visual recording, excluding planning failures and intermediate progress.
- The study evaluates thirteen detectors across sources and analyzes failures by the visual evidence needed to distinguish object motion from physical contact.
- An evidence-localization pipeline improves failure detection by directing models toward the recording region that determines the outcome.
2 RELATED WORK
Prior work includes policy-dependent and policy-agnostic detectors, richer outcome predictions, and human evaluation, but published evaluations provide limited cross-source evidence. Existing benchmarks often confound failure detection with source-specific collection or construction procedures.
- Policy-agnostic detectors judge completed recordings and instructions across policies, whereas policy-dependent monitors rely on policy representations, actions, or trajectories.
- Existing failure benchmarks commonly hold the robot, camera, scene, task family, and labeling procedure constant within one collection campaign.
- Constructed failures can leave traces of their generation procedure, allowing detectors to perform well by recognizing those traces.
- Unstaged failures from RH20T and contact-rich REASSEMBLE are available but were not built for failure detection and are among the hardest measured slices.
- Recent systems predict explanations, categories, corrections, progress, or graded scores, but these outputs still contain a binary success label and lack validation across sources.
- Policy-dependent detectors trade accuracy against detection time and face calibration challenges under clutter, lighting, novel objects, and reworded instructions.
- Human evaluations require substantial coordination and are often based on only 20 to 30 real-world trials, limiting statistical separation between policies.
- No published evaluation scores detectors across independently collected sources under one protocol, so source effects remain difficult to separate from detector accuracy.
3 FAILBENCH
FailBench curates labeled robot attempts from diverse public sources into a common benchmark while preserving variation in failures, tasks, robots, scenes, and media. It samples sources to support balanced and failure-type-aware evaluation.
- FailBench represents each curated sample with the robot instruction, visual input, and one binary outcome label from its original source.
- The benchmark retained fourteen sources, twelve real and two simulated, after screening candidate corpora for completed manipulation, outcome labels, and nonredundant coverage.
- Sources include organic failures, simulated rollouts, human teleoperation, and constructed failures, reflecting different routes by which failures arise.
- The sources span tabletop manipulation, household chores, contact-rich assembly, insertions, cable clipping, tool handling, and bin-to-bin picking across multiple robots and laboratories.
- The benchmark subsamples each source to preserve diversity and samples failures and successes roughly 50/50, while retaining native imbalance when necessary.
- About 25% of samples undergo manual inspection, with ambiguous or incorrect labels replaced by samples from the same source and distribution.
4 EXPERIMENTS
The experiments evaluate thirteen detectors across benchmark subsets using balanced accuracy, then analyze model agreement and test localized visual inputs. General-purpose models lead the panel, performance varies by task distribution, and localization yields a modest improvement.
- Evaluation setup: Thirteen detectors are evaluated with their intended prompts and settings, using balanced accuracy to account for unequal class distributions.Macro averages weight each two-class subset equally, while micro averages pool answered samples; the failure-only reflect subset is excluded from macro averages.
- Benchmark results: Gemini 3 Flash reaches 0.77 balanced accuracy, while purpose-built failure detectors trail general-purpose models and their own pretrained baselines.The two embodied-robot models occupy the same performance band as the broader panel, whereas two specialists nearly always answer failure.
- Benchmark results: 0.52 is the worst average performance on reassemble, where even the best detector reaches only 0.60 on contact-rich assembly tasks.The subset comprises teleoperated NIST Task Board episodes involving precise manipulation of small objects.
- Benchmark results: 0.95 is the Spearman correlation between real and simulated detector rankings, while real subsets show greater score dispersion than simulated subsets.Simulation is about 1.3 points easier on average, with variance approximately 0.0089 versus 0.0135 in real environments.
- Model agreement: 0.28 is the mean error-set IoU across detector pairs, and the strongest 0.82 overlap reflects shared failure-prediction bias rather than shared perception.A three-detector majority vote reaches 0.78 versus 0.76 for the best model alone, but only 0.55 on contact-rich assembly while roughly tripling evaluation cost.
- Input intervention: +2.3 points raises Gemini 3 Flash from 0.773 to 0.797 when localization crops outcome-relevant regions before detection.The pipeline fixes 223 samples and breaks 160 without retraining; gains are largest when the workspace is wide and the decisive object is small.
5 LIMITATIONS
FailBench’s conclusions are bounded by its narrow outcome definition, vision-only inputs, limited embodiment coverage, and restricted diagnostic evidence. These constraints define what the benchmark can and cannot establish about robot failure detection.
- Diagnostics: The diagnostic mechanism is established on Gemma-4-31B-it and assumed rather than demonstrated for the rest of the detector panel.Failure-type counts and inspected reasoning traces come from one accessible model.
- Scope: FailBench measures execution-time failures as one binary outcome per attempt, not bad plans, anticipated failures, failure types, or progress.The thin label improves comparability across fourteen sources but limits what a score can reveal.
- Modalities: The benchmark uses only visual input and task instructions, despite four sources recording force-torque or audio signals.It therefore does not measure whether contact signals could improve performance on the hardest slices.
- Embodiment: Every source uses parallel-jaw tabletop arms, with no humanoid, mobile-manipulation, or dexterous-hand coverage.Whether the measured contact-level gap transfers to those platforms remains untested.
6 FUTURE WORK
Future work extends FailBench beyond its current tabletop and binary visual setting toward broader embodiments, richer labels, additional modalities, progress estimation, and contact-aware reasoning.
- Benchmark expansion: Extending the benchmark to humanoid and dexterous-hand corpora is identified as the next step beyond tabletop arms.This would test whether the observed detector behavior transfers across broader embodiments.
- Richer evaluation: Failure-type classification and force-torque or audio inputs would test why attempts fail and how much difficulty comes from visual observation alone.Some sources already contain failure annotations or non-visual signals, but these are not used for the current binary evaluation.
- Richer evaluation: Progress estimation requires dense annotations throughout an attempt rather than a single end-of-attempt label.Planning-time failures would require a separate benchmark evaluated against action plans and executed trajectories.
- Model development: Models and objectives should explicitly reason about fine-grained spatial and contact relationships to address contact-rich assembly failures.Localization removes irrelevant context but does not teach detectors to infer physical state.
7 CONCLUSION
FailBench shows that current VLM detectors remain unreliable across sources: even the strongest model reaches only 0.77 mean balanced accuracy, while specialists underperform general-purpose models. Performance improves with outcome-relevant cropping, but contact-level failures remain difficult.
- 0.77 mean balanced accuracy is the strongest result among thirteen tested detectors.
- Purpose-built failure detectors consistently underperform general-purpose VLMs, and fine-tuning can worsen performance outside the training data.
- Detectors perform better when success is established from coarse object motion but approach chance when it depends on fine-grained physical contact.
- Cropping outcome-relevant regions improves the strongest detector by 2.3 points without retraining, although gains vary across sources and do not solve contact-level failures.
- FailBench provides a cross-source evaluation for measuring progress toward detectors that can be trusted for labels, rewards, and robot decisions.
A DESCRIPTION OF BENCHMARK SUBSET
FailBench constructs source-specific benchmark subsets from public robot-manipulation datasets, preserving varied robots, tasks, media, labels, and failure origins. The benchmark documents both the available pools and the frozen samples used for evaluation.
- RH20T maps ratings 0 and 1 to failure and 2–9 to success, grouping curated tasks into 15 manipulation-skill categories.
- ArmNetBench retains only π0.5 policy rollouts, excludes demonstrations and the ambiguous suboptimal tier, and covers tabletop tasks including insertion and folding.
- SimplerEnv samples 100 rollouts per policy and balances success and failure within policy and task, using simulated goal predicates.
- UR5FAIL contributes 70 failures and 69 successes represented by start and end frames from three external views.
- Table 5 organizes sources by failure origin, rollout generation, and visual evidence shown to detectors.View counts refer to evaluation inputs rather than every camera released by a source.
- Table 6 distinguishes eligible source pools from frozen FailBench subsets, while documenting unlabeled records, source-specific filters, and ordering by contributed slice size.
- REASSEMBLE contributes 64 failures and 60 successes sampled across eight annotated assembly failure types.
B.1 SCORE VALIDITY
The score-validity analysis checks whether FailBench results reflect the evaluation setup rather than model or harness errors. It finds that published specialist performance can conceal class-imbalance effects, while specialists consistently trail their base models under a shared protocol.
- Guardian reproduces published performance closely: 0.741 on UR5FAIL versus 0.77 published, and 0.850 on BDV2FAIL versus 0.85 published.
- RoboFAC-7B reaches 0.53 balanced accuracy on the balanced FailBench subset despite a published 80.6% plain-accuracy score on an imbalanced split.It labels all 30 failures and 28 of 30 successes as failures, yielding recalls of 1.000 and 0.067.
- All fine-tuned specialists score below their corresponding base models under the same prompt, decoding, and input recipe.
- Guardian is nearly tied with its base model at 0.627 versus 0.629, but on the remaining eleven subsets it reaches 0.596 versus InternVL3-8B at 0.639.
- Because base and specialist models share the harness, prompt, and model family, the analysis attributes the difference to fine-tuning outside the tuned data.
B.2 MANUAL ERROR ANALYSIS
Manual inspection suggests that detector errors arise from ambiguous or physically subtle evidence, alongside a bias toward accepting apparent progress as success. These observations are explicitly not statistically supported and do not associate accuracy with a particular robot or task.
- The manual inspection covered a small share of errors without a fixed protocol, so its claims are not statistically supported.
- The inspected errors provide no evidence that accuracy is associated with robot embodiment, since the same detector succeeds and fails on the same arm class.
- Semantic failures are mostly caught, whereas physical failures such as wrong-place releases, empty grasps, and slipping grasps are mostly missed.
- A viewpoint example shows grapes appearing held from one camera while remaining on the table from two others, yet the model answers success.
- The traces often perform perceptual analysis but treat failure as something requiring proof, then predict success under unresolved ambiguity.
- Wrong answers from Gemma-4-31B-it average 1,868 reasoning characters versus 1,464 for right answers, with matching median differences.
C BOOSTING ABLATIONS
The ablations show that localization gains depend on both the quality of the predicted region and the detector’s ability to use it. Rendering changes clarify that narrowing the view removes irrelevant context but does not teach contact-state recognition.
- Component attribution: +0.01 point is lost when Gemini localization is paired with Gemma detection, while simplerenv gains +10.7 with Gemini detection but nothing with Gemma.The component swap results show that detector choice changes how much value the same localized boxes provide.
- Component attribution: Both localization and detection contribute to the improvement; neither component carries the result alone.A well-drawn box helps only when the detector can read it, while a strong detector cannot compensate for poor localization.
- Crop rendering: 0.787 is reached by the original localization pipeline using a crop cut to the predicted box at its own size.The ablation compares full-frame, cropped, resized-crop, and blackout-masked inputs.
- Crop rendering: Cropping or masking removes irrelevant context, but neither teaches the model to read contact state.Enlarging the crop adds nothing, and gains concentrate where the decisive region is smallest against its surroundings.
D REPRODUCING THE PUBLISHED-SCORE CHECKS
The published-score checks show that class imbalance can make failure-detection scores look strong even for trivial predictors. These checks therefore distinguish reported benchmark performance from balanced, in-domain evaluation.
- Published-score checks: RoboFAC-7B catches all 30 failures but only 2 of 30 successes on the balanced FailBench subset.The example illustrates why balanced accuracy is needed when evaluating binary failure detection.
- Published-score checks: 960 failures and 244 successes in the published real-world split allow an all-failure answer to score 960/1204 = 0.797.The reported comparison is against 0.806, showing how class prior affects raw accuracy-like scores.
- Published-score checks: 0.900 for Qwen3-VL-8B base is only one point above the constant all-failure score of 0.890 on ViFailback’s imbalanced split.FailBench does not project its recalls onto this split or claim an in-domain reproduction.