Source-linked AI summary
MMErroR: A Benchmark for Erroneous Reasoning in Vision-Language Models
Yang Shi, Yifeng Xie, Minzhe Guo, Liangsi Lu, Mingxuan Huang, Jingchao Wang, Zhihong Zhu, Boyan Xu, Zhiqi Huang
TL;DR
VLMs can perform strongly on multimodal tasks, but it remains unclear whether they understand cross-modal content well enough to recognize and diagnose erroneous reasoning. MMErroR addresses this gap with a controlled benchmark of single-error reasoning chains and two diagnosis tasks; across 12 VLMs, the strongest reported model reaches only 66.65% overall accuracy, while the benchmark exposes substantial limitations in error identification and classification.
Problem
Existing evaluations emphasize final-answer correctness, leaving limited evidence about whether VLMs can detect and explain errors in multimodal reasoning.
Method
MMErroR contains 1,997 samples across 24 subdomains and six top-level domains, each with one verified root-cause error, and supports Error Type Classification and Error Presence Detection.
Results
66.65% overall accuracy is achieved by the top-performing model in the evaluation, while current VLMs still show significant limitations in identifying and classifying reasoning errors.
Takeaways & Limitations
Error diagnosis provides a fine-grained way to assess multimodal reasoning weaknesses and offers insights into models’ ability to perform trustworthy self-oversight.
Takeaways & Limitations
The benchmark isolates one coherent error per sample and relies on model-assisted synthesis, so it does not currently model cascading or simultaneous real-world errors and may inherit generator-specific biases.
Abstract
from arXiv · showhide
Recent advances in Vision-Language Models (VLMs) have improved performance in multi-modal learning, raising the question of whether these models truly understand the content they process. Crucially, can VLMs detect when a reasoning process is wrong and identify its error type? To answer this, we present MMErroR, a multi-modal benchmark of 1997 samples, each embedding a single coherent reasoning error. These samples span 24 subdomains across six top-level domains, ensuring broad coverage and taxonomic richness. Unlike existing benchmarks that focus on answer correctness, MMErroR targets a process-level, error-centric evaluation that requires models to detect incorrect reasoning and classify the error type within both visual and linguistic contexts. We evaluate 12 representative VLMs, and even the best model, Gemini-3-Pro-Preview, classifies the error correctly in only 66.65\% of cases, underscoring the challenge of identifying erroneous reasoning. Furthermore, the ability to accurately identify errors offers valuable insights into the capabilities of multi-modal models. Project Page: https://mmerror-benchmark.github.io
1 Introduction
MMErroR addresses whether VLMs can detect and diagnose erroneous multimodal reasoning rather than merely produce correct final answers. It introduces a process-level benchmark and reports substantial difficulty among current models in fine-grained error diagnosis.
- Existing benchmarks mainly evaluate final-answer correctness or locate an incorrect reasoning step, providing limited insight into why multimodal reasoning fails.
- MMErroR evaluates whether VLMs can detect flawed reasoning and classify its root cause in visual and linguistic contexts.
- 1,997 samples span six reasoning domains, with each coherent reasoning chain containing exactly one injected error for unambiguous attribution.
- The benchmark supports Error Type Classification and Error Presence Detection, with the latter serving as a controlled stress-test because the current release contains only erroneous chains.
- The evaluation of 12 representative VLMs reveals significant difficulty with introspective error detection and classification, limiting trustworthy self-oversight in multimodal reasoning.
2 MMErroR
MMErroR constructs a broad, controlled benchmark by curating challenging multimodal problems, injecting one categorized error into each reasoning chain, and applying multi-stage human quality checks. Its final dataset contains 1,997 verified samples distributed across six domains and 24 subdomains.
- Task Classification: MMErroR evaluates error diagnosis through Error Type Classification and Error Presence Detection using reasoning chains guaranteed to contain exactly one error.
- Benchmark Construction: The benchmark curates image–question–answer triplets from established multimodal datasets, then balances domains and filters out overly simple or low-information problems.
- Benchmark Construction: Each plausible reasoning chain receives one contextually coherent error from four categories: Visual Perception, Knowledge Deployment, Question Comprehension, or Reasoning Error.
- Quality Assurance: 1,997 high-quality erroneous reasoning samples remain after expert verification and scoring for coherence, clarity, error localizability, and semantic consistency.
- Data Analysis: Physics & Engineering is the largest domain at 22.98% (459 samples), while Knowledge Deployment Error is the most prevalent error type at 44.07% (880 samples).
3 Experiment Settings
The experiments evaluate 12 VLMs under two complementary protocols, measuring both error-type classification and error presence detection across six domains and aggregate accuracy metrics. EPD remains a controlled stress-test because all evaluated chains are erroneous.
- Models: The study evaluates 12 VLMs divided into open-weights and proprietary groups.
- Evaluation Protocols: Both protocols use an image, question, and step-by-step reasoning chain as input.
- Evaluation Protocols: ETC requires selecting one of four predefined error types from a chain guaranteed to contain exactly one error.
- Evaluation Protocols: EPD first requires a No Error or Error Present decision and then error-type classification when an error is predicted.
- Evaluation Protocols: EPD is a controlled stress-test under uniformly erroneous conditions, and always predicting an error does not exceed ETC-level performance without correct type classification.
- Metrics: Performance is reported across six domains, their Macro Average Score, and Overall Weighted Accuracy using a multiple-choice output format.
4 Empirical Results and Analysis
MMErroR evaluation shows that current VLMs struggle to classify erroneous reasoning, with performance varying across domains, model families, and task settings. Error diagnosis is also linked to visual grounding and the ability to correct answers.
- ETC Evaluation Results: 66.65% overall accuracy is the best ETC result, achieved by Gemini-3-Pro-Preview, while a substantial gap remains to Human Expert performance.Doubao-Seed-2.0-pro reaches 64.80% and GPT-5.2 (xhigh) reaches 64.30%.
- ETC Evaluation Results: 69.81% is Gemini-3-Pro-Preview’s best domain result in CM, but domain leadership is distributed across multiple models.GPT-5.2 (xhigh) leads DA at 69.59%, Doubao-Seed-2.0-pro leads PE at 67.32%, Qwen-VL-Max leads BH at 66.39%, and Kimi-K2.5 leads EE at 66.67%.
- ETC Evaluation Results: 60.19% is the strongest overall ETC result among open-weights models, achieved by Kimi-K2.5, followed by Qwen3-VL-32B-Thinking at 59.29%.Other open-weights models remain clearly below the leading group, providing discrimination across model families and capability levels.
- EPD Evaluation Results: 61.39% overall accuracy is Gemini-3-Pro-Preview’s best EPD result, and every model performs worse under EPD than ETC.EPD requires models to first decide whether to invoke an error label before classifying its type; Gemini also achieves a 61.88% macro-average.
- EPD Evaluation Results: EPD domain strengths remain distributed, with Gemini-3-Pro-Preview leading ML, PE, CM, and BH, Claude-Opus-4.5 leading EE, and GPT-5.2 (xhigh) leading DA.The evaluation compares VLM performance under ETC and EPD settings.
- Visual Consistency Analysis: Visual error diagnosis succeeds when text tokens align with relevant image regions and fails when the model extracts irrelevant or ambiguous visual information.The logit-lens analysis illustrates this contrast using the token “darkest cone.”
- Reasoning Consistency Analysis: Accurate diagnosis provides directly actionable information for answer correction, whereas merely exposing an erroneous chain yields almost no improvement over baseline VQA.Annotating the erroneous step gives a modest but consistent gain, while the largest improvement occurs when the error type is known.
5 Related Work
MMErroR extends multimodal evaluation from final-answer correctness and error-step localization to process-level error diagnosis. It covers visual, factual, logical, and comprehension failures, while related analyses examine how error awareness affects correction.
- Evaluation Comparisons: Table 2 compares model accuracy under EPD, ranking VLMs within open-weights and proprietary groups by overall accuracy.Best, second-best, and third-best models in each column are marked with dark, medium, and light shades.
- Diagnosis and Answering: Table 3 contrasts original VQA accuracy on 200 examples where each model correctly versus incorrectly identified the error type in ETC.The two subsets are used to examine the relationship between error diagnosis and question-answering ability.
- Process-Level Evaluation: MMErroR shifts evaluation from final-answer correctness to verification of whether the reasoning path itself is valid.This process-level focus is intended to provide a more transparent assessment of multimodal understanding.
- Error Taxonomy: MMErroR covers failure modes beyond visual perception, including Knowledge Deployment Error, Reasoning Error, and Question Comprehension Error.The benchmark requires models to identify errors in logic and factual application as well as visual grounding.
- Error Awareness: Table 4 compares correction accuracy when models receive the erroneous chain, its erroneous step, or its exact error type, against original VQA.The controlled analysis uses a randomly sampled subset of 200 MMErroR examples.
- Error Diagnosis: Existing error-localization benchmarks identify which reasoning step is incorrect, whereas MMErroR enforces Error Type Classification to assess why the model failed.The distinction targets diagnostic information about the nature of a reasoning failure.
6 Conclusion
MMErroR introduces a broad, process-level benchmark for diagnosing multimodal reasoning errors and finds that even leading VLMs remain limited at this task. The benchmark supports further work on reliable and interpretable multimodal reasoning.
- Conclusion: MMErroR contains 1,997 samples spanning 24 subdomains across six top-level domains, with one verified root-cause error per reasoning chain.It supports Error-Type Classification and Error Presence Detection across 12 representative VLMs.
- Conclusion: 66.65% overall accuracy is achieved by the top-performing model, showing substantial limitations in identifying and classifying reasoning errors.Verified clean chains and multi-error cascades are identified as important future extensions.
Limitations
MMErroR isolates one coherent reasoning error per sample for precise diagnosis, but this design omits cascading or simultaneous real-world failures and may reflect generator-specific biases.
- Each sample contains a single coherent reasoning error, enabling precise diagnostic attribution but not modeling cascading or simultaneous failures.The authors identify multi-error scenarios as a direction for future work.
- Model-assisted synthesis of erroneous reasoning chains may introduce biases in error patterns or linguistic styles.The chains undergo multi-stage human verification, but their initial generation remains model-assisted.
A Complexity-Aware Filtering
The benchmark estimates question complexity from linguistic and domain-specific features, then oversamples harder instances while retaining some lower-complexity examples for robustness.
- Complexity combines equally weighted z-scored features for comparisons, negations, open-ended questions, and domain-specific formulas.Formula presence is detected with a regular-expression match.
- 10 000 samples are drawn from the Gaussian upper-half tail between µ + 0.5σ and µ + 2σ to oversample harder instances.The procedure preserves medium-easy diversity while emphasizing higher-complexity questions.
- The mean complexity rises from 0.00 to +0.82 while a few lower-complexity items remain for evaluation robustness.
B Prompt Template
The prompt template constructs plausible incorrect reasoning chains by assigning exactly one error type while requiring correct visual use, contextual relevance, and an incorrect final answer.
- Erroneous chains are designed to remain linguistically coherent, plausible, and contextually relevant.
- Visual perception errors include misreading text, confusing chart values, or misidentifying colors, shapes, positions, counts, and locations.For this category, the subsequent reasoning should remain mostly correct given the wrong visual input.
- Reasoning errors preserve correct visual perception but introduce mistakes such as arithmetic or calculation errors.
- Question-comprehension errors apply otherwise consistent reasoning to the wrong interpretation, such as a different question, constraint, entity, subset, or superset.
- Knowledge-deployment errors use incorrect external knowledge, formulas, or irrelevant facts despite correct visual perception and question understanding.
- Each generated chain must use visual information, contain exactly one taxonomy error, and end with a prediction different from the correct answer.The label is restricted to one of four named error categories.
C Few-shot Learning Exploration
The authors examine whether self-oversight can be elicited or improved through in-context learning and few-shot prompting using a controlled auxiliary experiment.
- The experiment targets self-oversight capabilities in the Error Type Classification task.
- The auxiliary experiment evaluates 0-shot, 1-shot, 2-shot, and 4-shot prompts across various models.It uses a randomly sampled subset of 200 MMErroR examples.