Source-linked AI summary
MIRAGE: The Illusion of Visual Understanding
Mohammad Asadi, Jack W. O'Sullivan, Fang Cao, Tahoura Nedaee, Kamyar Rajabalifardi, Fei-Fei Li, Ehsan Adeli, Euan Ashley
TL;DR
Multimodal visual reasoning is difficult to measure directly, and benchmark accuracy may not reliably reflect genuine visual access. The paper defines mirages and evaluates image-free benchmark performance, finding that models can behave as though absent images exist. These findings motivate modality-ablation testing and private or dynamically updated benchmarks, particularly because medical mirages can be pathology-biased and consequential.
Problem
Direct visual understanding is challenging to quantify, while benchmark accuracy is commonly assumed to reflect genuine visual understanding.
Method
The paper defines the mirage effect and measures image-free answers to visual questions, including a mirage-score comparing no-image accuracy with original image-access accuracy.
Results
Across tested model-benchmark pairs, image-free accuracy exceeded the additional accuracy gained from providing images, and a text-only 3-billion-parameter model outperformed frontier multimodal systems and human radiologists on a held-out chest radiology benchmark.
Takeaways & Limitations
Multimodal evaluation should include modality ablations and use private or dynamically updated benchmarks to test genuine dependence on visual inputs.
Takeaways & Limitations
The findings indicate a systematic artifact of joint image-text training and a broader challenge to current measures of multimodal reasoning.
Abstract
from arXiv · showhide
Multimodal AI systems have achieved remarkable performance across a broad range of real-world tasks, yet the mechanisms underlying visual-language reasoning remain surprisingly poorly understood. We report three findings that challenge prevailing assumptions about how these systems process and integrate visual information. First, Frontier models readily generate detailed image descriptions and elaborate reasoning traces, including pathology-biased clinical findings, for images never provided; we term this phenomenon mirage reasoning. Second, without any image input, models also attain strikingly high scores across general and medical multimodal benchmarks, bringing into question their utility and design. In the most extreme case, our model achieved the top rank on a standard chest X-ray question-answering benchmark without access to any images. Third, when models were explicitly instructed to guess answers without image access, rather than being implicitly prompted to assume images were present, performance declined markedly. Explicit guessing appears to engage a more conservative response regime, in contrast to the mirage regime in which models behave as though images have been provided. These findings expose fundamental vulnerabilities in how visual-language models reason and are evaluated, pointing to an urgent need for private benchmarks that eliminate textual cues enabling non-visual inference, particularly in medical contexts where miscalibrated AI carries the greatest consequence. We introduce B-Clean as a principled solution for fair, vision-grounded evaluation of multimodal AI systems.
1 Main
Multimodal AI models are widely evaluated through image-based benchmarks, but the paper shows that benchmark success can reflect mirage behavior rather than genuine visual understanding. This effect involves confidently describing and reasoning about images that were never provided.
- Benchmark performance is commonly used to assess visual understanding because direct quantification in real-world applications is challenging.
- The paper identifies the mirage effect as a side-effect of joint image-text training that challenges assumptions about multimodal reasoning and deployment.
- Frontier models frequently describe and reason from multimodal inputs, including images, that were never provided.
- Medical mirages are heavily biased toward pathology, while generated descriptions can appear indistinguishable from normal visual reasoning and lack uncertainty.
2 AI sees mirages
The paper defines mirages as confident descriptions of nonexistent visual inputs and measures them using image-dependent questions presented without images. Across tested models and domains, mirages are common and medical outputs are disproportionately pathology-oriented.
- A mirage is a confident answer describing nonexistent visual inputs without acknowledging missing images, uncertainty, or hypothetical assumptions.
- Phantom-0 removes images from visual questions spanning 20 medical, science, technical, and general categories to measure mirage rates.
- Over 60% of the time on average, every tested modern frontier model produced confident visual details across all categories.
- Across five medical categories, mirage-based diagnoses were heavily pathology-biased, including time-sensitive conditions such as STEMI, melanoma, and carcinoma.
3 Mirages give the illusion of visual understanding
Multimodal benchmarks can show high apparent visual performance even when models receive no images. This mirage behavior spans models, benchmarks, question types, and domains, while reasoning traces can remain visually plausible despite relying on nonexistent features.
- 70–80% of fully image-enabled accuracy is retained by frontier models in mirage-mode across six visual benchmarks.Individual benchmarks show 60–99% susceptibility to non-visual inference, with medical benchmarks at the upper end.
- In every tested model–benchmark pair, image-free accuracy exceeded the additional accuracy gained when images were provided.
- High mirage scores occur across combined textual-visual and completely visual questions, domains, question types, and model families.
- Models’ reasoning traces describe nonexistent visual features and resemble outputs produced with actual visual input, even when answers are correct.Dataset-name mentions can also significantly increase accuracy, suggesting models learn benchmark-level structural patterns.
- A 3-billion-parameter text-only Qwen-2.5 super-guesser trained without images outperformed frontier multimodal models and radiologists on ReXVQA’s held-out test benchmark.It surpassed human radiologists by more than 10% on average while relying on textual cues and benchmark structure.
- Mirage-based accuracy and realistic visual descriptions can coexist with ignored visual information, and accuracy or reasoning-trace inspection alone cannot guarantee vision-grounded behavior across fields.
4 Mirage-enablers cannot be manually detected
Image-free benchmark performance can arise from both known textual shortcuts and hidden benchmark structures that manual curation may not detect. Comparing mirage-mode with explicit guessing further reveals that prompting models to acknowledge missing images changes performance.
- Known image-less categories include text-answerable questions, redundant images, familiar images, language shortcuts, and weak distractors.
- Hidden benchmark patterns, synthetic-data biases, and latent cues extend beyond manually identified artifacts and enable image-free correct answers.
- GPT-5.1 accuracy declines across most general and medical categories when the prompt acknowledges the absent image and asks for a best guess.
- In Figure 4, mirage-mode outperforms or matches guess-mode across medical categories and wins 18 of 23 MMMU-Pro categories.Guess-mode outperforms mirage-mode in five MMMU-Pro categories and ties in two.
5 Mirage-proofing the benchmarks can change the visual AI landscape
Public benchmark leakage and hidden answer patterns can undermine visual evaluation, motivating B-Clean’s post-hoc removal of questions answerable without images. Applied across three benchmarks, cleaning retained roughly one quarter of questions, substantially altered accuracies, and changed model rankings.
- Motivation: Public benchmarks become temporary solutions because web-crawled pretraining absorbs them, while hidden benchmark structures enable non-visual inference.These vulnerabilities make repeated benchmark curation vulnerable to evolving data leakage.
- Motivation: Medical multimodal benchmarks remain largely unexamined despite being especially susceptible to the biases identified in prior benchmark research.
- Method: B-Clean separately identifies each candidate model’s compromised questions, then removes their union so remaining questions require visual input from every candidate.Compromised questions include vision-independent, prior-knowledge-answerable, and data-contaminated items.
- Results: After B-Clean filtering, 240/1,042 MicroVQA, 514/2,000 MedXpertQA-MM, and 428/1,730 MMMU-Pro questions remained, corresponding to 77.0%, 74.3%, and 75.3% removed.
- Results: On cleaned benchmarks, accuracies declined substantially and model rankings changed on two of three benchmarks, consistent with partial inflation from non-visual inference.For example, MedXpertQA-MM accuracies fell from 65.5% to 41.1% for GPT-5.1 and from 77.8% to 52.3% for Gemini-3-Pro.
- Limitation: B-Clean does not guarantee removal of every compromisable question, so its accuracies are not absolute metrics or directly comparable with models outside the cleaning procedure.Its supported use is relative, vision-grounded comparison among the evaluated candidate models.
6 Discussion
The discussion shows that multimodal models can achieve apparently strong visual reasoning without images, while fluent reasoning traces and benchmark accuracy may fail to establish genuine visual grounding. It argues for modality-sensitive evaluation and architectural safeguards, especially in high-stakes settings.
- Without images, frontier models exceeded the additional accuracy gained from providing images across every tested model-benchmark pair.
- Mirage reasoning can produce coherent, correct answers and meticulous traces while constructing an image-based epistemic frame unsupported by any provided image.
- Image-free success may reflect language priors and context reconstruction learned during multimodal training, rather than consistent use of visual information.
- Explicit guess-mode reduced performance across most benchmark categories, suggesting a conservative text-only regime distinct from mirage-mode.
- Medical mirages were richly detailed and pathology-biased, potentially propagating fabricated interpretations through downstream agents, reports, or clinical decisions.
- Counterfactual verification and cross-modality validation reduced a cardiac system’s composite mirage rate to zero while preserving diagnostic accuracy in a companion study.
- The authors limit their claims to multimodal evaluation, do not claim models never use images, and describe their mechanistic interpretation as inferential.
- The paper recommends modality-ablation tests, private or dynamically updated benchmarks, and image-present versus image-absent deltas as evaluation priorities.
7 Methods
The methods evaluate frontier models across diverse multimodal benchmarks and use image-absent prompts to measure mirage behavior. Model access, reasoning settings, and benchmark composition are specified across the evaluation pipeline.
- OpenAI, Google, and Claude models were accessed through Azure OpenAI API or Google Vertex AI platforms, with most non-OpenAI models evaluated in batch mode.
- Mirage scores were calculated in each model’s thinking or reasoning mode, with model-specific reasoning parameters and temperature settings.
- Non-thinking evaluations used model-specific low or absent reasoning configurations, including fixed thinking budgets or reasoning effort settings.
- Performance was evaluated across eight multimodal benchmarks spanning medical, academic, scientific, and video-understanding domains.
- Phantom-0 contains 200 open-ended image questions presented without images across 20 domain-category pairs to assess descriptions of non-existent visual content.
7.3 Mirage rate quantification
The study measures mirage behavior by presenting image-based questions without images and identifying whether responses acknowledge the missing visual input. It also probes medical mirages by requesting descriptions and diagnoses for non-existent images.
- Mirage-rate procedure: Phantom-0 questions were submitted without images or prompts acknowledging their absence, so visual descriptions could reflect mirage behavior rather than prompt-induced behavior.Ambiguous questions sometimes received the added instruction “Answer based on the image.”
- Mirage-rate procedure: GPT-5 judged whether each response mentioned an unavailable image, uncertainty, or insufficient confidence.
- Mirage-rate computation: Mirage rate equals 100% minus the fraction of responses acknowledging missing images or expressing uncertainty.
- Medical characterization: Gemini-3-Pro was prompted to describe a non-existent image and provide a final diagnosis across skin mole, ECG, brain MRI, chest X-ray, and pathology-image categories.
7.5 Benchmark evaluation prompts
Benchmark evaluations compared image-present and image-omitted conditions while preserving the dataset-specific prompts and question content. The prompts standardized answer formats across radiology, microscopy, medical, academic, and video benchmarks.
- Evaluation conditions: In mirage mode, images were omitted while the original dataset-specific system prompts were used without modification.
- Dataset prompts: The VQA-RAD prompt required answers based on radiology images, with one-word or very short responses formatted as [[your answer]].
- Dataset prompts: MicroVQA required exactly one multiple-choice answer based on microscopy images, formatted as [[X]].
- Dataset prompts: MedXpertQA-MM required one multiple-choice answer based on a provided medical image, formatted as [[X]].
- Dataset prompts: MMMU-Pro, Video-MMMU, and Video-MME required exactly one multiple-choice answer based on provided images or video content, formatted as [[X]].
- Evaluation conditions: Questions and answer options were identical across conditions, while only images or sampled video frames were removed in mirage mode.Models were evaluated in extended thinking or reasoning modes.
7.6 Mirage score computation
The paper defines mirage scores from image-absent benchmark accuracy and aggregates them by model and benchmark. These averages quantify overall dependence on mirage behavior and benchmark susceptibility to non-visual inference.
- Definition: A model–benchmark mirage score is defined from the model’s accuracy in mirage mode.
- Aggregation: The per-model mirage score averages mirage scores across six benchmarks to quantify the model’s overall dependence on the mirage effect.
- Aggregation: The per-benchmark mirage score averages results across four models to quantify susceptibility to non-visual inference.
7.7 Mirage-mode versus guess-mode comparison
The study compares image-absent mirage mode with an explicitly acknowledged image-absent guess mode, and separately tests whether text-only fine-tuning can exploit benchmark cues. The resulting model is evaluated on public and private chest-radiology test sets against multimodal models and radiologists.
- Mode comparison: Mirage mode omitted images without acknowledging their absence, whereas guess mode explicitly stated that the image was missing and asked the model to choose the best possible answer.
- Text-only fine-tuning: Qwen2.5-3B-Instruct was fine-tuned on 100,000 ReXVQA training samples with all images removed, using only question–answer pairs in mirage mode.
- Text-only fine-tuning: The text-only model used parameter-efficient LoRA fine-tuning across all linear layers.
- Training setup: Training used supervised fine-tuning for three epochs with AdamW, a cosine learning-rate schedule, and an effective batch size of 256.
- Evaluation: The resulting model was evaluated on public and held-out private ReXVQA test sets and compared with frontier multimodal models and radiologist performance.
7.9 B-Clean
B-Clean removes benchmark questions answerable without visual input, then evaluates candidate models on the remaining questions with images provided.
- B-Clean begins by evaluating each candidate model in mirage-mode on the full benchmark without image access.
- The framework removes the union of questions answered correctly by any candidate without visual input.
- An optional text-only model can remove additional questions answerable through textual patterns, further restricting evaluation to pattern-free questions.
- All candidate models are then evaluated with images on the B-Clean benchmark.
Supplementary Material
Supplementary experiments show that prompting and interface cues increase mirage behavior, while text-only responses can fabricate highly specific medical, scientific, technical, and everyday visual details.
- Commonly used evaluation instructions raise mirage rates: A prompt instructing models to answer based on nonexistent data consistently increases mirage rates in all but one extended-thinking model.
- The implemented guardrails could have an adverse effect: Explicitly mentioning multimodal attachments can disable interface guardrails and produce mirage rates similar to API behavior.
- Qualitative examples of mirage behavior: Text-only Phantom-0 responses fabricate detailed clinical findings, including radiology impressions, retinal abnormalities, stroke, and dermatological descriptions.
- Qualitative examples of mirage behavior: Different models give contradictory but equally specific tissue identifications for the same image-free histology question.
- Qualitative examples of mirage behavior: OCR-oriented questions elicit invented license plates and handwritten messages, while domain-expert prompts produce elaborate art-historical and chemical analyses.
- Qualitative examples of mirage behavior: Exact numerical answers, such as ninety-eight people, demonstrate how mirage outputs can project unwarranted certainty.
- Qualitative examples of mirage behavior: Image-free responses also fabricate languages, celestial objects, technical diagrams, floor plans, plant species, and crowd counts.