Seeing Is Believing? A Benchmark for Multimodal Large Language Models on Visual Illusions and Anomalies

Wenjin Hou; Wei Liu; Han Hu; Xiaoxiao Sun; Serena Yeung-Levy; Hehe Fan

Source-linked AI summary

Seeing Is Believing? A Benchmark for Multimodal Large Language Models on Visual Illusions and Anomalies

Wenjin Hou, Wei Liu, Han Hu, Xiaoxiao Sun, Serena Yeung-Levy, Hehe Fan

arXiv:2602.01816v1Feb 2, 2026cs.CV

TL;DR

Existing multimodal evaluations rarely test whether models follow visual evidence when it conflicts with learned priors. VIA-Bench addresses this gap with a human-reviewed benchmark of visual illusions and anomalies, then evaluates more than 20 MLLMs. The results show substantial weaknesses, including brittle CoT behavior, while the benchmark remains bounded by its current formats and evaluation scope.

Problem
Standard visual-language benchmarks rarely stress-test MLLMs under illusions, perceptual conflicts, and other atypical conditions where canonical priors can contradict visual evidence.
Method
VIA-Bench uses human-in-the-loop review to curate 1,004 multiple-choice questions across six categories of visual illusions and anomalies.
Results
Even SOTA MLLMs peak at 69.23% accuracy, leaving a 24.07% gap relative to human performance; CoT reasoning is brittle and often inconsistent.
Takeaways & Limitations
VIA-Bench exposes a persistent machine-perception bottleneck and provides a testbed for studying robustness when visual evidence conflicts with internal priors.
Takeaways & Limitations
The current evaluation covers 20+ MLLMs and multiple-choice QA, while future updates may add new models and open-ended or more compositional formats.

Abstract

from arXiv · show

Multimodal Large Language Models (MLLMs) have shown remarkable proficiency on general-purpose vision-language benchmarks, reaching or even exceeding human-level performance. However, these evaluations typically rely on standard in-distribution data, leaving the robustness of MLLMs largely unexamined when faced with scenarios that defy common-sense priors. To address this gap, we introduce VIA-Bench, a challenging benchmark designed to probe model performance on visual illusions and anomalies. It includes six core categories: color illusions, motion illusions, gestalt illusions, geometric and spatial illusions, general visual illusions, and visual anomalies. Through careful human-in-the-loop review, we construct over 1K high-quality question-answer pairs that require nuanced visual reasoning. Extensive evaluation of over 20 state-of-the-art MLLMs, including proprietary, open-source, and reasoning-enhanced models, uncovers significant vulnerabilities. Notably, we find that Chain-of-Thought (CoT) reasoning offers negligible robustness, often yielding ``brittle mirages'' where the model's logic collapses under illusory stimuli. Our findings reveal a fundamental divergence between machine and human perception, suggesting that resolving such perceptual bottlenecks is critical for the advancement of artificial general intelligence. The benchmark data and code will be released.

1. Introduction

VIA-Bench targets a gap in standard visual-language evaluations by testing MLLMs on illusions and anomalies that conflict with common-sense priors. Its human-reviewed benchmark and broad evaluation reveal persistent machine-perception weaknesses, including brittle CoT behavior.

Standard multimodal benchmarks emphasize ordinary visual contexts and often lack difficulty sufficient to distinguish genuine visual intelligence.
Atypical illusions and anomalies expose whether models prioritize visual evidence over canonical priors, such as assuming every hand has five fingers.
VIA-Bench covers six categories: color, motion, Gestalt, geometric and spatial, general visual illusions, and visual anomalies.
1,004 multiple-choice questions were produced through a multi-stage, human-in-the-loop quality process.
20+ MLLMs were evaluated, and advanced systems remained substantially behind humans on the benchmark.
CoT reasoning often provides little robustness against illusions and can amplify internalized priors.

2. Preliminaries and Problem Formulation

VIA-Bench formalizes visual-illusion evaluation as multiple-choice prediction under conflicts between image evidence and internal priors. The task specifies image, question, options, formatting, and optional CoT inputs while requiring answers grounded in intrinsic image properties.

2. Preliminaries and Problem Formulation: VIA-Bench evaluates whether MLLMs reconcile conflicting visual stimuli with internal common-sense priors.
2.1. Formal Task Definition: Each instance is an input tuple containing an image x, question q, candidate options O, and formatting instruction i.
2.1. Formal Task Definition: The question q is designed to probe the model’s visual perception.
2.1. Formal Task Definition: The option set O contains m candidates, with m ≥2, and exactly one ground-truth label y.
2.1. Formal Task Definition: The instruction i standardizes answer formatting so model outputs can be parsed deterministically.
2.1. Formal Task Definition: The parameterized MLLM fθ maps concatenated visual and textual inputs to a predicted answer a.
2.1. Formal Task Definition: System-level CoT prompting is used to examine reasoning stability by requesting intermediate steps before answer selection.
2.2. Evaluation Criteria: Ground truth is based on intrinsic image properties and is designed to be statistically independent of textual priors in q.

3. The VIA-Bench Dataset

VIA-Bench is a human-reviewed diagnostic suite designed to isolate failures arising when MLLM priors conflict with visual evidence. Its six-category, multiple-choice dataset combines controlled annotation, debiasing, broad capability coverage, and varied visual demands.

1,004 high-quality QA pairs form a diagnostic benchmark for conflicts between internal priors and raw visual evidence.
Human-in-the-loop review is integrated into a systematic multi-stage construction pipeline.
The dataset uses six domains: color, motion, Gestalt, geometric/spatial, general visual illusions, and visual anomalies.
Questions use targeted prompts, localized visual cues, plausible shortcut-based distractors, and a never-correct “Not Sure” option.
Secondary expert annotation verifies that each image’s intrinsic truth remains undisputed.
Option permutation and binary polarity flipping reduce positional and wording-based shortcut learning.
VA comprises 23.8% and GI 11.7% of the category distribution, the reported upper and lower bounds.
The benchmark covers seven capability dimensions and includes question-length variation and images up to 8334 × 2501.

4. Evaluation on VIA-Bench

VIA-Bench evaluates more than 20 MLLMs across six illusion and anomaly categories using deterministic accuracy protocols, revealing substantial, uneven gaps from human performance. Models often fail because visual misperception, prior-driven uncertainty, and overthinking persist despite multimodal and reasoning enhancements.

Benchmark setup: 20+ MLLMs spanning proprietary, open-source, and reasoning-enhanced systems were evaluated across VIA-Bench.The evaluation covered models ranging from 3B to 235B parameters.
Evaluation protocol: Accuracy was measured with standard multiple-choice scoring, using rule-based Match and LLM-as-a-Judge protocols.The judge protocol used GPT-4.1-mini and averaged five independent runs to reduce randomness.
Overall results: 93.30% human accuracy versus 69.23% for the best MLLM leaves a gap of at least 24.07 percentage points on VIA-Bench.The result indicates that current MLLMs remain substantially below human performance on the benchmark.
Model-group results: No single proprietary model performs uniformly well, although o4-mini reaches 97.16% on geometric and spatial illusions.Gemini-3-pro leads in three categories, but performance remains non-uniform across illusion types.
Model-group results: Reasoning-enhanced models reach 60.2% and 59.72% average accuracy, yet their highest accuracies are only 25.02% on visual anomalies and 51.84% on color illusions.These models also remain weak on general visual and gestalt illusions, while GLM-4.5V and Claude-3.5-sonnet-20241022 fail on motion illusions.
Reasoning analysis: CoT generally fails to improve VIA-Bench performance because extended reasoning often reinforces an initial visual error instead of correcting it.A Qwen2.5-VL-7B exception improved motion-illusion accuracy by +14.8%, but the gain relied on textual priors rather than genuine visual analysis.

5. Related Works

Prior multimodal benchmarks largely assess natural images, broad capabilities, or specialized tasks, while newer efforts probe spatial relations and visual puzzles. These evaluations leave visual illusions and anomalies comparatively underexamined.

MLLMs still struggle with cognitive illusions, perceptual conflicts, and intuition-breaking scenarios that restrict robust real-world generalization.
Most prior multimodal benchmarks focus on natural-image contexts despite their broad coverage and domain-specific specialization.
Recent benchmarks extend evaluation to perception-to-reasoning tasks, basic spatial relations, and visual puzzles.

6. Conclusions and Future Outlook

The paper introduces VIA-Bench to evaluate MLLMs on visual illusions and anomalies, then uses broad model evaluations to expose persistent perceptual weaknesses. It reports a substantial human–model accuracy gap and brittle CoT behavior, while outlining future benchmark expansion and robustness research.

69.23% accuracy was the highest result among evaluated MLLMs, leaving a 24.07% gap relative to human performance.
VIA-Bench evaluates frontier MLLMs on visual illusions and anomalies through 1,004 human-reviewed multiple-choice question–answer pairs.
CoT reasoning was brittle and often inconsistent on VIA-Bench, according to the paper’s analysis.
The benchmark is planned for public release, continued updates with new models, and possible expansion to open-ended and compositional illusion questions.
The paper positions VIA-Bench as a way to expose perceptual failure modes and inform future work toward more trustworthy multimodal AI.

Appendix Outline

The appendix documents VIA-Bench’s taxonomy, examples, supplementary experiments, evaluation implementation, model details, prompts, and qualitative case analyses.

Appendix A details the VIA-Bench taxonomy, and Appendix B provides typical question examples.
Appendix C reports additional experiments on OpenAI o3 and Qwen3-VL-235B-A22B-Thinking.
Appendix D describes regular match patterns, while Appendix E lists the judge-model input prompt.
Appendix F specifies detailed model versions for the 20+ evaluated MLLMs, and Appendix G shows system prompts.
Appendix H presents case studies and analysis based on models’ concrete reasoning procedures.

A. Taxonomy of VIA-Bench

VIA-Bench organizes visual illusions and anomalies into six categories and uses challenging multiple-choice questions to test perception and reasoning when visual evidence conflicts with commonsense priors. Its examples span color, motion, Gestalt, geometric and spatial, general visual, and anomaly phenomena.

Taxonomy: VIA-Bench covers Color Illusions, Motion Illusions, Gestalt Illusions, Geometric and Spatial Illusions, General Visual Illusions, and Visual Anomalies.
Color Illusions: Color illusions target misperception from color constancy, contrast, illumination, and shadow, including Checker Shadow and Ishihara examples.
Motion Illusions: Motion illusions use static textures and contrasts to elicit subjective perceptions of motion or pulsation, such as stripe-induced drift and spiral patterns.
Gestalt Illusions: Gestalt illusions test fine-grained object detection under compositional bias, including identification of an odd item in repetitive patterns.
Geometric and Spatial Illusions: Geometric and spatial illusions probe judgments of length, parallelism, alignment, and 3D existence using impossible figures and geometric traps.
General Visual Illusions: General visual illusions involve misperception or misdirection from perspective, composition, or occlusion, including leaves resembling birds and mirror illusions.
Visual Anomalies: Visual anomalies violate commonsense priors, with the study primarily focusing on biological anomalies such as extra or missing fingers or toes.
Question Design: The benchmark avoids trivial binary questions and uses category-specific multiple-choice distractors to assess low-level perception and higher-level reasoning.

C. More Experiment Results

The evaluation reports results across repeated runs and uses automated matching plus judge-based interpretation to map model responses to answer choices.

Mean accuracy is reported over five runs to reduce variation caused by model output randomness.
The match protocol uses regular expressions to extract a final option label and compare it with the gold answer.
A judge model is prompted to infer which multiple-choice option the response indicates and return only A, B, C, or D.

F. Model Versions

The paper documents the evaluated model versions and identifies the system prompt used by default in experiments.

Table 6 lists provider model identifiers for API systems and repository links for models run locally.
All evaluated models were available as of November 18, 2025.
The normal system prompt was used in all experiments unless otherwise specified.

H. Case Study

The case studies inspect concrete reasoning traces across selected models and illusion types, highlighting overthinking, visual-grounding failures, and both correct and incorrect conclusions.

Case study scope: The case study analyzes InternVL-3.5-8B, Qwen3-VL-30B-A3B-Thinking, and Gemini-2.5-pro using examples from Figures 7–12.
Reasoning-trace analysis: Correct reasoning is marked in green, while incorrect or visually ungrounded reasoning is marked in red.
Color illusions: In color illusions, InternVL-3.5-8B fails to identify a clear number, while Qwen3-VL-30B-A3B-Thinking recognizes reversed order but repeatedly confirms a low-confidence answer.
Motion illusions: In motion illusions, Qwen3-VL-30B-A3B-Thinking recognizes the illusion but overthinking produces an incorrect answer, whereas Gemini-2.5-pro answers directly.
Gestalt illusions: In gestalt illusions, recognizing a discrepancy does not prevent incorrect selection when models remain unconfident or exceed the maximum output length.
Geometric and spatial illusions: For geometric and spatial illusions, InternVL-3.5-8B repeats an incorrect parallel-lines view and fails to provide a final choice after exceeding the output limit.
Visual illusions: For visual illusions, Qwen3-VL-30B-A3B-Thinking correctly concludes that the car can pass despite excessive reasoning, while InternVL-3.5-8B follows an incorrect path.
Visual anomalies: For visual anomalies, InternVL-3.5-8B and Qwen3-VL-30B-A3B-Thinking miscount a six-fingered hand by relying on a five-finger prior, while Gemini-2.5-pro identifies and counts the fingers.