MVI-Bench: A Comprehensive Benchmark for Evaluating Robustness to Misleading Visual Inputs in LVLMs

The gap

• Prior robustness benchmarks focus mainly on hallucination, adversarial inputs, or misleading text, while naturally arising misleading visual cues remain underexplored.The paper distinguishes visual cues arising in real-world scenes from inaccurate or irrelevant information injected into questions.Source: §1 · pp. 1–2, Figure 1 · p. 1

Benchmark design

• MVI-Bench organizes misleading visual inputs into three hierarchical levels: Visual Concept, Visual Attribute, and Visual Relationship.The taxonomy is grounded in fundamental visual primitives.Source: §1 · p. 2
• The benchmark contains 624 expertly annotated VQA pairs, or 1,248 instances, with paired images differing by subtle misleading visual cues.Each pair preserves nearly identical semantic content across the normal and misleading images.Source: §1 · p. 2

Evaluation

• MVI-Sensitivity quantifies relative performance degradation from normal to misleading visual inputs, providing a fine-grained measure of LVLM robustness.The metric is designed to characterize model sensitivity to misleading visual inputs.Source: §1 · p. 2

Findings

• Experiments across 18 open- and closed-source LVLMs reveal limited robustness, complementary roles for visual perception and reasoning, and spurious correlations.The authors present these as three key observations from their experiments and analyses.Source: §1 · p. 2

Robustness benchmarks

• LVLM robustness studies have primarily examined hallucination and adversarial vulnerabilities, which together provide only a partial view of robustness.Hallucination work measures discrepancies between responses and visual content; adversarial work studies carefully crafted perturbations.Source: §2 · p. 3

Misleading-input benchmarks

• Recent misleading-scenario studies most commonly introduce misleading textual inputs, while robustness to naturally arising misleading visual inputs remains underexplored.The paper contrasts injected textual misinformation with visual cues arising in real-world environments.Source: §2 · p. 3
• IllusionVQA and O-Bench investigate visual misleading phenomena, but each focuses on one type and relies on coarse metrics such as accuracy.The paper says these limits constrain fine-grained robustness analysis.Source: §2 · p. 3
• MVI-Bench is positioned as addressing these limits through broader category coverage and paired evaluation.The paper states that this design enables insights unavailable from prior benchmarks.Source: §2 · p. 3

3. MVI-Bench

• Each benchmark pair uses the same four-option MCQ for a normal image and a visually similar misleading counterpart, with a plausible distractor tied to the misleading cue.MVI-Bench contains 624 pairs, or 1,248 instances, across six categories.Source: §3 · pp. 3–4

3.1. Taxonomy of Visually Misleading Types

• The taxonomy groups misleading inputs into Visual Concept, Visual Attribute, and Visual Relationship levels, covering six categories from visual resemblance to visual illusion.It excludes visual combination to isolate the effect of each individual misleading level.Source: §3.1 · p. 4

3.2. Data Curation

• Three trained experts curated natural, synthetic, and edited image pairs; additional experts reviewed annotations and discarded controversial cases; retaining some low-difficulty cases reduced the dataset from 1,578 to 1,248 instances.Low-difficulty cases were those both InternVL3-2B and Qwen2.5-VL-3B answered correctly for both images. Paired images had high CLIP-Large cosine similarity across all six categories.Source: §3.2 · pp. 4–5

3.3. Evaluation Metrics

• MVI-Sensitivity measures the accuracy difference between normal and misleading samples; a lower value indicates less effect from misleading cues and stronger robustness.The benchmark also reports accuracy on each of the two input types.Source: §3.3 · p. 5

4.1. Models and Experimental Settings

• The evaluation covers 18 closed- and open-source LVLMs, uses deterministic decoding settings, and shuffles answer options to mitigate option-position bias.Closed-source models use temperature 0.0; open-source models use greedy decoding.Source: §4.1 · p. 6

4.2. Main Results (RQ1)

• LVLMs handle coarse visual concepts more reliably than attributes: visual resemblance and representation confusion are stronger categories, but material-confusion performance is lower.Gemini-2.5-Pro reaches 66.00% Accm in material confusion, while GPT-4o reaches 43.00%.Source: §4.2 · p. 6
• Mirror reflection and occlusion remain spatial weaknesses, with only three models exceeding 50% Accm on occlusion confusion.Even Qwen2-VL-72B reaches only 50.00% Accm on mirror reflection; small models named in the passage exceed 70% MVI-Sensitivity there.Source: §4.2 · p. 6
• All evaluated LVLMs degrade on misleading cues; closed-source models have MVI-Sensitivity above 20%, while open-source models are more susceptible, with Molmo-7B at 48.69%.GPT-5-Chat drops from 90.00% to 61.00% accuracy on visual illusion; Qwen2-VL-72B has the lowest open-source MVI-Sensitivity at 32.38%.Source: §4.2 · p. 6, Table 2 · p. 7

4.3. Assessing the Impact of Visual Perception and Reasoning (RQ2)

• 53.85% Accm with GPT-4.1 captions versus 45.99% without captions shows a perception-associated gain for Qwen2.5-VL-7B across all categories.The caption-assisted setup uses a strong auxiliary LVLM to describe the image, then combines that caption with the image and question.Source: Table 3 · p. 7, §4.3 · p. 7
• Scaling Qwen2.5-VL with a fixed vision encoder generally raises overall Accm and lowers MVI-Sensitivity, but mid-scale gains are non-monotonic across categories.From 7B to 32B, performance on misleading images declines in representation confusion and mirror reflection.Source: §4.3 · p. 8
• Long-thinking open-source models show lower Accm, especially in perception-heavy categories, while proprietary models can gain; overattention to details may hurt or help by category.The think mode can overemphasize fine details, and the thinking-enhanced Gemini-2.5-Flash still trails non-reasoning GPT-5-Chat by 0.64% in Accm.Source: §4.3 · pp. 7–8, Figure 5 · p. 8

4.4. Case Study: When LVLMs Outperform on Misleading Visual Inputs (RQ3)

• About 4% of cases reverse the usual pattern: masking a receipt that Qwen2.5-VL-7B counts as a book flips its accidentally correct answer.The authors speculate that answer-only VQA supervision encourages shortcut cues and argue that evaluation should consider reasoning processes as well as answer correctness.Source: §4.4 · pp. 8–9, Figure 6 · p. 9

• MVI-Bench is designed to evaluate how misleading visual inputs undermine LVLM robustness.Source: §5 · p. 9
• MVI-Sensitivity characterizes LVLM robustness at a fine-grained level.Source: §5 · p. 9
• Evaluation reveals pronounced LVLM vulnerabilities to misleading visual inputs, while analyses offer insights for developing more reliable and robust models.Source: §5 · p. 9

• Analysis of counterintuitive cases offers guidance for future training and evaluation methodologies.Source: Impact Statement · p. 9
• MVI-Bench examines LVLM behavior from visual perception and reasoning perspectives to provide insights for developing more reliable models.Source: Impact Statement · p. 9
• The authors report collecting data only from publicly accessible sources and manually reviewing generated images to mitigate potential biases.Source: Impact Statement · p. 9
• The authors foresee no direct negative societal consequences from the work.Source: Impact Statement · p. 9

A. Limitations and Future Work

• The paired-image design enables counterfactual comparisons, but the spurious-correlation methodology does not directly generalize to other datasets without adaptation.Source: A. Limitations and Future Work · p. 14
• Although spurious-correlation cases are a very small benchmark fraction, their implications may be more substantial in real-world settings.Source: A. Limitations and Future Work · p. 14
• Weak supervision with answer labels but no rationales makes shortcut learning difficult to mitigate, while VQA evaluations emphasize correctness over causal grounding.Source: A. Limitations and Future Work · p. 14
• Future work is urged to develop training objectives and evaluation protocols that discourage shortcut exploitation and promote faithful, causally aligned visual reasoning.Source: A. Limitations and Future Work · p. 14

B. Ethics Statement

• The authors state that Seedream-generated content reflects biases and limitations in its training data, including potential stereotypes or inappropriate content.Source: B. Ethics Statement · p. 14
• The authors report manual review and careful curation to mitigate risks, and encourage stronger methods for identifying and mitigating biases.Source: B. Ethics Statement · p. 14

C.1. Data Curation Pipeline

• The benchmark curation pipeline moves from image collection to VQA annotation, data filtering, and MVI-Bench, with human verification at key stages.Source: Figure 7 · p. 14

C.2. Annotation Quality Control

• Annotators were trained experts with at least a bachelor’s degree, and received category guidance, examples, and common pitfalls before annotation.Source: §C.2 · p. 14
• Three experts independently curated image pairs and multiple review stages checked category fit, question clarity, and answer correctness before a final ambiguity check.Source: §C.2 · pp. 14–15
• The think mode guides SAIL-VL with historical thoughts and tends to overemphasize fine details, unlike the non-think mode’s direct use of visual evidence.Source: Figure 8 · p. 15
• Inter-annotator agreement on a random sample of 200 instances was κ = 0.93, which the authors describe as near-perfect.Source: §C.2 · p. 15
• Reviewers discussed disagreements, which occurred in approximately 5% of cases, and discarded instances without consensus to prioritize reliability over dataset size.Source: §C.2 · p. 15

C.3. More Examples

• Figure 9 presents additional examples from each MVI-Bench category.Source: §C.3 · p. 15

D.1. Prompt Format for Caption-Assisted Inference

• Caption-assisted inference concatenates an auxiliary LVLM’s image caption with the image and question before inputting them to the inference model.The auxiliary model is prompted to describe the image; the caption is intended to compensate for limitations of the inference model’s vision encoder, and the answer prompt requests a choice letter.Source: §D.1 · pp. 15–16

D.2. Model Details

• Table 5 presents the release times and model sources of LVLMs evaluated in MVI-Bench.Source: §D.2 · p. 16

Human performance baseline

• The authors selected participants with different educational backgrounds to assess whether MVI-Bench requires specialized knowledge and remains solvable for the general population.Source: §D.3 · p. 16

Procedure

• Each participant independently completed all 1,248 VQA instances, comprising 624 pairs across six categories, in randomized order, with a five-second limit per question.Responses after five seconds were marked incorrect. Before formal evaluation, each participant completed a 10-instance practice session that was excluded from MVI-Bench.Source: §D.3 · p. 16

Results

• 98.24% Accm and 1.63% MVI-Sensitivity were the four participants’ average results despite the five-second limit per question.The authors characterize this as strong resilience to misleading visual cues and say the scenarios remain easily resolvable for humans through rapid perceptual processing and contextual reasoning.Source: §D.3 · p. 16

D.4. Implementation of Attention Visualization

• Relative attention normalizes question-conditioned answer-to-image-token attention against attention from a generic image-description prompt, highlighting regions whose attention increases for the question.The question-conditioned attention is extracted from the first generated answer token to image tokens and averaged across attention heads.Source: §D.4 · p. 16

E. Additional Case Studies on Long-form CoT

• Figure 8 presents additional cases comparing Non-think and Think modes.Source: E. Additional Case Studies on… · p. 16
• Figure 9 provides more examples from the six misleading categories defined in MVI-Bench.Source: Figure 9 · p. 17
• Table 5 lists the release times and model sources of the LVLMs evaluated in MVI-Bench.Source: Table 5 · p. 18