Source-linked AI summary
Seeing is Believing: Benchmarking and Mitigating the Interference of Language Priors on Visual Perception in MLLMs
Li Ma, Yan Zhuang, Jiawen Deng, Fuji Ren
TL;DR
Existing benchmarks assume shared commonsense priors across models, despite cross-model prior agreement below 80%, and over 15% of items can be answered correctly without images. ViSiP-Bench elicits model-specific priors and measures adherence and visual-confidence shifts, while MAPS softens each model’s strongest priors with targeted, factually grounded knowledge. Across 24 models, prior adherence is pervasive and does not diminish with scale; MAPS improves PAR and VIG while preserving reasoning and general-vision performance.
Problem
Existing benchmarks assume shared commonsense priors across models, despite cross-model prior agreement below 80%, and over 15% of items can be answered correctly without images.
Source: §3 · p. 3Method
ViSiP-Bench elicits model-specific priors and measures adherence and visual-confidence shifts, while MAPS softens each model’s strongest priors with targeted, factually grounded knowledge.
Source: §1 · p. 2, §3.2 · p. 4, §6 · p. 9Results
Across 24 models, prior adherence is pervasive and does not diminish with scale; MAPS improves PAR and VIG while preserving reasoning and general-vision performance.
Source: §6 · p. 9Takeaways & Limitations
MAPS shows that targeted prior softening can enhance visual reliance while preserving broader model capabilities.
Source: §6 · p. 9ViSiP-Bench’s fixed six-category taxonomy excludes specialized and rapidly evolving domains, so whether the observed adherence patterns extend to them remains open.
Source: Limitations · p. 9
Abstract
from arXiv · showhide
Language priors in MLLMs can override visual evidence, impairing visual perception. Existing benchmarks assume shared priors across models and rely solely on accuracy. However, cross-model prior agreement falls below 80%, and over 15% of items can be answered correctly without any image, causing accuracy to overestimate prior resistance. To address these limitations, we introduce ViSiP-Bench, a benchmark grounded in rare-but-real scenarios across six prior categories, with a model-aware dynamic prior induction protocol and three metrics that more precisely and continuously measure prior resistance than accuracy alone. Evaluating 24 models reveals that prior adherence is pervasive, does not improve with scale, and larger models tend to either strongly override their priors or firmly adhere to them. We therefore propose MAPS, which identifies each model's strongest priors and fine-tunes on factually grounded knowledge to soften excessive prior reliance without disrupting general capabilities.
1 Introduction
Language priors can override visual evidence, while existing benchmarks assume shared priors and rely on accuracy alone. The section introduces ViSiP-Bench and MAPS to measure and mitigate prior reliance.
1 Introduction
- Prior benchmarks assume shared model priors and use accuracy alone, despite below-80% cross-model prior agreement and over 15% no-image-correct items.No-image correct responses arise from chance success, low-quality framing, and anomalous priors.Source: §1 · pp. 1–2
- ViSiP-Bench uses rare real-world scenes across six MLLM-tailored prior domains, with human-reviewed samples and a model-specific Blank-condition prior.The dataset supplements web-retrieved scenes with generated images across visual styles.Source: §1 · p. 2
- Its dynamic protocol measures prior adherence with PAR and filtered accuracy, and visual-evidence confidence shifts with VIG.Items answered correctly under the Blank condition are filtered out to refine PAR and accuracy.Source: §1 · p. 2
- Across ViSiP-Bench categories, prior adherence is pervasive and does not improve with scale; prompt strategies yield marginal gains, while chain-of-thought amplifies adherence.Larger models adhere more strongly on several categories, and inference-time strategies alone are insufficient.Source: §1 · p. 2
- MAPS substantially improves VIG and reduces PAR on ViSiP-Bench while preserving reasoning and general-vision benchmark performance.It softens excessive prior reliance with targeted, factually grounded knowledge of rare-but-real scenarios.Source: §1 · p. 2
2 Related Works
Related work describes language priors as useful for reasoning but potentially misleading in perception-critical tasks. Prior benchmarks share assumptions and accuracy-based evaluation, while mitigation methods intervene at decoding, attention, or training.
2.1 Language Priors in MLLMs
- Language priors can benefit high-level reasoning yet mislead perception-critical tasks; studies locate visual influence in the transformer stack and associate extended chain-of-thought with reduced visual attention and amplified hallucination.Han et al. (2026), Long et al. (2026), and Xu et al. (2026b) report these findings.Source: §2.1 · p. 2
2.2 Benchmarking Language Priors
- Existing benchmarks vary in stimuli and protocols but assume shared priors across models and rely solely on accuracy.Examples include HallusionBench (Guan et al., 2024), VLind-Bench (Lee et al., 2025), and VisuLogic (Xu et al., 2026a).Source: §2.2 · p. 3
2.3 Mitigating Language Priors
- Mitigation methods intervene during decoding, redirect attention toward visual tokens, or modify models through training objectives.The surveyed training approaches include preference optimization, counterfactual training, and visual grounding objectives.Source: §2.3 · p. 3
3 ViSiP-Bench
ViSiP-Bench evaluates whether models follow their own language priors when visual evidence conflicts, using rare-but-real scenarios and model-aware measures. Across 24 models, prior adherence is widespread and does not reliably decrease with scale.
3 ViSiP-Bench
- Prior agreement falls below 80%, over 15% of existing items are answerable without images, and impossible counterfactuals conflate prior dependency with reasoning about impossibility.Image-free correct answers arise from chance success, low-quality framing, or models treating the anomalous answer as their own prior.Source: §3 · p. 3
- ViSiP-Bench addresses these issues with rare-but-real scenarios, rigorous quality review, and model-aware metrics that filter out items answerable without an image.Its six domains follow Ilievski et al. (2021).Source: §3 · p. 3, §3.1 · p. 3
3.1 Data Construction
- 4,500 image-question pairs span six prior categories, using web photographs and supplementary generated images with diverse visual styles and reviewed, unambiguous questions.The benchmark comprises 1,200 enumerated priors, each paired with three rare-but-real scenarios.Source: §3.1 · pp. 3–4
3.2 Dynamic Prior Induction and Metrics
- Dynamic prior induction elicits each model’s own blank-condition answer; PAR measures adherence on genuine conflicts, filtered accuracy excludes image-free correct answers, and VIG measures image-induced log-odds change.VIG requires token-level probabilities and applies only to open-source models; PAR and filtered accuracy apply to all models.Source: §3.2 · pp. 4–5
3.3 Evaluation
- Prior resistance varies across categories but remains largely consistent across models; Scientific & Mathematic has the highest PAR, while Physical Properties is most amenable to visual override.Larger models also show no consistent advantage in prior resistance as model size increases.Source: §3.3 · p. 5, Figure 5 · p. 6
- Larger models show VIG polarization: more samples have high VIG, while low-VIG samples do not consistently decrease and may increase.The pattern indicates that larger models can either respond strongly to visual evidence or firmly adhere to their language priors.Source: §3.3 · p. 5, Figure 5 · p. 6
4 Model-Aware Prior Softening
MAPS targets priors that interfere with visual perception by enriching models with rare-but-real knowledge rather than suppressing prior knowledge indiscriminately. Its pipeline identifies model-specific strong priors, generates grounded counterexamples, and fine-tunes on them.
4.1 Motivation
- MAPS softens interfering priors by adding rare-but-real knowledge while leaving existing knowledge intact, rather than suppressing priors indiscriminately.The motivation is that directly suppressing prior-driven outputs may disrupt real-world understanding and general capabilities.Source: §4.1 · p. 6
4.2 Method
- Questions elicit each model’s prior probabilities; questions with an option above threshold α are selected as high-confidence priors for targeted knowledge generation.Claude Opus 4.7 generates each question with 10 plausible answer options.Source: §4.2 · p. 6
- For each selected prior, Claude Opus 4.7 generates 20 factually grounded rare-but-real descriptions, kept disjoint from ViSiP-Bench, then aggregates them for model-specific LoRA fine-tuning.Each generated sentence serves as a supervised training target to enrich world knowledge.Source: §4.2 · pp. 6–7
5 Experiments
Experiments compare inference-time and training-based interventions on ViSiP-Bench, then test MAPS across models and generalization benchmarks. MAPS improves prior resistance while largely preserving general capabilities, with the probing threshold affecting the balance.
5.1 Experimental Setup
- Experiments use Qwen3-VL-8B-Instruct for MAPS fine-tuning and assess generalization on two reasoning and two general-vision benchmarks.MAPS uses LoRA for two epochs with a default prior-probing threshold of α = 0.90; six additional models are tested under the same configuration.Source: §5.1 · p. 7, Table 4 · p. 8
5.2 Main Results
- CoT consistently hurts performance, while VFP and AS yield marginal gains; VCD and PAI perform best among inference-time methods, but overall gains remain limited.The reported results suggest decoding-stage suppression of prior-driven logits is more effective than visual instruction or attention redistribution alone.Source: §5.2 · pp. 7–8
- MAPS achieves the strongest PAR and VIG improvements while keeping FC at 154, close to the baseline; Post-hoc raises FC from 147 to 385.Extending MAPS to six additional models across three architecture families consistently reduces PAR and improves VIG, with FC close to each base model.Source: §5.2 · p. 8
5.3 Further Analysis
- MAPS keeps performance close to the base model across four reasoning and general-vision benchmarks, with the largest absolute change across six additional models reaching only 0.87 points.Post-hoc degrades all four benchmarks, while NaPO declines to a lesser extent.Source: §5.3 · p. 8, Table 4 · p. 8
- With training data quantity controlled, α = 0.50 selects priors indiscriminately and α = 0.95 covers too few; α = 0.90 gives the best reported balance.The balance maximizes visual-perception improvement while preserving reasoning and general capabilities.Source: §5.3 · pp. 8–9, Table 5 · p. 8
6 Conclusion
ViSiP-Bench addresses shared-prior and accuracy-only assumptions with model-aware evaluation of rare-but-real scenarios. Across 24 models, prior adherence remains pervasive and does not diminish with scale; MAPS improves visual reliance while preserving broader capabilities.
6 Conclusion
- Cross-model prior agreement falls below 80%, and over 15% of existing-benchmark items can be answered without an image.ViSiP-Bench uses rare-but-real scenarios, a model-aware dynamic prior induction protocol, and three complementary metrics for prior resistance.Source: §6 · p. 9
- Across 24 models, prior adherence is pervasive and does not diminish with scale; larger models show VIG Polarization, either decisively overriding priors or adhering firmly.PAR and Filtered Accuracy assess decision-level prior resistance conditioned on each model’s elicited prior, while VIG measures confidence shifts induced by visual evidence.Source: §6 · p. 9
- MAPS softens each model’s strongest priors with targeted, factually grounded rare-but-real knowledge rather than indiscriminately suppressing priors.It substantially improves PAR and VIG while preserving reasoning and general-vision performance across seven models spanning three architecture families.Source: §6 · p. 9
Limitations
The benchmark’s fixed taxonomy and VIG’s reliance on token-level output probabilities limit the scope of the paper’s coverage and evaluation.
Limitations
- The fixed six-category taxonomy cannot capture every form of world knowledge, leaving the reach of the reported adherence patterns beyond its scope unresolved.The authors leave whether those patterns extend to excluded cases as an open question for future benchmarks.Source: Limitations · p. 9
- VIG requires token-level output probabilities, so it cannot be applied to closed models accessed only through APIs.Consequently, the reported VIG Polarization phenomenon remains unverified for proprietary models, whose confidence dynamics may differ.Source: Limitations · p. 9
Ethical Considerations
The paper describes its use of generative tools, image and annotation review, and voluntary annotator compensation.
Ethical Considerations
- Generative models helped enumerate priors and scenarios and synthesize images; web-sourced and generated content underwent three-criteria quality review.The review also filtered inappropriate or misleading generations.Source: Ethical Considerations · p. 9
- Voluntary annotators were informed of the task and fairly compensated, and the annotation work involved ordinary, uncommon real-world scenes.The authors state that generative AI tools were used only for wording and clarity edits, not the underlying research.Source: Ethical Considerations · p. 9
A Data Quality Analysis
The appendix screens correct answers given without an image to distinguish weak-prior chance responses, flawed question framing, and anomalous priors. It also describes representative cases and four visual styles used in ViSiP-Bench.
A.1 Identification of No-Image Correct Responses
- Blank-condition correct answers are screened through human question-quality review, probability analysis, and anomalous-prior identification.Annotators flag weak or ambiguous priors and wording that hints at an anomaly; probability distributions then distinguish chance responses from strong anomalous priors.Source: §A.1 · pp. 11–12
A.2 Typical Cases
- Table 6 provides representative examples of the three sources of no-image correct responses.The sources are Chance Success, Low-quality Framing, and Anomalous Prior.Source: §A.2 · p. 12, Table 6 · p. 12
B Visual Style Examples
- ViSiP-Bench uses four visual styles to reduce confounding by image style or source, with prompts listed in Table 7 and examples in Figure 7.Figure 7 shows one representative image per style, each depicting a rare-but-real scenario that conflicts with commonsense priors.Source: B Visual Style Examples · p. 12, Table 7 · p. 12
C Cross-Model and Cross-Family Validation of MAPS
MAPS is evaluated across six models from three architecture families under a shared fine-tuning configuration. Across these models, it improves ViSiP-Bench results while preserving performance on general reasoning and vision benchmarks; the appendix also supplies annotation guidelines.
C Cross-Model and Cross-Family Validation of MAPS
- Six additional models spanning three architecture families use the same prior-probing threshold and two-epoch LoRA fine-tuning, without per-model hyperparameter adjustment.The evaluation includes InternVL3.5, InternVL3, Qwen2.5-VL, Qwen3.5, and LLaVA-OneVision models.Source: C Cross-Model and Cross-Family… · pp. 12–13
D Annotation Guidelines
- Annotation guidelines require clear image-grounded questions, neutral wording, four plausible choices, and unanimous reviewer agreement before inclusion.They reject implicit anomaly hints, eliminable distractors, ambiguous references, and images that do not support one clear answer.Source: D Annotation Guidelines · pp. 13–15
C Cross-Model and Cross-Family Validation of MAPS
- Across six models, MAPS reduces PAR by 3.44 to 8.06 points and improves Filtered Accuracy by 3.62 to 8.15 points, while FC rises by only 2 to 7 samples.VIG improves for every model; the reported PAR reduction is smaller for LLaVA-OneVision-7B, which has the highest base PAR among the six.Source: C Cross-Model and Cross-Family… · p. 13, Table 8 · p. 14
- Across 24 model–benchmark pairs, the largest absolute generalization change is 0.87 points, with no consistent direction of degradation.The benchmarks cover reasoning and general vision; changes vary in sign across models and benchmarks.Source: C Cross-Model and Cross-Family… · p. 13, Table 9 · p. 14
E Per-Category Analysis
Per-category results show that prior resistance varies by category. The prompt template is held identical between Blank and Visual conditions so response differences track image presence rather than prompt differences.
E Per-Category Analysis
- Scientific & Mathematic has the highest PAR across model families, while Physical Properties is most amenable to visual override.The category pattern remains largely consistent across models, suggesting category-level difficulty is intrinsic to prior type rather than model-specific.Source: E Per-Category Analysis · p. 15
F Prompt Template
- The Blank and Visual conditions use an identical prompt, which asks respondents without an image to guess from prior knowledge.The template presents the question and four options, then instructs respondents to guess if they cannot see the image.Source: F Prompt Template · pp. 15–16