PaperlayerView exact revision on arXiv

Source-linked AI summary

When Words Speak Louder than Images: Towards Understanding Language Bias in Vision-Language Models

Yizhou Fang, Siyue Chen, Zimo Qi, Zhiyu Xue, Xi Chen, Guangliang Liu

arXiv:2609.35272v1cs.CL

TL;DR

VLMs can favor linguistic cues over visual evidence, and conflicting prior findings are difficult to reconcile when bias propagation through black-box inference is hard to trace. The paper analyzes word completion with a four-stage diagnostic framework, examining linguistic priors and cross-modal coverage. It reports that coverage relates to progression through inference, while strong priors can diminish its benefits before final decisions.

  • Problem

    Conflicting findings on understanding and mitigating language bias, alongside difficulty tracing its propagation through black-box VLMs, leave its mechanisms challenging to analyze.

    Source: §1 · p. 2

  • Method

    The paper analyzes word completion using a four-stage diagnostic framework to examine linguistic priors and cross-modal coverage across inference.

    Source: §1 · p. 2

  • Results

    Higher cross-modal coverage is associated with progression to later stages, while linguistic priors can diminish its benefits before decision and allow recognized visual evidence to be overridden.

    Source: §1 · p. 3

  • Takeaways & Limitations

    Reducing linguistic influence may also limit the knowledge and generalization ability that VLMs gain from large language models.

    Source: §4 · p. 9

    The controlled reformulations show that perceptron–decision-gap cases can occur, but do not estimate their frequency or establish linguistic prior as their sole cause.

    Source: §3.2 · p. 8

Abstract

from arXiv · show

Despite substantial progress across downstream applications, vision-language models (VLMs) remain susceptible to language bias, often prioritizing linguistic cues over visual evidence and consequently producing incorrect predictions. Prior studies have proposed various approaches to understanding and mitigating language bias in VLMs, yet their findings often conflict due to the difficulty of tracing how language bias propagates within black-box VLMs. Building on the word completion task, we trace how language bias propagates through VLM inference by (1) proposing a diagnostic framework that decomposes the inference process into four distinct yet interdependent stages to trace the propagation of language bias; and (2) examining how two key factors underlying language bias, i.e., linguistic priors and cross-modal coverage, evolve across these stages and ultimately give rise to incorrect predictions. The linguistic prior captures the strength of statistical bias induced by the language model component of a VLM and represents the origin of language bias, whereas cross-modal coverage measures the extent to which linguistic cues cover the visual content. By decomposing inference into four stages and characterizing the interplay between linguistic priors and cross-modal coverage across these stages, we propose a systematic framework for tracing the propagation of language bias throughout the inference process; and uncover the underlying mechanism of language bias by revealing the interplay between linguistic priors and cross-modal coverage.

1 INTRODUCTION

VLMs can prioritize linguistic cues over visual evidence, while conflicting findings and the black-box nature of these models make language-bias propagation difficult to trace. This paper introduces a four-stage diagnostic framework and examines how linguistic priors and cross-modal coverage relate to bias across inference.

1 INTRODUCTION

  • Prior approaches report conflicting findings on visual attention, cross-modal associations, and counterfactual data, leaving bias propagation difficult to trace in black-box VLMs.Counterfactual analyses rely on an unverified assumption about whether the data-generating function can adequately account for counterfactual data.Source: §1 · p. 2
  • The framework decomposes inference into visual evidence identification, visual relationship identification, misleading-cue exclusion, and decision, while tracking linguistic priors and cross-modal coverage.Linguistic priors capture statistical bias from the LLM component; cross-modal coverage measures how much linguistic cues describe visual content.Source: §1 · p. 2
  • The four-stage framework suggests that successful inference requires identifying relevant visual relationships and excluding misleading cues, not merely forming visual–linguistic associations.It also explains why greater visual attention alone may not prevent reliance on linguistic priors.Source: §1 · p. 3
  • Linguistic priors can diminish the benefits of cross-modal coverage before the decision stage, allowing correctly recognized visual evidence to be overridden in the final prediction.Higher cross-modal coverage is associated with progression to later inference stages.Source: §1 · p. 3

2 PRELIMINARY

This preliminary section establishes a multimodal word-completion setting and a four-stage framework for tracing language bias, then operationalizes linguistic prior and cross-modal coverage. No-bias cases outperform biased cases at each of the three measured stages, supporting the framework’s diagnostic value.

2 PRELIMINARY

  • The analysis uses Ma et al. (2023)’s multimodal word-completion task, with image, blank-containing text, gold completion, and separate multimodal and text-only predictions.The notation also defines the text-only probability of a candidate completion and names the linguistic-prior and cross-modal-coverage scores.Source: §2 · p. 3

2.1 A FOUR-STAGE DIAGNOSTIC FRAMEWORK

  • The framework separates visual evidence identification, visual relationship identification, misleading-cue exclusion, and final decision, treating the first three as visual perception and S4 as prediction.Its stages distinguish identifying relevant evidence, linking it to visual relationships, rejecting misleading cues, and producing the final completion.Source: §2.1 · pp. 3–4

2.2 EVALUATION OF VLM PERFORMANCE AT EACH STAGE

  • Natural-language probes operationalize each stage, while cases are labeled by whether multimodal input preserves or corrects an incorrect text-only completion.The no-bias cases perform better than biased cases at all three measured stages, with the largest average gap at S3.Source: §2.2 · pp. 4–5
  • No-bias cases exceed biased cases by 9.8 percentage points at S1, 10.7 at S2, and 33.7 at S3 across four VLMs.The stages measure visual evidence identification, visual relationship identification, and misleading visual cue exclusion, respectively.Source: §2.2 · p. 5, Table 1 · p. 5

2.3 LINGUISTIC PRIOR

  • The linguistic-prior score measures the text-only model’s relative preference for its prediction over the gold completion, with larger scores indicating stronger preference.The paper frames linguistic prior as statistical bias induced by the VLM’s language-model component.Source: §2.3 · p. 5

2.4 CROSS-MODAL COVERAGE

  • Cross-modal coverage is the proportion of visual cues relevant to the gold completion that are already expressed in the text.Annotators identify relevant cues, including those missed by initial LLM verbalization, and then determine which are covered by the textual input.Source: §2.4 · pp. 5–6

3 MECHANISTIC ANALYSIS

The section analyzes language bias through cross-modal coverage and linguistic priors, using stage-wise analyses and controlled reformulations. Greater coverage is associated with stronger visual evidence perception, while strong linguistic priors can still interfere with the final decision after successful visual-perception probes.

3.1 CROSS-MODAL COVERAGE ENHANCES VISUAL EVIDENCE PERCEPTION

  • Across five models, greater cross-modal coverage is positively associated with correctness for four, while stronger linguistic prior is negatively associated with correctness for all five.The logistic regression jointly considers both factors.Source: §3.1 · p. 7
  • As cross-modal coverage increases, diagnostic-stage performance generally improves in both biased and no-bias noun-target cases, most clearly at S3.The same qualitative pattern appears in additional Gemma 3 12B and OneVision 1.5 8B results.Source: §3.1 · p. 7, Figure 5 · p. 7
  • Among incorrect noun-target completions, errors assigned to S3 or S4 increase with coverage, and those stages remain prominent even at higher coverage.Some S4 cases answer all S1–S3 probes correctly but complete the original caption incorrectly.Source: §3.1 · p. 7, Figure 5 · p. 7

3.2 THE LINGUISTIC PRIOR OVERWRITES VISUAL EVIDENCE PERCEPTRON

  • With all S1–S3 judgments correct, final accuracy is consistently lower in the very-high-prior regime across all five models.The table compares final completion accuracy for pling < 0.99 and pling ≥0.99.Source: §3.2 · p. 8, Table 3 · p. 8
  • The study constructs adversarial reformulation pairs that preserve cross-modal coverage while inducing different linguistic priors, retaining successful S1–S3 judgments.It creates 100 pairs per model, or 500 across the five evaluated models.Source: §3.2 · p. 8
  • Across the controlled examples, weakening the linguistic prior can change the final completion despite unchanged coverage and S1–S3 judgments.These examples demonstrate that such cases can occur, not how frequently they occur in the full evaluation set; surface-form changes also prevent attributing the gap solely to linguistic prior.Source: §3.2 · p. 8

4 DISCUSSION

The discussion emphasizes that language bias may become harder to observe in stronger models under simple visual–linguistic conflicts. It also describes a trade-off: linguistic priors can support knowledge and generalization while contributing to instance-level bias.

Evaluation scope

  • Language bias is increasingly difficult to observe in stronger VLMs under relatively straightforward visual–linguistic conflicts.The discussion suggests evaluating harder cases with incomplete, ambiguous, distributed, or compositional visual evidence.Source: §4 · pp. 8–9

The dual role of linguistic priors in VLMs

  • Linguistic priors that contribute to biased predictions are also described as a major source of LLM knowledge and generalization, so suppressing them may limit those capabilities.The discussion frames this as a trade-off rather than an isolated failure mode.Source: §4 · p. 9

Revisiting vision–language grounding

  • The discussion suggests grounding semantic spaces in linguistic structures and cognitive principles, rather than relying only on statistical image–text associations.Such mechanisms could potentially reduce language bias while preserving, or improving, knowledge utilization.Source: §4 · p. 9

5 CONCLUSION

The conclusion presents a four-stage diagnostic framework and examines linguistic priors and cross-modal coverage in a word-completion task. It reports an association between greater coverage and improved visual evidence perception, alongside a gap in which priors can diminish coverage’s benefits at the decision stage.

Framework and factors

  • The paper studies language bias in VLMs through a word-completion task using a four-stage diagnostic framework and two factors: linguistic priors and cross-modal coverage.Linguistic priors capture statistical biases from the LLM component; cross-modal coverage measures how much visual information is represented in the textual input.Source: §5 · p. 9

Findings

  • Greater cross-modal coverage is associated with improved visual evidence perception, while linguistic priors can diminish its benefits and contribute to a perception–decision gap.The conclusion describes this gap as correctly recognized visual evidence being overridden in the final decision.Source: §5 · p. 9

Scope

  • The paper reports using an existing benchmark and states that it does not involve sensitive personal data or human-subject deployment.Human annotations are used to evaluate task-relevant visual and textual information and model correctness.Source: ETHICS STATEMENT · p. 10

REPRODUCIBILITY STATEMENT

The reproducibility statement points readers to definitions, annotation procedures, additional results, examples, and experimental details in the paper and appendix.

Reproducibility

A.1 RELATED WORKS

Related work studies behavioral language bias, mitigation strategies, and mechanisms inside language and vision-language models. The paper positions its analysis alongside evaluations of language dependence and interventions on model computation.

Language Bias in Vision–Language Models

  • Behavioral evaluations probe misleading-text conflicts, commonsense-driven answers, retrieval shortcuts, compositional failures, and inconsistencies across related responses (Deng et al., 2025; Liu et al., 2025b; Lin et al., 2024b; Yuksekgonul et al., 2023; Guan et al., 2024).Benchmarks also use capability controls and counterfactual or out-of-distribution cases to examine when predictions follow linguistic cues or visual evidence (Lee et al., 2025; Liu et al., 2025b; Luo et al., 2025).Source: §A.1 · p. 16
  • Mitigation methods use training, decoding, added visual evidence, and output revision, but gains can depend on the evaluation setting (Wu et al., 2026; Leng et al., 2024; Li et al., 2025; Ghosh et al., 2025; Zhou et al., 2024; Geigle et al., 2024).The paper complements these evaluations by diagnosing functional judgments and examining how patterns vary with linguistic prior and cross-modal coverage.Source: §A.1 · p. 16

Mechanistic Analysis of Language and Vision–Language Models

  • Mechanistic studies examine representations, circuits, internal interventions, and visual-information transfer; explanation reliability can depend on the evaluation metric and corruption method.Other work finds visual information may remain encoded even when the final prediction contradicts it, with its influence depending on where and how it is used (Nooralahzadeh et al., 2026; Wang & Xu, 2026).Source: §A.1 · pp. 16–17

A.2 DIAGNOSTIC PROMPT TEMPLATES AND EXAMPLES

Diagnostic questions use shared prompt templates and item-specific wording while keeping each stage’s functional judgment fixed. The templates distinguish correct from misleading visual relationships and define the concepts instantiated for each item.

A.2 DIAGNOSTIC PROMPT TEMPLATES AND EXAMPLES

  • Annotators identify each item’s ground-truth concept, competing concept, and associated visual entities and relationships using shared templates.Surface wording varies by semantic type, but the functional judgment at each stage remains fixed.Source: §A.2 · p. 17
  • S2 tests correct visual relationships, whereas S3 creates misleading relationships by exchanging the concepts paired with corresponding entities.Bracketed template fields are instantiated from each word-completion item.Source: Table 4 · p. 17
  • S1 adapts its surface form to the ground-truth concept’s semantic type, while S4 uses the original image and incomplete caption without an additional diagnostic prompt.The target entity is queried by the original completion task; the competing concept is visually present but forms the alternative interpretation.Source: §A.2 · p. 17

A.3 CROSS-MODAL COVERAGE ANNOTATION

The annotation example organizes image content into candidate visual cues, selects a visual-fact set, and measures how many selected facts appear in the caption. A separate figure reports stage-wise results for additional model sizes.

A.3 CROSS-MODAL COVERAGE ANNOTATION

  • The example image’s observable entities, attributes, and relations are organized into 24 candidate visual cues.Source: Figure 6 · p. 18
  • Five facts are selected from those cues to form the visual-fact set F: an adult woman, baby, woman holding baby, white signed surface, and marker.Source: Figure 7 · p. 18
  • Four of the five selected facts appear in the non-blank caption text, giving cross-modal coverage of 4/5 = 0.8.Source: Figure 8 · p. 19
  • For Gemma 3 12B and OneVision 1.5 8B, S1–S3 performance across cross-modal coverage intervals shows the same qualitative pattern observed in the main-text models.Source: Figure 9 · p. 20

A.4 ADDITIONAL STAGE-WISE RESULTS

Additional model sizes reproduce the reported stage-wise pattern, while ten counterfactual examples hold visual content and coverage fixed as caption reformulations lower competing-word priors. The examples are qualitative evidence, not a population repair-rate estimate.

A.4 ADDITIONAL STAGE-WISE RESULTS

  • S1 and S2 remain relatively stable across coverage levels, while S3 improves more clearly as coverage increases for Gemma 3 12B and OneVision 1.5 8B.This matches the qualitative pattern in the representative main-text models.Source: §A.4 · p. 20

A.5 PRIOR-COUNTERFACTUAL EXAMPLES

  • Ten counterfactual cases on distinct images hold the image, visual proposition, target, slot, gold answer, and coverage fixed while reformulating captions to lower the competing word’s prior.The complete five-model witness set and validation records are in the supplementary material.Source: §A.5 · p. 20
  • In Case 3, the prior of paddle falls from 0.9399 to 0.0067 after caption reformulation; in Case 9, watches falls from > 0.9999 to 7.3×10−7.Source: §A.5 · pp. 21–22
  • Recognition, Grounding, Relational Binding, and Exclusion succeed before and after reformulation in all ten cases, but the gallery is not a population repair-rate estimate.Source: §A.5 · p. 22
Loading 2609.35272v1…