PaperlayerView exact revision on arXiv

Source-linked AI summary

Vision-Language Models Integrate Conflicting Evidence Like Biased Bayesian Observers

Josh Barua, Kyle Mahowald, William Rudman

arXiv:2610.19407v1cs.CLcs.AIcs.CV

TL;DR

Existing modality-conflict evaluations hold evidence fixed, limiting what they reveal about how VLM preferences depend on evidence certainty. This paper models arbitration as Bayesian cue combination with modality-specific biases and prior beliefs; the model predicts preferences across unseen degradation combinations and transfers to new unreliability types without refitting.

  • Problem

    Because existing modality-conflict benchmarks hold text and image evidence fixed, they cannot show how preference depends on evidence certainty.

    Source: §2 · p. 2

  • Method

    The paper derives a Bayesian cue-combination model that treats text and image as conditionally independent evidence, weighted by modality-specific biases and combined with prior beliefs.

    Source: §1 · p. 2, §3.1 · p. 4

  • Results

    The two-parameter model predicts per-example preferences across datasets, architectures, and held-out degradation combinations, and transfers to new unreliability types without refitting.

    Source: §5 · p. 8

  • Takeaways & Limitations

    Separating evidence strength from stable modality-specific integration biases reconciles text- and vision-dominant preferences under one combination rule.

    Source: §5 · p. 8

    The modality weights are fitted empirically rather than derived normatively, and the log-likelihood preference measure only approximates posterior log-odds and does not directly measure open-ended generation frequencies.

    Source: Limitations · p. 9

Abstract

from arXiv · show

Vision-language models must reconcile conflicting text and images. Prior work has characterized the resulting behavior in terms of modality biases--e.g., a "blind faith in text"--but why preferences vary across models, domains, and input conditions remains unclear. We develop a predictive account from a Bayesian perspective, positing that VLMs combine conditionally independent evidence from each modality with prior beliefs, subject to modality-specific biases. The resulting two-parameter cue combination model accurately predicts VLM preferences under unseen combinations of input degradation and transfers to new forms of unreliability without refitting. Separating evidence strength from integration bias explains how VLMs can follow text even while weighting visual evidence more heavily, reconciling reports of text and vision dominance under a single combination rule.

1 Introduction

Prior accounts describe VLM modality preferences as biases, but this paper asks whether Bayesian cue integration can explain how preferences vary with reliability. Its model separates evidence strength from modality-specific weighting to predict arbitration between conflicting text and images.

1 Introduction

  • The paper adapts Bayesian cue integration from human perception, where influence shifts with cue reliability and additional parameters capture systematic biases and individual differences.Earlier accounts often framed VLM conflicts as one modality dominating another, although reported preferences also vary across models and queries.Source: §1 · p. 1
  • A two-parameter Bayesian cue-combination model predicts preferences under unseen degradation combinations and transfers to new unreliability types without refitting.It treats modality evidence as conditionally independent, weights it with modality-specific biases, and combines it with prior beliefs.Source: §1 · p. 2
  • Separating evidence strength from integration bias explains how a VLM can follow text even when visual evidence receives greater weight.The account recasts apparent “blind faith in text” as largely reflecting stronger textual evidence, which modality bias can amplify or offset.Source: §1 · p. 2

2 Do Modality Preferences Track Evidence Reliability?

The section tests whether VLM preferences track independently manipulated text and image reliability in conflicting VQA. Preferences shift with degradation and vary across models and datasets, motivating a framework that separates evidence strength from weighting.

2.1 Task, Data, and Models

  • Conflicting VQA pairs a question with text and an image that support different answers, isolating source arbitration from answer accuracy.The dataset contains 2,765 examples from DocVQA, VQAv2, and RealworldQA, spanning document images, natural photographs, and real-world scenes.Source: §2.1 · pp. 2–3
  • The study evaluates four open-weight VLMs that differ in vision encoder, language-model attention, and vision–language fusion.The models are Qwen2.5-VL-3B, Qwen3.5-4B, Gemma3-4B, and GLM-4.6V-Flash.Source: §2.1 · p. 3

2.2 Manipulating Evidence Reliability

  • Text reliability is degraded by nested random word replacement, while image reliability is degraded with isotropic Gaussian blur.Text noise levels p range from 0.0 to 1.0 and blur levels σ from 0 to 32 pixels, yielding a 6 × 6 grid of conditions.Source: §2.2 · p. 3

2.3 Measuring Modality Preference

  • The joint probe measures the log-likelihood preference between the text- and vision-supported answers, with its sign indicating the preferred answer and magnitude indicating preference strength.The study converts the log-odds to PJ, the probability of choosing the text-supported answer.Source: §2.3 · p. 3

2.4 Results

  • Corrupting text shifts preference toward the image-supported answer, while blurring images shifts it toward the text-supported answer.Preferences vary across models and datasets; on DocVQA, mild blur (σ=2) moves Qwen3.5-4B from image preference to text preference, while shifts on other datasets are more gradual.Source: Figure 2 · p. 3, §2.4 · p. 4
  • Qwen2.5-VL-3B and Gemma3-4B lean more toward text than Qwen3.5-4B and GLM-4.6V-Flash, while Qwen3.5-4B's clean-input preference differs between DocVQA and RealworldQA.DocVQA's fine-grained details may help explain why mild blur is enough to shift Qwen3.5-4B toward text there.Source: §2.4 · p. 4

3 A Bayesian Cue Combination Model

The section develops a Bayesian cue-combination account in which VLMs combine text and visual evidence with prior beliefs, then allows modality-specific weighting to capture systematic deviations from a calibrated observer.

3 A Bayesian Cue Combination Model

  • The model adapts human cue-combination accounts to VLMs, starting with a calibrated observer and relaxing its assumptions into explicit, testable deviations.The account is motivated by reliability-weighted cue integration in human perception.Source: §3 · p. 4

3.1 A Calibrated Bayesian Observer

  • The calibrated observer treats text and image as conditionally independent given the answer, so their Bayes factors add to prior log-odds with unit weight.The unimodal probes provide text and vision Bayes factors after subtracting the prior from each probe.Source: §3.1 · p. 4

3.2 Allowing for Integration Bias

  • Shared nonnegative βT and βV weights scale each example’s text and vision evidence, separating evidence reliability from integration bias.The ratio βT/βV measures integration bias; values above one indicate greater weight on text evidence.Source: §3.2 · p. 5

3.3 Competing Predictors

  • Competing predictors test selection versus blending, interaction, unit modality weights, and prior/intercept restrictions to assess the model’s assumptions.The alternatives include a more-confident-single-modality probe, bilinear cue combination, calibrated Naive Bayes, and unconstrained linear regression.Source: §3.3 · p. 5

4 Testing the Bayesian Model

The Bayesian model is tested on held-out examples and degradation combinations, then compared with alternative integration rules and new reliability manipulations. Its results support linear cue combination as a predictive account and separate evidence strength from modality weighting.

4.1 Fitting and Evaluation

  • The evaluation fits on 11 boundary cells with at most one modality degraded and tests on 25 interior cells where both are degraded.The protocol targets generalization to new examples and unseen combinations of the same degradation levels; test-set R2 and KL divergence compare predictions with observations.Source: §4.1 · pp. 5–6

4.2 Predicting Joint Preferences from Unimodal Probes

  • R2 is 0.80–0.86 for Qwen models and GLM-4.6V, and 0.69 for Gemma3; predicted preference boundaries closely track observed boundaries across three datasets.Figure 3 shows empirical and predicted preferences for Qwen3.5-4B; Table 1 reports fit metrics averaged across datasets.Source: Figure 3 · p. 6, §4.2 · p. 6, Table 1 · p. 6

4.3 Comparing Alternative Explanations

  • The best single probe trails by 0.22–0.35 in R2 and calibrated naive Bayes by 0.14–0.32, while added interaction or regression flexibility improves R2 by at most 0.001 or 0.017.For three of four VLMs, the unconstrained regression’s prior coefficient is close to one; Qwen3.5 is the exception at 0.53–0.66, and enforcing the Bayesian constraint costs little predictive accuracy.Source: §4.3 · pp. 6–7

4.4 Do Modality Weights Transfer?

  • Refitted weights reach R2 of 0.68–0.85, while weights transferred from text noise × blur come within 0.021 of refitting for every VLM.The refitted weights are also close to the transferred weights, supporting their stability within each VLM–dataset pair.Source: §4.4 · p. 7, Table 2 · p. 7

4.5 Explaining Modality Preferences

  • Clean-input mean textual evidence exceeds visual evidence by 1.4–3.8×, while fitted modality weights vary: Qwen3.5 and GLM-4.6V weight vision more heavily, unlike Gemma3’s near-equal or modestly text-favoring weights.Gemma3’s βT / βV is 0.96–1.27, compared with 0.56–0.68 for Qwen3.5 and GLM-4.6V; stronger textual evidence can therefore yield text-following despite vision-favoring weights.Source: §4.5 · pp. 7–8, Table 3 · p. 8
  • For Qwen3.5, the clean-input evidence ratio rises from 1.44 on DocVQA to 2.03 on VQAv2 and 2.88 on RealworldQA, while weighted mean contribution ratios rise from 0.95 to 1.14 and 1.94.The growing textual advantage qualitatively matches its increasing text preference; evidence strength can shift preference even with fixed modality weights.Source: §4.5 · p. 8

5 Conclusion

The paper presents a two-parameter Bayesian cue combination model for predicting how VLMs arbitrate between conflicting text and images. Its results characterize modality preference as an interaction between evidence strength and stable modality-specific integration biases.

5 Conclusion

  • The two-parameter model predicts per-example joint preferences across datasets, architectures, and held-out degradation combinations, and transfers fitted weights to new unreliability forms without refitting.The distinction between evidence strength and integration bias explains how apparent text-following can coexist with greater visual weighting under one combination rule.Source: §5 · p. 8

Limitations

The framework explains VLM behavior without claiming optimality, but its weights are empirical and its preference measure only approximates posterior log-odds.

Limitations

  • The framework is intended to explain VLM behavior, not to establish that models act optimally or perfectly rationally.Its modality weights are fitted empirically because the framework does not yet predict how models weight text and vision.Source: Limitations · p. 9
  • Preferences use differences in length-normalized candidate-answer log-likelihoods, which only approximately correspond to posterior log-odds and do not directly measure open-ended generation frequencies.The authors report good agreement with multiple-choice reasoning when candidate answers are supplied as options.Source: Limitations · p. 9
  • The study manipulates supplied evidence extensively but does not independently manipulate prior beliefs; conflicts with parametric knowledge remain a complementary test setting.Applying the framework to counterfactual benchmarks could test whether elicited priors and measured evidence account for those preferences.Source: Limitations · p. 9

A.1 Dataset Construction

The datasets are converted into conflicting question, text, image, and answer tuples using benchmark-specific procedures, with Gemini 3 Flash used for construction tasks.

A.1 Dataset Construction

  • Both dataset-construction procedures use Gemini 3 Flash, with task-specific generation settings described separately.Table 4 reports example counts and split sizes, excluding repeated degradation variants; its caption specifies 36 cells per example for ID and 11/25 boundary/interior cells for OOD.Source: §A.1 · p. 12, Table 4 · p. 12
  • For DocVQA and VQAv2, source examples already pair questions and images with text corrupted to suggest a different answer; Gemini extracts the text-supported answer from the question, image answer, and passage.The extraction uses temperature 0, a 64-token output limit, and no thinking budget.Source: §A.1 · p. 12
  • For RealworldQA, Gemini rewrites questions as open-ended, restates the visual answer as a phrase, and generates conflicting text supporting a different answer.The generator receives the original image, question, and answer; conversion uses temperature 0.7, a 512-token output limit, and a 1,024-token thinking budget.Source: §A.1 · p. 12

A.2 Dataset Examples

The appendix documents conflicting examples, reliability manipulations, probe scoring and prompting, and checks on the constructed labels and score interpretation.

A.2 Dataset Examples

  • DocVQA, VQAv2, and RealworldQA examples pair image-supported answers with conflicting text-supported answers, including invoice-number, sky-color, and traffic-light examples.Images are resized to approximately 275,998 pixels while preserving aspect ratio to remove the pixel-count confound between document scans and photographs.Source: §A.2 · p. 13, §A.3 · p. 13

A.4 Text Noise and Gaussian Blur

  • Primary reliability conditions cross six Gaussian-blur levels with six text-noise levels, producing 36 conditions per example.Blur ranges from σ=0 to 32 pixels, while text noise replaces 0% to 100% of passage words using nested substitutions.Source: §A.4 · p. 13

A.5 Alternative Reliability Manipulations

  • Transfer experiments replace blur with nested black-pixel occlusion and text noise with generated passages that progressively weaken support for the text answer while preserving fluency.The illustrated ski-brand conflict has Fischer as the visual answer and Rossignol as the text answer.Source: §A.5 · p. 14, Figure 5 · p. 14, Table 5 · p. 14

A.6 Generation Prompts

  • The RealworldQA conversion and textual-relevance degradation prompts are reproduced, including the conversion prompt’s question, visual answer, conflicting passage, and text-supported answer outputs.The textual-degradation prompt bounds output length around the original passage’s word count.Source: Figure 6 · p. 15, §A.6 · p. 15, Figure 7 · p. 16

B.1 Candidate-Answer Scoring

  • Each candidate answer is scored by mean teacher-forced log-likelihood, and each probe contrasts the same text-supported and vision-supported answer strings across prior, unimodal, and joint contexts.Positive scores favor the text answer; negative scores favor the vision answer.Source: §B.1 · p. 17

B.2 Probe Prompt Templates

  • Probe templates use each model’s native chat format, score candidate-answer tokens only, and remain fixed across examples and degradation levels.Gemma3 uses additional assistant prefixes to reduce surface-form effects when eliciting the supplied answer forms.Source: §B.2 · p. 17, Table 6 · p. 17, §B.3 · p. 17

B.3 Gemma3 Prefixes and Prompt Audit

  • Gemma3’s answer probabilities are sensitive to response surface form, so sentence-continuation prefixes are used to reduce formatting effects and elicit candidate answers more naturally.In a matched 90-example audit, the joint-probe median combined probability at the first candidate-token divergence rose from 3.01 × 10^-8 to 0.595 with the prefix.Source: §B.3 · p. 17

B.4 Clean Probe-Sign Agreement

  • Text and image probes recover their intended answer signs on 92–96% and 69–91% of clean examples, respectively, with visual agreement especially lower on RealworldQA.Visual-sign failures may reflect model perception errors or genuine ambiguity in the image evidence.Source: §B.4 · p. 18, Table 7 · p. 18

B.5 Agreement with Multiple-Choice Answers

The authors test whether the joint-probe preference corresponds to multiple-choice answers after reasoning. Agreement is substantial among retained responses, especially when probe scores are strongest.

B.5 Agreement with Multiple-Choice Answers

  • The test compares joint-probe preferences with answers generated after models reason over multiple-choice versions of the conflicting inputs.Each cell in the degradation grid is presented twice with answer labels swapped, and only position-consistent pairs count toward agreement.Source: §B.5 · p. 19
  • 75.3–82.0% of retained multiple-choice pairs agree with sign(sJ), rising to 81.8–93.9% in the highest |sJ| tertile.Retention is 75–80% across the four VLMs; retained pairs require the same answer string under both letter assignments.Source: §B.5 · p. 19, Table 8 · p. 19

C.1 Example Splits and Evaluation Protocols

The protocols keep all degradation variants of each example in the same split, then compare predictors on either all degradation cells or held-out combined-degradation cells. Predictors range from parameter-free baselines to fitted linear, bilinear, unconstrained, and nonlinear models, evaluated with score and preference-distribution metrics.

C.1 Example Splits and Evaluation Protocols

  • OOD training and validation use 11 unimodal boundary cells, while testing uses the 25 interior cells with combined degradation.Under ID, all 36 cells contribute to training, validation, and testing; example IDs remain disjoint across splits.Source: §C.1 · p. 20, Figure 8 · p. 20

C.2 Predictor Definitions and Fitting

  • The five predictors include two parameter-free baselines, constrained linear and bilinear cue combinations, unconstrained linear regression, and a random forest.The linear model uses nonnegative modality weights with a unit prior coefficient and no intercept; the bilinear model adds an evidence interaction.Source: §C.2 · pp. 20–21

C.3 Evaluation Metrics

  • Evaluation reports score-space R2 and KL divergence between observed and predicted two-answer distributions, averaged equally across datasets.Metrics are computed for each VLM–dataset pair on applicable test cells.Source: §C.3 · p. 21

C.4 Refit-Bootstrap Uncertainty

  • Pointwise 95% intervals come from 2,000 refit bootstraps that resample training example IDs while keeping each example’s degradation cells together.Test examples are not resampled or used in the refits.Source: §C.4 · p. 21

D Detailed Main Results

D.1 Per-Dataset Predictor Comparison

The appendix provides per-dataset OOD predictor results and preference-grid visualizations, then reports alternative-predictor coefficients and tests transfer of modality weights. It also compares in-distribution predictors, but the supplied passages do not state the table’s performance values.

D.1 Per-Dataset Predictor Comparison

  • Table 9 gives per-dataset OOD results for all predictors, with KL and R2 comparisons, but the supplied text does not report the values.The table marks the best of three predictors for each comparison.Source: Table 9 · p. 22, §D.1 · p. 22

D.2 OOD Preference Grids

  • Figures 9–11 show empirical and predicted preference grids for three models, using weights fitted on OOD boundary cells and evaluating metrics on 25 interior cells.The displayed grids contain all 36 cells; solid and dashed contours mark empirical and predicted sJ = 0 boundaries.Source: Figure 9 · p. 23, §D.2 · p. 23, Figure 10 · p. 24

D.3 Fitted Weights of Alternative Predictors

  • Table 10 reports fitted bilinear and unconstrained-linear coefficients from example-level OOD fits on 11 boundary cells.The bilinear predictor fixes the prior coefficient at one and the intercept at zero; the unconstrained predictor leaves all four coefficients free.Source: Table 10 · p. 26, §D.3 · p. 26

D.4 Frozen and Refitted Weights

  • Across 21 VLM–dataset pairs, transferred weights’ modality-weight ratio differs from the refitted ratio by a median of 4.5% and a maximum below 10%.Frozen weights come from text noise × blur; refitted weights use relevance × occlusion, with both evaluated on the new grid’s held-out interior examples.Source: Table 11 · p. 27, §D.4 · p. 27

D.5 Cue Weights and Mean Evidence

  • Table 12 compares example-level and cell-level weights with clean held-out evidence and prior summaries; cell-level fitting describes population means, not measurement-error correction.The weights use boundary-cell data, while evidence and prior summaries use clean held-out inputs.Source: Table 12 · p. 28, §D.5 · p. 28

E.1 In-Distribution Predictor Comparison

  • Table 13 repeats the predictor comparison with all 36 degradation cells represented during fitting while keeping example IDs disjoint.This tests whether relaxing linear constraints or adding a bilinear interaction improves prediction under the ID protocol.Source: §E.1 · p. 29, Table 13 · p. 29

E.2 Random Forest Results

The random forest serves as a flexible nonlinear reference using the same unimodal probes and prior as the parametric predictors. Its in-distribution R2 improvement over linear cue combination is small; the appendix also documents larger-model replications and study resources.

E.2 Random Forest Results

  • The random forest improves mean R2 over linear cue combination by at most 0.015 under the ID protocol.The comparison represents all 36 degradation cells during fitting and covers the four main-paper VLMs.Source: §E.2 · p. 30, Table 14 · p. 30

E.3 Larger-Model Replications

  • The larger-model replications include Qwen2.5-VL-7B, Qwen3.5-9B, and Gemma3-12B, but exclude the 108B-parameter GLM-4.6V checkpoint.Tables 15–18 extend predictor, weight, evidence-ratio, and transfer comparisons; the supplied passages do not give their outcomes.Source: §E.3 · pp. 31–32, Table 15 · p. 31, Table 16 · p. 31

F.1 Computational Resources

  • Experiments ran inference-only on one node with eight NVIDIA L40S GPUs, while predictor fitting ran on CPU.The VLMs were not trained or fine-tuned.Source: §F.1 · p. 33

F.2 Models and Checkpoints

  • The appendix lists checkpoint identifiers and says each VLM uses its native processor and chat template.Scoring conventions, including Gemma3 assistant prefixes, are specified elsewhere in Appendix B.Source: §F.2 · p. 33, Table 19 · p. 33

F.3 Artifact Licensing and Intended Use

  • The benchmark and model artifacts are used solely for non-commercial research, consistent with their intended research use.The named sources include DocVQA, VQAv2, RealworldQA, and the conflicting-text benchmark.Source: §F.3 · p. 33

Related papers

Loading 2610.19407v1…