PaperlayerView exact revision on arXiv

Source-linked AI summary

Tri-PvP: Exposing Modality Bias in Omni-Modal Large Language Models through Perceptual-Propositional Evidence Conflicts

Yen-Ting Piao, Shu-Yun Chen, Chin-Hui Chu, Chun-Wei Chen, Shih-Yun Shan Kuan, Hung-yi Lee, Yun-Nung Chen

arXiv:2609.06011v1cs.CLcs.AIcs.CVcs.SD

TL;DR

Existing benchmarks conflate modality bias with evidence-form bias, limiting clean attribution of preferences to either source. TRI-PVP evaluates tri-modal conflicts while varying evidence form in vision and audio. It finds visual bias and an asymmetric evidence-form preference, with bias decodable in early representations and only partially mitigated by contrastive decoding.

  • Problem

    Existing benchmarks do not control perceptual versus propositional evidence across modalities, confounding measurements of modality bias with evidence-form bias.

    Source: §1 · pp. 1–2

  • Method

    TRI-PVP evaluates five OLLMs on 8,000 tri-modal conflicts, varying perceptual and propositional evidence in vision and audio while text remains propositional.

    Source: §1 · p. 2, §4 · p. 3

  • Results

    Visual bias dominates across most settings, with stronger perceptual preference in vision but propositional preference in audio; bias is linearly decodable from early representations and contrastive decoding only partially reduces it.

    Source: §1 · p. 2, §7 · p. 9

  • Takeaways & Limitations

    The findings support evidence-form-controlled evaluations and motivate more fundamental mitigation strategies and representation-level causal analysis.

    Source: §7 · p. 9

    The benchmark focuses on everyday-scale conflicts and bias-neutral questions, leaving broader epistemic scales and non-neutral instructions for future work.

    Source: Limitations · p. 9

Abstract

from arXiv · show

Omni-modal large language models (OLLMs) jointly process vision, audio, and text, yet their modality bias under cross-modal conflict remains underexplored. Existing benchmarks conflate two distinct forms of evidence within a single modality: perceptual signals (e.g., a photograph or recording of a dog) and propositional signals (e.g., the declarative claim "this is a dog"), such that any measured modality bias is inherently confounded with evidence-form bias, precluding clean attribution to either source. To address this, we introduce Tri-PvP, an 8,000-sample tri-modal conflict benchmark crossing vision, audio, and text, where vision and audio each take perceptual or propositional form. Evaluating five OLLMs, we find robust visual bias across most models and evidence-type conditions. Crucially, we reveal a systematic asymmetry in evidence-form bias: models exhibit a stronger bias toward perceptual signal in vision but propositional in audio. Further analyses via layer-wise linear probing and contrastive decoding reveal that modality bias is already linearly decodable from early representation layers and can only be partially mitigated, calling for mitigation strategies beyond surface-level interventions.

1 Introduction

Existing tri-modal benchmarks confound modality bias with evidence-form bias because the forms of evidence are not matched across modalities. TRI-PVP controls evidence form across vision and audio and finds visual bias alongside an asymmetric preference for perceptual vision evidence and propositional audio evidence.

1 Introduction

  • Existing benchmarks confound modality bias with evidence-form bias because evidence forms are not matched across vision, audio, and text.A visual preference could reflect models' preference for perceptual evidence, which the visual channel disproportionately supplies.Source: §1 · pp. 1–2
  • TRI-PVP evaluates five OLLMs on 8,000 tri-modal conflicts, varying perceptual and propositional evidence in vision and audio while keeping text propositional.The benchmark spans animal, emotion, environment, and music domains.Source: §1 · p. 2
  • Across most models and conditions, visual bias dominates, and changing vision or audio from perceptual to propositional evidence reduces but does not reverse that preference.Models generally favor perceptual evidence in vision but propositional evidence in audio.Source: §1 · p. 2
  • The study also uses layer-wise probing to find bias linearly decodable from intermediate hidden states and adapts contrastive decoding to partially reduce image bias.The intervention preserves general omni-modal competence but leaves residual text bias.Source: §1 · p. 2

2 Related Work

Prior work measures modality bias through cross-modal conflicts, but existing tri-modal benchmarks do not fully test controlled three-way conflicts or disentangle evidence form within each modality.

2 Related Work

  • Modality bias is a model's tendency to over-rely on one input stream when modalities provide unequal or conflicting evidence.Such outputs may ignore or contradict other modalities.Source: §2 · p. 2
  • Vision–text and audio–text benchmarks report language overriding visual evidence and strong text bias when audio and text disagree.Wu et al. (2025) study vision–text conflicts, while Wang et al. (2025) study audio–text conflicts.Source: §2 · p. 2
  • CMM studies tri-modal hallucinations and MMA-BENCH studies audio–visual misalignment, but neither tests full three-way conflicts with distinct competing labels or separates evidence form.TRI-PVP addresses the evidence-form gap using a perceptual/propositional axis.Source: §2 · p. 2

3 Preliminary: Perceptual and Propositional Evidence

The paper distinguishes perceptual evidence, acquired through direct sensory experience, from propositional evidence, conveyed as declarative content. It applies this distinction to vision and audio while treating text as propositional by nature.

3 Preliminary: Perceptual and Propositional Evidence

  • Perceptual evidence comes from direct sensory experience, while propositional evidence is declarative content that can be true or false.The distinction draws on an epistemological framework separating perception from testimony and language.Source: §3 · pp. 2–3
  • In vision, perceptual evidence is a natural image and propositional evidence is a written statement rendered as an image; audio uses real-world recordings and synthesized spoken statements, respectively.Text is propositional by nature because it conveys declarative statements rather than direct sensory experience.Source: §3 · p. 3
  • Controlling evidence form provides an axis for testing evidence-form preferences and separating them from modality effects in existing benchmarks.The study focuses on everyday scenarios, where each modality can supply a distinct, unambiguous signal for controlled three-way conflicts.Source: §3 · p. 3

4 TRI-PVP: A Tri-Modal Conflict Benchmark for Perceptual vs. Propositional Evidence

TRI-PVP measures modality bias under tri-modal conflict while independently varying whether image and audio evidence is perceptual or propositional. Its matched evidence conditions and response-classification protocol define a controlled comparison across 8,000 samples.

4 TRI-PvP: A Tri-Modal Conflict Benchmark for Perceptual vs. Propositional Evidence

  • TRI-PVP presents conflicting vision, audio, and text signals with image and audio evidence independently varied between perceptual and propositional forms.A model response is classified to identify which modality it relied on.Source: §4 · p. 3

4.1 Dataset Construction

  • Four matched evidence conditions reuse the same label triples; 500 triples per domain yield 8,000 samples, with text always propositional.The three modality labels are mutually distinct, and label–modality combinations are balanced.Source: §4.1 · pp. 3–4
  • Perceptual assets are filtered for distinctness, single subjects, and clarity; propositional inputs come from templates rendered as image text or TTS audio.Questions avoid modality-specific wording and use an open-ended format rather than predefined choices.Source: §4.1 · p. 4, Figure 2 · p. 4

4.2 Evaluation Protocol

  • The eight-category taxonomy distinguishes single- and dual-modality bias, NO_BIAS, and HALLUCINATION according to how responses relate to the conflicting labels.NO_BIAS includes acknowledging the conflict or reporting all three contents without reaching a conclusion.Source: §4.2 · p. 4
  • Because no modality is more reliable by design, NO_BIAS is the appropriate behavior in this benchmark.The design holds signal quality, source reliability, and task requirements equal across modalities.Source: §4.2 · p. 5
  • An LLM judge classifies each response, with author verification on 800 samples achieving 97.1% agreement.The 800 samples were uniformly stratified by model, domain, evidence-type condition, and bias type.Source: §4.2 · p. 5

5 Benchmarking Modality Bias

Across five OLLMs and four evidence-type conditions, visual bias dominates, while evidence form shifts vision and audio bias in opposite directions. Fully propositional inputs produce the highest NO_BIAS rate for each model.

5.1 Setting

  • The five models were tested in four evidence-type conditions with image-audio-text-question input order; Table 1 reports bias rates by prompt position.Source: §5.1 · p. 5, Table 1 · p. 6

5.2 Main Results

  • BIAS_IMAGE is dominant in 18 of 20 model-condition bars, while BIAS_AUDIO is the weakest single-modality bias in 18 of 20 bars.The exceptions to image dominance are Qwen2.5’s PropI conditions, where BIAS_TEXT or NO_BIAS dominates; BIAS_IMAGE often exceeds 60%, while BIAS_AUDIO is typically below 10%.Source: §5.2 · pp. 5–6
  • BIAS_IMAGE on Qwen2.5 is 49.7% under PercI-PropA but 12.7% under PropI-PropA; Gemma4 BIAS_AUDIO is 0.9% under PercI-PercA and 25.4% under PercI-PropA.The authors hypothesize that this asymmetry reflects vision encoders optimized for perceptual image content and audio encoders often initialized with ASR-style objectives.Source: §5.2 · p. 6
  • NO_BIAS peaks under PropI-PropA for all five models: 60.1% for Qwen2.5, 13.6% for MiniCPM, 11.6% for Gemini3, 7.2% for Gemma4, and 1.5% for Qwen3.The paper reports weaker and less distinct modality-preference signals under strictly propositional conditions.Source: §5.2 · p. 6

6 A Closer Look at Modality Bias

The closer analysis examines input-order effects and representations in Gemma4 and Qwen2.5, then tests contrastive decoding on Gemma4.

6.1 Impact of Input Modality Ordering

  • Qwen2.5 consistently favored the modality placed immediately before the query, while Gemma4 showed different positional patterns across modalities.The reported modality-bias distributions average across all six input orders and evidence-type conditions.Source: §6.1 · p. 6, Table 2 · p. 7

6.2 Probing Modality Bias in Representations

  • ≈80% peak image-probe accuracy for Qwen2.5 under both PercI conditions corresponded to ≈50% output-level image bias.Image-involving probes ranked among the top probes across both models and evidence conditions; Gemma4's audio probe remained weakest under PercA conditions.Source: Figure 4 · p. 7, §6.2 · p. 7
  • Image- and text-bias signals were readily linearly recoverable, whereas audio-bias decodability depended more on model and evidence condition.The analysis uses hidden states at the last prompt-token position and logistic-regression probes, with 50% balanced accuracy as chance.Source: §6.2 · pp. 7–8
  • Most probes exceeded chance within the first two layers, with Gemma4 strengthening bias signals later and Qwen2.5 reaching near-final accuracy early.Gemma4's audio probe under its two PercA conditions showed only sporadic above-baseline spikes.Source: §6.2 · p. 8

6.3 Toward Mitigating Modality Bias

  • 74.9% → 38.8% (PropI-PropA) and 74.4% → 39.6% (PropI-PercA): contrastive decoding reduced image bias most in propositional-image conditions.Unbiased responses increased in all four conditions, with the largest gain in PropI-PropA (7.2% → 38.6%).Source: Figure 5 · p. 8, §6.3 · pp. 8–9
  • Contrastive decoding also raised textual bias, while audio remained the least-biased category; Gemma4 retained 37.4% OMNIBENCH accuracy versus 38.4% unmodified.The additional benchmark comparison indicates no large degradation in general omni-modal competence.Source: §6.3 · p. 9

7 Conclusion

TRI-PVP controls perceptual and propositional evidence across vision and audio to separate modality bias from evidence-form bias. Its evaluations find visual bias across most settings, an evidence-form asymmetry, and only partial mitigation with contrastive decoding.

7 Conclusion

  • The cited passage introduces TRI-PVP as a tri-modal conflict, but provides no further supported details about its design, sample count, or experimental findings in the supplied excerpt.Source: §7 · p. 9
  • Across most settings, models privilege perceptual image evidence but propositional audio evidence, and modality-bias information is linearly decodable from early layers.Contrastive decoding partially reduces image bias but produces residual text bias.Source: §7 · p. 9
  • The findings support evidence-form-controlled evaluation and call for more fundamental mitigation strategies and representation-level causal analysis.Source: §7 · p. 9

Limitations

The benchmark is limited to everyday-scale conflicts and bias-neutral questions, leaving behavior on broader epistemic tasks and under non-neutral instructions open. The appendix specifies reproducibility settings, computational costs, and dataset-template construction details.

Limitations

  • The benchmark covers everyday-scale conflicts; behavior on complex diagrams or scientific data and under non-neutral instructions remains unexamined.The latter includes prompts requesting all sources, requiring a single answer, or providing source-reliability cues.Source: Limitations · p. 9

B Hyperparameters and Computational Information

  • Greedy decoding is the default, with model-recommended settings for Qwen3 and Gemma4; prompts use no custom system instruction and seeds are fixed.Qwen3 uses temperature=0.6, top_p=0.95, top_k=20; Gemma4 uses temperature=1.0, top_p=0.95, top_k=64.Source: B Hyperparameters and Computational… · p. 12
  • Full benchmark runs take approximately 40 minutes for Qwen3 and 1.5 hours per other model; contrastive decoding takes approximately 24 hours.Linear-probe extraction and training take approximately 30 minutes per model; contrastive decoding lacks vLLM generation acceleration in this implementation.Source: B Hyperparameters and Computational… · p. 12

C.1 Dataset Construction Details

  • Dataset questions and propositional sentences use manually authored template pools, with rephrased question variants generated by GPT-5.4 and manually verified.TTS utterances use four voices distributed approximately evenly across samples within each domain.Source: §C.1 · p. 12

C.2 Dataset Sources

TRI-PVP draws perceptual images and audio from source datasets across four domains, while its release retains source-specific licensing conditions. Its intended use is non-commercial research and education, and its generated propositional data and metadata are released under CC BY-NC 4.0.

C.2 Dataset Sources

  • TRI-PVP covers animal, emotion, environment, and music, with source datasets supplying perceptual images and audio across these domains.The paper identifies Table 3 as the source-dataset inventory for perceptual images and audio.Source: §C.2 · p. 12, Table 3 · p. 13
  • Source files retain their original terms; ImageNet and ESC-50 make non-commercial use the binding condition for the combined benchmark.The benchmark compilation and generated data are released under CC BY-NC 4.0, which does not replace source-file licenses.Source: §C.2 · p. 12
  • TRI-PVP is intended solely for non-commercial research and education, not commercial applications or products.Perceptual data retain their original access conditions, while generated propositional data and metadata use CC BY-NC 4.0.Source: §C.2 · pp. 12–13
  • Perceptual data come from publicly licensed academic datasets, while generated text, synthesized speech, and word-card images contain no personal information.The dataset covers general categories and contains no hate speech or offensive content; the authors state that no additional anonymization was required.Source: §C.2 · p. 13

D Fine-grained Analysis of NO_BIAS

This section analyzes NO_BIAS responses using a three-dimension rubric, with manual verification of the LLM judge’s predictions. It decomposes responses into conflict acknowledgment, multi-reporting, and refusal, examining how their behavioral composition varies across models and evidence-type conditions.

D Fine-grained Analysis of NO_BIAS

  • Each NO_BIAS response was independently assessed for conflict acknowledgment, reporting all three modalities without commitment, and refusal; these indicators can co-occur.The analysis pooled domains within each model-condition pair across all four evidence-type conditions.Source: D Fine-grained Analysis of NO_BIAS · p. 13
  • NO_BIAS responses varied by model: Gemini3 showed high conflict acknowledgment and multi-reporting with near-zero refusal, while Gemma4 shifted from refusal to multi-reporting across evidence types.Qwen2.5 had high refusal across conditions, with multi-reporting reaching 71.55% under PropI-PropA; MiniCPM showed moderate conflict acknowledgment and multi-reporting with low refusal. Qwen3’s NO_BIAS rate was at most 1.5%, too sparse for reliable interpretation.Source: D Fine-grained Analysis of NO_BIAS · p. 13, Table 4 · p. 14

E.1 Main Evaluation Judge Prompt

  • The main evaluation judge labels a response NO_BIAS when it acknowledges source conflict or reports all three modalities without committing, and also assigns NO_BIAS to refusals.Source: §E.1 · pp. 14–15

E.2 NO_BIAS Analysis Judge Prompt

  • The NO_BIAS analysis judge further characterizes responses along three independent dimensions, allowing multiple behaviors or none to apply.Source: §E.2 · p. 16
  • The NO_BIAS judge scores conflict, multi_report, and refusal independently, distinguishing recognition of disagreement, reporting all three modalities, and uncertainty or clarification-seeking.A positive score on one dimension does not automatically imply a positive score on another.Source: §E.2 · p. 16

F Manual Verification of the LLM Judge

  • 97.1% of 800 manually inspected responses received author labels aligned with the LLM judge’s predictions.The sample was uniformly stratified by model, evidence-type condition, domain, and judge-assigned bias label; verification was conducted by the authors, with no external annotators or crowdworkers.Source: F Manual Verification of the LLM Judge · p. 17

G Detailed Statistics for the Main Experiment and Input Modality Ordering Effect

The supplementary statistics document condition-level bias distributions and input-ordering effects, while the probing details specify how modality labels are decoded from layer representations. Contrastive decoding shifts some Gemma4 responses toward joint image–text commitments, and a case study illustrates a response that acknowledges conflicting inputs.

G Detailed Statistics for the Main Experiment and Input Modality Ordering Effect

  • Tables 5 and 6 detail bias distributions across six input orders, while Tables 7–9 report Image →Audio →Text results and 95% confidence intervals.These tables supply per-condition statistics underlying Figure 3.Source: G Detailed Statistics for the Main… · p. 17

H Training Details for Linear Probing

  • The probes use last-prompt-token hidden states from each layer and train per-modality binary targets, excluding hallucination-labeled samples.A target is positive when the judge-assigned bias label involves that modality; the input representation is extracted in a single forward pass without generation.Source: H Training Details for Linear Probing · p. 17
  • Each layer has a logistic-regression classifier per modality, applied to standardized hidden states with sigmoid outputs and a 0.5 decision threshold.Training uses balanced class weights and a stratified 80%/20% train/test split.Source: H Training Details for Linear Probing · p. 17

I.2 Case Study

  • In the Gemma4 case study, contrastive decoding changes a response from bias toward the propositional image to explicit recognition of the cross-modal conflict.The baseline answer is piano, classified as BIAS_IMAGE; the post-intervention response meets the paper’s NO_BIAS criterion.Source: §I.2 · p. 19

I.1 Detailed Bias-Type Distribution

  • Under PropI-PercA, BIAS_IMAGE_TEXT rises from 1.4% to 13.6% with contrastive decoding, alongside residual text bias.The report notes that some responses shift to joint image–text commitment rather than audio; the table covers baseline and contrastive decoding across evidence-type conditions for Gemma4.Source: Table 10 · p. 19, §I.1 · p. 19

I.3 General-Capability Evaluation on OMNIBENCH

On OMNIBENCH, contrastive decoding (CD) preserves Gemma 4’s overall performance within a small margin but redistributes errors unevenly across task categories. It tends to help inference-oriented tasks while hurting tasks relying more on perceptual grounding.

I.3 General-Capability Evaluation on OMNIBENCH

  • The evaluation compares baseline decoding with CD on 1,142 aligned image-audio multiple-choice items, using α = 0.5 and adaptive plausibility β = 0.1.Answers are scored against dataset labels with OmniBench’s official multiple-choice parser.Source: §I.3 · p. 19
  • CD scores 427/1142 correct (37.4%) versus the baseline’s 439/1142 (38.4%), a net change of −1.1 percentage points.Table 11 reports multiple-choice accuracy, counting invalid parses as incorrect.Source: §I.3 · p. 19, Table 11 · p. 20
  • CD corrects 141 baseline errors while introducing 153 new errors, preserving overall benchmark performance within a small margin and redistributing errors across task types rather than uniformly lowering performance.Source: §I.3 · p. 20
  • CD tends to help plot inference and story description, while hurting object identification and contextual or environmental questions.By audio type, declines are small for speech and sound events and absent for music.Source: §I.3 · p. 20

Related papers

Loading 2609.06011v1…