Source-linked AI summary
Beyond Text-Dominance: Understanding Modality Preference of Omni-modal Large Language Models
Xinru Yan, Boxi Cao, Yaojie Lu, Hongyu Lin, Weixiang Zhou, Le Sun, Xianpei Han
TL;DR
OLLMs introduce an underexplored modality-preference problem as diverse modalities are integrated into a unified representation space, with implications for understanding and reliability. The paper measures preference with conflicting multimodal inputs, traces its layer-wise emergence, and uses internal preference signals to diagnose hallucinations. Most OLLMs favor visual information, preference emerges in mid-to-late layers, and probes support hallucination diagnosis across three benchmarks without task-specific data.
Problem
Modality preference in unified OLLMs is underexplored, although unequal modality weighting can make models ignore factual signals from other modalities and complicate mechanistic interpretability and application.
Method
The paper combines a conflict-based benchmark and Modality Selection Rate with layer-wise probing of hidden states to study preference and diagnose cross-modal hallucinations.
Results
Most OLLMs exhibit a pronounced visual preference, which emerges progressively in mid-to-late layers; layer-wise probes diagnose hallucinations across three benchmarks without task-specific data.
Takeaways & Limitations
Modality preference provides a mechanistic signal for interpreting OLLM decisions and a practical tool for building more trustworthy systems.
Abstract
from arXiv · showhide
Native Omni-modal Large Language Models (OLLMs) have shifted from pipeline architectures to unified representation spaces. However, this native integration gives rise to a critical yet underexplored phenomenon: modality preference. To bridge this gap, we first systematically quantify modality preference of OLLMs using a newly-curated conflict-based benchmark and the modality selection rate metric. Our evaluation of ten representative OLLMs reveals a notable paradigm shift: unlike the ``text-dominance'' of traditional VLMs, most OLLMs exhibit a pronounced visual preference. To further understand the underlying mechanism, we conduct layer-wise probing and demonstrate that such modality preference is not static but emerges progressively in the mid-to-late layers. Building upon these insights, we leverage these internal signals to diagnose cross-modal hallucinations, achieving competitive performance across three downstream multi-modal benchmarks without task-specific data. Our work provides both a mechanistic understanding and a practical tool for building more trustworthy OLLMs. Our code and related resources are publicly available at: https://github.com/icip-cas/OmniPreference
1 Introduction
This paper studies modality preference in OLLMs, where the internal competition among modalities remains underexplored and may contribute to cross-modal hallucinations. It introduces conflict-based measurement, traces preference emergence across layers, and uses the resulting signals for hallucination diagnosis.
- Motivation: OLLM modality competition is an unresolved problem because unequal modality weighting can cause models to ignore factual signals from other modalities.The paper frames this gap as relevant to both mechanistic interpretability and model application.
- Quantifying Preference: The proposed benchmark presents mutually conflicting text, image, and audio inputs, while Modality Selection Rate measures how often each modality is selected.The framework evaluates ten open-source and proprietary OLLMs.
- Quantifying Preference: Most evaluated OLLMs show a pronounced visual preference, contrasting with the text-dominance commonly reported for traditional VLMs.The models exhibit diverse preference spectra rather than a uniform preference pattern.
- Mechanistic Analysis: Layer-wise probing reveals that modality preference emerges abruptly and stabilizes in the mid-to-late layers rather than forming in shallow layers.The probes use hidden states from each layer to predict the model’s final modality preference.
- Downstream Diagnosis: Layer-wise probes diagnose cross-modal hallucinations without task-specific downstream data across POPE, AVHBench, and AHa-Bench.On POPE, the probes reached 94% average AUROC across three OLLMs, compared with 50% for random guessing and 51% for earlier-layer predictions.
- Contributions: The paper presents modality-preference analysis as both an interpretation of OLLM decision-making and an empirical basis for more trustworthy, hallucination-resistant systems.Its contributions span systematic measurement, layer-wise analysis, and downstream hallucination diagnosis.
2 Related Work
Related work studies modality bias through conflict benchmarks, internal diagnostics, and probing, but has focused mainly on vision-language models. These studies consistently report that VLMs over-rely on text.
- Multimodal Models: Recent multimodal research spans native OLLMs and VLMs that jointly process modalities such as text, image, audio, and video.The literature includes proprietary systems such as GPT-5 and Gemini 3 alongside open-source OLLMs.
- Preference Analysis: Researchers probe modality preference using conflicting image-text benchmarks, causal mediation, gradient-based diagnostics, and attention analysis.These methods investigate behavioral preference and the internal origins of modality bias.
- Preference Analysis: Existing studies consistently find that traditional VLMs over-rely on textual cues, despite using diverse methodologies.The related work notes that this evidence is predominantly limited to the vision-language setting.
- Probing: Linear probing and representational-geometry methods are established tools for decoding learned syntactic, semantic, and other properties from model activations.These approaches provide methodological precedent for analyzing internal representations.
3 Framework Design
The framework creates semantically conflicting multimodal inputs so an OLLM’s selected answer reveals its modality preference, then quantifies that preference with MSR.
- 3.1 Problem Formulation: The tri-modal conflict setting makes text, vision, and audio convey three mutually contradictory answers, forcing the model’s output to align with one modality.Each modality pair is semantically contradictory, so the selected answer exposes the model’s underlying preference.
- 3.1 Problem Formulation: A modality’s preference is defined as the conditional probability that the model’s output aligns with that modality when it conflicts with the others.Higher values indicate stronger preference for the modality.
- 3.2 Data Construction: The dataset combines modality inputs from different semantic categories to construct balanced conflict triplets across the semantic space.The construction uses category-based sampling and enumerates 20 valid category triplets.
- 3.2 Data Construction: The standardized, modality-agnostic question and randomized answer-option order reduce direct modality and position cues during evaluation.Semantic labels are converted into declarative statements before the common question is applied.
- 3.3 Quantitative Metrics: Modality Selection Rate (MSR) is the proportion of conflict samples whose model response matches the candidate option assigned to modality m.For |M| competing modalities, MSR(m) = 1/|M| is the uniform baseline, while higher values indicate preference.
4 Landscape of Modality Preference in OLLMs
Across tri-modal and bi-modal conflicts, evaluated OLLMs show diverse but generally visual preferences, while audio is systematically selected least often. The results indicate that current OLLMs do not yet achieve balanced multimodal integration.
- Evaluation Setup: 10 OLLMs are evaluated using deterministic inference for open-source models and standardized audio preprocessing, with 1,000 constructed samples.Closed-source Gemini models are accessed through the official API with default generation parameters.
- Tri-modal Results: Most OLLMs favor vision in tri-modal conflicts: eight models exceed 50% image MSR, while Ming-Lite-Omni 1.5 and Qwen3-Omni each have 52% text MSR.Gemini 3 Flash reaches an image MSR as high as 82%.
- Bi-modal Results: In text+image and image+audio conflicts, every OLLM has a higher image MSR than the paired modality.This establishes a consistent image preference in both pairwise settings.
- Bi-modal Results: In text+audio conflicts, every OLLM has a higher text MSR than audio MSR.The pairwise results show that the favored modality depends on the input combination, but audio remains lower than its paired modality.
- Cross-setting Pattern: Audio is systematically neglected: tri-modal audio MSR stays below 21% for all OLLMs, reaches 1% for Ming-Lite-Omni 1.5, and remains lower in bi-modal settings.The pattern persists whether audio is paired with image or text.
5 How Modality Preference Emerges Inside OLLMs
Layer-wise probing shows that modality preference is absent in shallow OLLM layers, emerges sharply in the mid-to-late layers, and then stabilizes or declines near the output.
- Emergence dynamics: Probe accuracy remains near chance, approximately 0.30 to 0.55, across the first 30% of layers.These shallow layers primarily encode low-level features rather than modality preference signals.
- Emergence dynamics: Between 40% and 70% depth, all evaluated models show a sharp increase in modality-preference probe accuracy.Qwen2.5-Omni-7B rises from approximately 0.50 to around 0.90, while MiniCPM-o-2.6 rises from around 0.50 to 0.80.
- Emergence dynamics: Qwen2.5-Omni-7B reaches the highest reported peak probe accuracy, approximately 0.90.
- Four-phase decomposition: The emergence process is decomposed into Absent, Emerging, Peak, and Declining phases using accuracy-based thresholds.The Emerging onset uses a median-plus-MAD threshold, Peak begins above 95% of maximum accuracy, and Declining requires a drop exceeding 2% across at least two consecutive layers.
- Model-scale patterns: Qwen3-Omni-30B-A3B-Instruct and Ming-Lite-Omni 1.5 have the earliest onset points, while Qwen2.5-Omni-3B shows a −0.120 Peak-phase decline and Ming-Lite-Omni 1.5 a −0.030 decline.
- Representation-level analysis: SVD projections progress from fully mixed early-layer representations to clustered mid-to-late representations and partial diffusion near the output.For Qwen2.5-Omni-7B, Layer 5 is fully interleaved, Layer 18 begins separating labels, and later projections show increased overlap relative to Layer 24.
6 Diagnosing Preference-Induced Hallucination
The paper uses layer-wise modality-preference probes to diagnose cross-modal hallucinations across diverse benchmarks. Hallucinated samples consistently show elevated preference probability for the interfering modality, and the probes outperform early-layer and random baselines.
- Diagnostic setup: Linear modality-preference probes are evaluated as hallucination detectors across POPE, AVHBench, and AHa-Bench without task-specific downstream data.The benchmarks cover image–text, audio–video, and audio–text hallucination settings.
- Preference and hallucination: Hallucinated samples consistently exhibit an abnormal increase in predicted preference probability for the interfering modality.This pattern appears across all evaluated benchmarks in the probability-density analysis.
- Diagnostic performance: Early Probe performance remains near the Random baseline, with AUROC values around 0.50, indicating that early layers lack the relevant detection signal.
- Diagnostic performance: For AVHBench(V→A) and AVHBench(A→V), the probe AUROC consistently exceeds 0.72 across all models.
- Representative results: MiniCPM-o-2.6 achieves an AUROC of 0.89 and an AUPRC of 0.82 on AVHBench’s Video-driven Audio sub-task.
- Case studies: Representative cases show hallucination-associated shifts from target-modality preference toward interfering-modality preference.On POPE, visual preference falls from 0.81 to 0.21 while text preference rises to 0.76 during hallucination.
7 Conclusion
The paper examines modality preference through behavioral evaluation, mechanistic analysis, and hallucination detection. It finds that OLLMs predominantly favor vision, neglect audio, and develop modality preference progressively in mid-to-late layers.
- The investigation spans behavioral evaluation, mechanistic analysis, and hallucination detection.
- OLLMs predominantly favor vision while systematically neglecting audio.
- Layer-wise probing shows that modality preference emerges progressively in mid-to-late layers.
- Hallucinations correlate with abnormal preference shifts that can be detected using linear probes.