PaperlayerView exact revision on arXiv

Source-linked AI summary

When Modalities Conflict: How Unimodal Reasoning Uncertainty Governs Preference Dynamics in MLLMs

Zhuoran Zhang, Tengyue Wang, Xilin Gong, Yang Shi, Haotian Wang, Di Wang, Lijie Hu

arXiv:2511.02243v1cs.AI

TL;DR

MLLMs must resolve contradictory visual and textual information, but coarse modality-following statistics overlook differences in unimodal reasoning confidence. This paper decomposes modality following into relative uncertainty and inherent preference, finding a monotonic uncertainty law, a balance-point measure of preference, and layer-wise oscillations near ambiguity.

  • Problem

    Prior modality-following analyses relied on coarse dataset-level statistics that overlook how differences in unimodal uncertainty shape aggregate outcomes.

  • Method

    The paper uses a controllable dataset, entropy-based unimodal uncertainty, and layer-wise prediction probing to analyze modality following through relative uncertainty and inherent preference.

  • Results

    The probability of following a modality decreases monotonically as its relative uncertainty increases, while ambiguous cases near the balance point exhibit layer-wise oscillations between modalities.

  • Takeaways & Limitations

    Relative uncertainty and inherent preference provide a quantitative and mechanistic account of how MLLMs resolve conflicting modalities.

  • Takeaways & Limitations

    Macro-level following ratios alone cannot capture the deeper interplay between case-specific uncertainty and inherent modality preference.

Abstract

from arXiv · show

Multimodal large language models (MLLMs) must resolve conflicts when different modalities provide contradictory information, a process we term modality following. Prior work measured this behavior only with coarse dataset-level statistics, overlooking the influence of model's confidence in unimodal reasoning. In this paper, we introduce a new framework that decomposes modality following into two fundamental factors: relative reasoning uncertainty (the case-specific confidence gap between unimodal predictions) and inherent modality preference( a model's stable bias when uncertainties are balanced). To validate this framework, we construct a controllable dataset that systematically varies the reasoning difficulty of visual and textual inputs. Using entropy as a fine-grained uncertainty metric, we uncover a universal law: the probability of following a modality decreases monotonically as its relative uncertainty increases. At the relative difficulty level where the model tends to follow both modalities with comparable probability what we call the balance point, a practical indicator of the model's inherent preference. Unlike traditional macro-level ratios, this measure offers a more principled and less confounded way to characterize modality bias, disentangling it from unimodal capabilities and dataset artifacts. Further, by probing layer-wise predictions, we reveal the internal mechanism of oscillation: in ambiguous regions near the balance point, models vacillate between modalities across layers, explaining externally observed indecision. Together, these findings establish relative uncertainty and inherent preference as the two governing principles of modality following, offering both a quantitative framework and mechanistic insight into how MLLMs resolve conflicting information.

1 INTRODUCTION

The paper reframes modality following as a conflict-resolution process governed by relative unimodal reasoning uncertainty and stable inherent modality preference. Controlled difficulty variation and entropy-based analysis show that following shifts predictably with uncertainty, while layer-wise oscillations explain hesitation near the balance point.

  • 1 INTRODUCTION: Traditional dataset-level following ratios overlook confidence differences between unimodal predictions and can therefore obscure the factors underlying modality preference.The proposed framework treats aggregate following statistics as emergent from uncertainty and preference.
  • 1 INTRODUCTION: The framework decomposes modality following into case-specific relative reasoning uncertainty and stable inherent modality preference.The preference is assessed when the modalities have comparable reasoning uncertainty.
  • 1 INTRODUCTION: A controllable dataset and entropy-based analysis show that a modality’s following probability monotonically decreases as its relative reasoning uncertainty increases.Relative uncertainty is derived from model-perceived uncertainty under unimodal inputs.
  • 1 INTRODUCTION: The balance point identifies the uncertainty threshold at which a model’s inherent preference determines which modality it follows.A vision-preferring model may require text to be substantially easier before following it.
  • 1 INTRODUCTION: Near the balance point, layer-wise top predictions oscillate between modalities, producing externally observed hesitation or averaged choices.In clearer regions, models commit quickly and stably to the less uncertain modality in early layers.

2 DEFINING CONFLICTING INPUTS AND QUANTIFYING MODALITY FOLLOWING

The paper formalizes conflicting inputs as image-text-question triplets whose unimodal predictions disagree, then classifies the multimodal output by which prediction it follows. Traditional text- and vision-following ratios provide aggregate behavior statistics but do not capture its underlying uncertainty dynamics.

  • 2 DEFINING CONFLICTING INPUTS AND QUANTIFYING MODALITY FOLLOWING: A conflicting input is an image-text-question triplet whose vision-only and text-only predictions disagree.The multimodal model must resolve the contradictory cues to produce an output.
  • 2 DEFINING CONFLICTING INPUTS AND QUANTIFYING MODALITY FOLLOWING: Multimodal outputs are classified as vision-following, text-following, or other according to their relation to the two unimodal predictions.This classification supports aggregate modality-following measurement.
  • 2 DEFINING CONFLICTING INPUTS AND QUANTIFYING MODALITY FOLLOWING: Text-following and vision-following ratios summarize aggregate behavior, but they cannot capture the deeper interplay between case-specific uncertainty and inherent modality preference.The paper uses these macro-level statistics as a baseline for later deconstruction.

3 PREPARING FOR THE ANALYSIS: A CONTROLLABLE DATASET AND AN UNCERTAINTY METRIC

The paper prepares its analysis with a dataset that independently controls visual and textual difficulty and uses output entropy to quantify unimodal uncertainty. Entropy rises with designed difficulty, supporting its use as a comparable uncertainty proxy across modalities.

  • 3.1 CONSTRUCTING A DATASET WITH CONTROLLABLE DIFFICULTY: The study constructs a toy dataset with independently controllable visual and textual difficulty tiers for systematic analysis.Visual difficulty can vary through perceptual manipulation, while textual difficulty varies through reasoning complexity.
  • 3.1 CONSTRUCTING A DATASET WITH CONTROLLABLE DIFFICULTY: Figure 1 organizes the framework into difficulty control, unimodal entropy measurement, and relative-uncertainty analysis of modality choice.The framework uses dv and dt for difficulty, Hv and Ht for entropy, and ΔHrel for the confidence gap.
  • 3.2 QUANTIFYING PERCEIVED UNCERTAINTY WITH ENTROPY: Output entropy measures model-perceived uncertainty over answer-token distributions, with lower values indicating sharper and more confident predictions.Because both modalities produce answers in the same token space, entropy provides a unified comparison between vision-only and text-only reasoning.
  • 3.2 QUANTIFYING PERCEIVED UNCERTAINTY WITH ENTROPY: Entropy consistently increases across higher visual and textual difficulty tiers, supporting its validity as a proxy for perceived unimodal uncertainty.The trend is reported across models and tiers, with visual entropy increasing steadily in the LLaVA-v1.6-7B example.
  • 3.2 QUANTIFYING PERCEIVED UNCERTAINTY WITH ENTROPY: The preparation concludes that controlled difficulty and output-token entropy provide the dataset and uncertainty foundation for the subsequent modality-following analysis.The dataset enables independent difficulty control, while entropy is characterized as a robust and reliable uncertainty proxy.

4 MODALITY FOLLOWING IS SHAPED BY RELATIVE UNCERTAINTY

Modality following is governed by case-specific relative unimodal uncertainty and a model’s inherent preference, rather than fixed dataset-level ratios. Across models and conditions, following probability decreases monotonically with a modality’s relative uncertainty, while the balance point measures stable modality bias.

  • A Finer Lens: Relative Unimodal Uncertainty: Relative unimodal uncertainty measures the normalized entropy difference between text-only and vision-only reasoning, exposing each case’s confidence gap.Negative values indicate greater text confidence; positive values indicate greater vision confidence.
  • A Unified Monotonic Law: Across six models, the probability of following text decreases smoothly and monotonically as text becomes more uncertain relative to vision.This pattern holds across architectures and scales, indicating that modality following changes dynamically with relative reasoning difficulty.
  • Quantifying Inherent Preference via the Balance Point: The balance point is the relative-uncertainty value where text-following probability reaches 50%, quantifying inherent modality preference independently of momentary difficulty.Negative balance points indicate vision preference, whereas positive values indicate text preference.
  • Reconciling Macro-Level Contradictions: Macro-level following ratios can conflate inherent preference with capability and dataset difficulty, as shown by divergent Qwen and LLaVA behaviors.Qwen models exhibit negative balance points, while LLaVA models are neutral or text-leaning; Qwen2-VL’s higher vision-following ratio is partly a dataset artifact.
  • Robustness and Generality: The monotonic relationship remains stable across high- and low-entropy subsets, with only minor shifts in balance points.In high-entropy cases, the balance point moves closer to the center.

5 THE INTERNAL MECHANISM: OSCILLATION IN THE FACE OF AMBIGUITY

The paper explains external hesitation as layer-wise oscillation between conflicting modality-supported answers, especially when their perceived difficulties are similar. Clear cases produce early commitment, while ambiguous cases sustain uncertainty and vacillation.

  • 5 THE INTERNAL MECHANISM: OSCILLATION IN THE FACE OF AMBIGUITY: Layer-wise top-1 predictions are tracked with a LogitLens-style method, and an oscillation counts a switch between vision-supported and text-supported answers.Switches count even when irrelevant-token predictions intervene between the two primary concepts.
  • 5 THE INTERNAL MECHANISM: OSCILLATION IN THE FACE OF AMBIGUITY: A case enters the ambiguous region when relative uncertainty lies within 0.5 of the model’s balance point; otherwise it is classified as clear.The single-image case study varies text difficulty to place examples in distinct regions of the relative-uncertainty spectrum.
  • 5 THE INTERNAL MECHANISM: OSCILLATION IN THE FACE OF AMBIGUITY: Ambiguous conflicts produce more layer-wise oscillations than clear conflicts, indicating that similar perceived difficulty drives modality vacillation.For LLaVA-1.6-7B, ambiguous-region oscillations average 1.43 versus 0.71 in the clear region.
  • 5 THE INTERNAL MECHANISM: OSCILLATION IN THE FACE OF AMBIGUITY: An irrelevant-conflict control has very few oscillations, supporting semantic contradiction rather than sentence structure as the source of internal struggle.The irrelevant-conflict group averages 0.35 oscillations for LLaVA-1.5-7B.
  • 5 THE INTERNAL MECHANISM: OSCILLATION IN THE FACE OF AMBIGUITY: Logit-difference heatmaps show near-zero evidence across many layers in ambiguous regions but rapid red-or-blue commitment in clear regions.The plotted difference is text-answer logits minus vision-answer logits, across model layers and relative-uncertainty levels.

6 RELATED WORK

Related work has documented varied modality preferences and proposed external or internal explanations for conflict resolution. This paper addresses the lack of a consistent account by framing modality following through relative uncertainty and inherent preference.

  • 6 RELATED WORK: Prior benchmarks report inconsistent modality preferences, including text “blind faith,” with variation across models and scenarios.Related work also studies inconsistency detection through benchmarks such as MMIR.
  • 6 RELATED WORK: The paper moves beyond coarse dataset-level observations by explaining varied outcomes through case-specific relative uncertainty and stable inherent modality preference.This framework is presented as a unifying account of apparently contradictory macro-level findings.
  • 6 RELATED WORK: Existing explanations examine input order, instructional prompts, and inconsistencies in learned knowledge representations as potential influences on modality preference.These studies span external steering factors and internal representational factors.

7 CONCLUSION

The paper concludes that modality following is jointly shaped by relative reasoning uncertainty and inherent modality preference. Its datasets and analyses support a monotonic following law, a balance-point measure of preference, and oscillatory internal dynamics near ambiguity.

  • 7 CONCLUSION: Following likelihood monotonically decreases as a modality’s relative uncertainty grows, while the balance point provides a measure of inherent modality preference.The conclusion presents this pattern as robust across models and datasets.
  • 7 CONCLUSION: Layer-wise probing shows strong oscillations between modalities near the balance point, directly explaining externally observed hesitation.The framework is intended to disentangle model capability from modality preference.
  • 7 CONCLUSION: The study uses Color Recognition and Attribution Recognition datasets to investigate multimodal conflict resolution across image and text information.The attribution dataset includes shape and material recognition tasks.

B.1 DATASET OVERVIEW

The study uses controllable color, shape, and material conflict datasets in which image-derived and text-derived answers intentionally disagree. Their groups vary image and text difficulty while preserving repeated conflict structures and balanced answer distributions.

  • B.1 DATASET OVERVIEW: The Color Recognition Dataset contains 400 groups, each with 14 images and questions covering three conflict-description types.Image difficulty levels range from 0–13, and text descriptions average 22.7 words.
  • B.1 DATASET OVERVIEW: Each conflict group preserves a common image answer and a different common text answer, allowing modality-specific information to be systematically contrasted.The dataset design supports controlled comparison of visual and textual answers within matched groups.
  • B.1 DATASET OVERVIEW: Color answers are distributed across six labels for both image and text conditions, with near-balanced counts in each modality.The listed counts range from 66 to 67 per color category.
  • B.1 DATASET OVERVIEW: The Attribution Recognition Dataset provides separate Shape and Material subsets, each containing 300 groups with four images and three conflict-description types.Images are 480×320 pixels, and text descriptions average 30.0 words.
  • B.1 DATASET OVERVIEW: Shape and material answer distributions are specified separately for image and text conditions, with deliberately differing counts across categories.Shape counts are listed for Sphere, Cube, and Cylinder; material counts are listed for Metal and Rubber.

B.2 IMAGE GENERATION OF COLOR RECOGNITION DATASET

The Color Recognition Dataset generates progressively harder visual inputs by varying target size, distractor count, and occlusion across fourteen difficulty tiers.

  • Fourteen visual difficulty tiers vary target size, distractor count, and occlusion to create progressively harder images.The tiers span dv = 0 to 13, with parameters specified in Table 1.

B.3 IMAGE SELECTION OF ATTRIBUTION RECOGNITION DATASET

The Attribution Recognition Dataset selects controlled CLEVR images and constructs increasingly difficult conflicting text descriptions while preserving image–text task correspondence.

  • CLEVR images are selected for unique attribute–color pairs and assigned difficulty levels according to scene complexity.A fixed number of images is sampled at each level to form task groups.
  • Textual conflicts increase in difficulty by adding reasoning steps and converting explicit reasoning into implicit reasoning.Each problem combines a conflict description, question, and command, with descriptions defined using target objects, attributes, and spatial relations.
  • Questions use fixed target shapes and identical text answers that contradict the corresponding image information.The standard question asks for an attribute of the target shape and requests a one-word answer.
  • Paraphrasing varies sentence form while preserving core semantics, reasoning structure, and key information tokens.The dataset also defines text-irrelevant and image-irrelevant controls by replacing target shapes in selected parts of the descriptions.

C CURVE OF ALL REMAIN DATASETS

The dataset examples illustrate direct and multi-step image–text conflicts, while Figure 10 reports text-following behavior across multiple datasets and models.

  • Figures 7–9 present sampled conflicting image–text pairings from the Color Recognition and Attribution Recognition datasets.Red-highlighted text marks descriptions and answers that conflict with image information.
  • The examples include direct, indirect, and indirect-simple conflicts whose vision-based and text-based answers disagree.The examples cover color or material questions with contradictory answers derived from images and text.
  • Figure 10 shows TFR decreasing monotonically as relative uncertainty increases across datasets and models.
Loading 2511.02243v1…