PaperlayerView exact revision on arXiv

Source-linked AI summary

Chaos in the Text: Revealing the Modality Preference in Mixed-Modality Retrievers

Yubo Sun, Chunyi Peng, Yukun Yan, Zhenghao Liu, Zhipeng Xu, Sen Mei, Linlin Xin, Zheni Zeng, Maosong Sun

arXiv:2610.11816v1cs.IR

TL;DR

How reliably retrievers handle corpora mixing text, image, and fused text-image documents has not been systematically quantified. Trident treats each document’s text, image, and fused views as co-equal positives in a jointly optimized objective; it improves mixed-modality retrieval and reduces sensitivity to modality composition.

  • Problem

    How reliably retrievers handle corpora mixing text, image, and fused text-image documents has not been systematically quantified, despite strong results on single-modality corpora.

    Source: §2 · p. 2

  • Method

    Trident treats each document’s text, image, and fused text-image representations as co-equal positives and jointly optimizes relevance discrimination and positive-view balance.

    Source: §1 · p. 2

  • Results

    Trident improves mixed-modality retrieval across CLIP- and VLM-based architectures, reduces modality-composition sensitivity, and increases average single-modality performance.

    Source: §4.2 · p. 8, §6 · p. 10

  • Takeaways & Limitations

    Text’s scoring advantage can let irrelevant text outrank relevant images, disrupting relevance-based ranking in mixed-modality retrieval.

    Source: §3.2 · p. 6

Abstract

from arXiv · show

Dense retrievers have made significant progress on text and image corpora, but whether these capabilities extend reliably to mixed corpora containing text, image, and fused text-image documents remains unclear. In this paper, we systematically examine retrievers across architectures and find that their performance is highly sensitive to modality composition. As image documents are progressively replaced with semantically corresponding text representations, retrieval performance follows a pronounced V-shaped curve, remaining strong on single-modality corpora but degrading substantially when modalities coexist. In particular, irrelevant text causes more severe degradation than an equal number of irrelevant images, a phenomenon we term Chaos in the Text. Further analysis reveals modality preference, whereby text representations receive systematically higher similarity scores, allowing irrelevant text to outrank relevant images. To mitigate this bias, we introduce Trident, which constructs text, image, and fused text-image views of each document as co-equal positives and jointly optimizes relevance discrimination and positive-view balance through Multi-Positive View InfoNCE. Experiments across visual document and natural image benchmarks show that trident improves mixed-modality retrieval on both CLIP-based and VLM-based architectures, reduces sensitivity to modality composition and text distractors, and increases average single-modality retrieval performance.

1 INTRODUCTION

Mixed-modality corpora expose a reliability gap: retrievers that perform well on text-only or image-only corpora can struggle when modalities coexist. Trident addresses modality preference with balanced positive views, improving retrieval across mixed and single-modality settings.

1 INTRODUCTION

  • Retrievers perform well on purely image or text corpora but degrade when the modalities coexist, with irrelevant text able to push answer-bearing images out of the top-K.The paper calls this failure mode “Chaos in the Text”; Figure 1 depicts an asymmetric V-shaped NDCG@10 curve as images are replaced by text.Source: §1 · p. 1, Figure 1 · p. 2
  • Text representations receive higher similarity scores than image representations even for relevant query–document pairs, giving text a ranking advantage over relevant images.This modality preference can let irrelevant text outrank relevant image documents in a shared candidate pool.Source: §1 · p. 2
  • Trident treats text, image, and fused text-image representations of each document as co-equal positives and jointly optimizes them with Multi-Positive InfoNCE.The objective also discourages positive probability mass from concentrating on one modality, aiming to mitigate modality preference.Source: §1 · p. 2
  • Experiments find that Trident reduces modality preference and mixed-corpus sensitivity while improving average retrieval performance on single-modality corpora across CLIP-based and VLM-based architectures.The reported reductions include sensitivity to modality proportions and text distractors.Source: §1 · p. 2

2 HOW RETRIEVAL MODELS PERFORM ON MIXED-MODALITY CORPORA?

Across two benchmarks, retrieval degrades when text and image documents coexist, and irrelevant text is more damaging than an equal number of irrelevant images. The section evaluates this pattern across modality compositions, models, and benchmarks.

2 HOW RETRIEVAL MODELS PERFORM ON MIXED-MODALITY CORPORA?

  • The study addresses the limited systematic quantification of how reliably multimodal retrievers perform on corpora mixing text, image, and fused documents.Relevant evidence may reside in any of the three modalities.Source: §2 · p. 2

2.1 EXPERIMENTAL SETUP AND MODELS

  • For mixed-modality retrieval, candidates from text, image, and fused sets are ranked jointly, requiring comparable relevance scores; NDCG@10 is the primary metric.The study evaluates state-of-the-art CLIP-based and VLM-based retrieval models under their default settings.Source: §2.1 · p. 3
  • The evaluation varies text, image, and fused-document proportions while preserving document identities and corpus size, using ChartQA and SlideVQA.Each image document has a text description, an image view, and a fused summary-image view; one view is sampled per document according to target proportions.Source: §2.1 · p. 3

2.2 RESULTS AND DISCUSSION

  • Retrieval follows a V-shaped pattern as image documents are replaced by semantically corresponding text: performance falls in mixed corpora, then recovers toward text-only, while fused documents alleviate the drop.The V-shaped behavior appears for both CLIP-based and VLM-based visual retrievers on ChartQA and SlideVQA.Source: Figure 2 · p. 3, §2.2 · p. 4
  • With the same number of irrelevant documents, text distractors cause a sharp retrieval decline while irrelevant images have a relatively modest effect across models and benchmarks.The comparison indicates that candidate-pool expansion alone cannot explain the degradation.Source: §2.2 · p. 4, Figure 3 · p. 4

3 WHY DO RETRIEVAL MODELS EXHIBIT CHAOS IN THE TEXT?

This section tests whether modality itself influences retrieval scores using semantically aligned text, image, and fused corpora. It finds higher scores for text positives and frequent cases where irrelevant text outranks relevant images.

3 WHY DO RETRIEVAL MODELS EXHIBIT CHAOS IN THE TEXT?

  • To investigate the source of mixed-modality retrieval disruption, the section tests whether modality itself systematically influences ranking across parallel corpora.The analysis considers score differences among relevant documents and ranking interference from irrelevant text.Source: §3 · p. 4, §3.2 · p. 5

3.1 EXPERIMENTAL SETUP

  • The analysis compares cosine-similarity distributions for the same queries across semantically aligned text, image, and fused corpora.It reports all-pair means and means for ground-truth relevant pairs to examine modality-related score differences.Source: §3.1 · p. 5

3.2 RESULTS AND DISCUSSION

  • Text positives receive higher mean similarity than image and fused positives, with image showing the largest gap and fused views narrowing but not eliminating it.The pattern holds across CLIP-based and VLM-based retrievers of different scales.Source: §3.2 · p. 5, Figure 4 · p. 5
  • For image positives, irrelevant text hard negatives outrank relevant documents for most queries across all three models.Interference is substantially reduced for text positives and partly alleviated for fused views, though many queries remain affected.Source: Figure 4 · p. 5, §3.2 · p. 6

4 HOW CAN WE MITIGATE MODALITY PREFERENCE?

Trident uses modality-balanced positive views and joint optimization to address modality preference. It improves mixed-modality retrieval across both architecture categories, reduces sensitivity to corpus composition, and narrows modality-specific similarity differences.

4.1 TRIDENT

  • Modality-balanced data construction gives each query text, image, and fused views of a relevant document as positive examples.The text view is a detailed description, the image view is the original image, and the fused view pairs a summary with that image.Source: §4.1 · p. 6
  • Multi-Positive View InfoNCE places all three positive views and in-batch negatives in one shared softmax denominator, jointly constraining scores within and across modalities.Independent modality-normalized objectives can learn within-modality rankings without constraining cross-modality score baselines.Source: §4.1 · p. 7
  • The objective combines relevance discrimination with positive-view balance, preventing one modality from monopolizing positive probability mass.It is intended to increase the competitiveness of underweighted relevant images and lower scores of high-scoring irrelevant text.Source: §4.1 · p. 7

4.2 RESULTS AND DISCUSSION

  • Trident achieves the highest Mix average in both architecture categories, and both variants improve Origin averages over their baselines.The full objective also improves average performance over its ablation, particularly for the VLM backbone.Source: Table 1 · p. 8, §4.2 · p. 8
  • After training, modality-composition performance valleys shrink and surfaces become higher and flatter, with particularly stable VLM performance and moderate CLIP variation.Figure 6 reports mean NDCG@10 on SlideVQA across fused and text ratios before and after training.Source: §4.2 · p. 8, Figure 6 · p. 8
  • Before training, same-modality similarity advantages differ by architecture; afterward, same- and cross-modality distribution centers converge.Figure 7 compares positive-pair cosine similarities before and after training on Jina-CLIP-v2 and Qwen3-VL-E-2B.Source: §4.2 · p. 9, Figure 7 · p. 9

5 RELATED WORK

Prior multimodal retrieval research developed from dual-tower contrastive encoders to unified-embedding VLMs, but common benchmarks largely evaluate single-modality corpora. Related modality-gap work emphasizes embedding geometry, whereas this paper identifies modality preference in mixed-corpus ranking.

5 RELATED WORK

  • Dual-tower encoders align text and images in a shared space using separate modality towers; later VLM backbones process interleaved text and image tokens in one transformer.The passage names CLIP, ALIGN, and SigLIP/SigLIP 2 among dual-tower systems, and VLM2Vec, GME, UniME, and Qwen3-VL-Embedding among VLMs.Source: §5 · p. 9
  • Common benchmarks and adopted document-VQA datasets are almost entirely built on single-modality corpora, so they cannot assess retrieval over mixed-modality corpora.Source: §5 · p. 9
  • Prior modality-gap studies measure geometric separation and cross-modal alignment, while this paper distinguishes ranking-corrupting modality preference in mixed-modality retrieval.The passage notes that a uniform cluster offset can be harmless when all ranking candidates share a modality.Source: §5 · p. 9

6 CONCLUSION

The conclusion identifies modality-composition sensitivity and “Chaos in the Text,” where irrelevant text degrades retrieval in image corpora. It presents Trident’s modality-balanced training and reports improved mixed- and single-modality retrieval.

6 CONCLUSION

  • Irrelevant text in image corpora substantially degrades retrieval, a failure the paper names “Chaos in the Text.”The conclusion also identifies sensitivity to corpus modality composition.Source: §6 · p. 9
  • Trident combines modality-balanced data construction with Multi-Positive View InfoNCE to promote relevance discrimination and positive-view balance.The method addresses systematic modality biases in similarity scores that compromise candidate comparability.Source: §6 · p. 10
  • Experiments on CLIP-based and VLM-based architectures show improved mixed-modality retrieval, reduced sensitivity to modality proportions, and higher average single-modality retrieval performance.Source: §6 · p. 10

A NDCG@10 SURFACES FOR THE REMAINING RETRIEVERS

Across the evaluated models, retrieval performance generally varies with the proportions of text, image, and fused documents. A V-shaped degradation persists even after GR-CLIP correction, which only partially mitigates the sensitivity.

A NDCG@10 SURFACES FOR THE REMAINING RETRIEVERS

  • Across nearly all evaluated models, performance follows an asymmetric V-shaped pattern as corpus composition shifts from image toward text, with a marked decline at intermediate mixtures.SigLIP2-L/16-384 and VisRAG-Ret peak near single-modality boundaries and drop sharply when another modality is introduced.Source: A NDCG@10 SURFACES FOR THE REMAINING… · p. 14
  • GR-CLIP narrows performance fluctuations and reduces degradation, but the V-shaped dependence on corpus modality proportions remains.Its correction requires modality-wise statistics from a large domain-specific reference corpus, limiting transferability and practicality in open-domain or unknown-distribution settings.Source: A NDCG@10 SURFACES FOR THE REMAINING… · p. 14
  • The results indicate that mainstream multimodal retrievers generally lack robustness to changes in text, image, and fused-document proportions, regardless of architecture, scale, training paradigm, or post-hoc correction.Source: A NDCG@10 SURFACES FOR THE REMAINING… · p. 14

B EXTENDED CHAOS-IN-THE-TEXT RESULTS ACROSS RETRIEVERS

Across a broader set of models, text distractors depress retrieval more than equally irrelevant image distractors, while Trident reduces this gap. GR-CLIP also narrows the gap but does not eliminate the asymmetry.

B EXTENDED CHAOS-IN-THE-TEXT RESULTS ACROSS RETRIEVERS

  • Text distractors depress NDCG@10 more than equally irrelevant image distractors across diverse models, and the gap widens as the distractor pool grows.The pattern appears across uncorrected models, GR-CLIP-corrected counterparts, and VLM-based retrievers.Source: B EXTENDED CHAOS-IN-THE-TEXT RESULTS… · p. 14
  • GR-CLIP shrinks the text–image distractor gap, but text distractors still outrank equally irrelevant image distractors across corpus sizes.Source: B EXTENDED CHAOS-IN-THE-TEXT RESULTS… · p. 14
  • Trident-Qwen3VL-2B has a markedly smaller text–image distractor gap than the two baselines on ChartQA and SlideVQA.Its text- and image-distractor curves remain close throughout the tested distractor protocol.Source: B EXTENDED CHAOS-IN-THE-TEXT RESULTS… · pp. 14–15

C ADDITIONAL EVIDENCE OF MODALITY PREFERENCE ACROSS RETRIEVERS

Additional analyses find modality preference across CLIP-based and VLM-based retrievers: text positives tend to receive higher scores, and irrelevant text can outrank relevant images. Trident aligns positive-view scores and reduces ranking interference.

C ADDITIONAL EVIDENCE OF MODALITY PREFERENCE ACROSS RETRIEVERS

  • The analysis examines both score differences among semantically corresponding positive views and ranking interference from irrelevant text, extending coverage to additional CLIP-based and multimodal embedding retrievers.Source: C ADDITIONAL EVIDENCE OF MODALITY… · p. 15
  • GR-CLIP can reduce positive-score gaps for some models, but substantial text-negative interference remains for image positives and varies by backbone and positive modality.The results distinguish score calibration from resolving modality-dependent ranking behavior.Source: C ADDITIONAL EVIDENCE OF MODALITY… · p. 16
  • UniME-Phi3.5 and Jina-V5-small show image-positive interference rates of 52.6% and 53.2%, versus 18.8% and 12.8% for text positives, respectively.Source: C ADDITIONAL EVIDENCE OF MODALITY… · pp. 16–17
  • Figure 10 reports that text distractors degrade NDCG@10 more than equally irrelevant image distractors as distractor counts increase.Source: Figure 10 · p. 17
  • The additional experiments find modality preference across retrievers with different architectures, training objectives, and model scales, distinguishing global score adjustment from learning less modality-dependent relevance rankings.Source: C ADDITIONAL EVIDENCE OF MODALITY… · pp. 17–18
  • Text positives receive higher similarity than corresponding image positives for CLIP-ViT-L/14, SigLIP2-L, and AltCLIP; fused views generally reduce but do not consistently eliminate the gap.Source: C ADDITIONAL EVIDENCE OF MODALITY… · p. 15, Figure 11 · p. 18
  • For CLIP-ViT-L/14 and SigLIP2-L, text hard negatives frequently outrank image positives, with nearly all sampled image-positive queries affected; interference is generally lower for text positives.Source: C ADDITIONAL EVIDENCE OF MODALITY… · p. 16, Figure 11 · p. 18
  • Trident-Qwen3VL-2B scores text, image, and fused positives at 0.674, 0.672, and 0.670, with interference rates of 10.4%, 11.8%, and 11.8%, respectively.The rates correspond to text, image, and fused positives in that order.Source: C ADDITIONAL EVIDENCE OF MODALITY… · p. 17, Figure 12 · p. 19

D ADDITIONAL TRAINING DETAILS

This appendix documents Trident’s training and objective, then evaluates its behavior on natural-image retrieval. Trident records the highest reported Mix averages on both trained backbones without reducing Origin averages.

D ADDITIONAL TRAINING DETAILS

  • Training uses 41k OpenDocVQA samples and one representative CLIP and VLM backbone, trained for one epoch with three positive views per sample.Qwen uses language-tower LoRA with a frozen vision tower; Jina fine-tunes both towers, and fused embeddings average text and image equally.Source: D ADDITIONAL TRAINING DETAILS · pp. 19, 21

E.1 OBJECTIVE DEFINITION

  • For each query, Trident treats the text, image, and fused document representations as three positives alongside negative documents in the candidate set.Multi-Positive View InfoNCE averages the negative log-probabilities of all positive views.Source: §E.1 · p. 21

E.2 RELEVANCE DISCRIMINATION AND POSITIVE-VIEW BALANCE

  • The objective combines positive-set relevance discrimination with a KL-based balance term that favors a uniform probability share across positive views.Relevance discrimination alone can favor an easy-to-match text view; the balance term discourages concentration on only some views.Source: §E.2 · p. 22

E.3 GRADIENTS WITH RESPECT TO POSITIVE AND NEGATIVE SCORES

  • Positive views below 1/M are encouraged upward and those above it downward, while high-scoring irrelevant documents receive stronger penalties.The negative-document update depends on relevance labels and current scores, not on an assumption that text is necessarily negative.Source: §E.3 · pp. 22–23

E.4 COMPLEMENTARY ROLES OF THE TWO COMPONENTS

  • Relevance discrimination raises positive scores in proportion to their current share, while the balance term pushes views toward equal shares.Together, the terms penalize high-scoring irrelevant text and support weaker relevant image views.Source: §E.4 · p. 23

E.5 CONSTRAINING CROSS-MODAL SCORE BASELINES

  • Shared normalization makes modality-specific score offsets affect the shared probability allocation, constraining relative score baselines across modalities.Separately normalized modality objectives cannot identify those relative offsets.Source: §E.5 · pp. 23–24

F MODEL PERFORMANCE ON NATURAL IMAGES

  • 70.50 and 76.78: Trident achieves the highest Mix averages on both backbones, surpassing Jina-CLIP-v2 (56.43) and Qwen3-VL-Embedding-2B (73.67), with no Origin-average decrease.Evaluation used 170K MMEB training samples and four natural-image benchmarks; Mix merged their mixed-modality corpora.Source: Table 4 · p. 24, F MODEL PERFORMANCE ON NATURAL IMAGES · pp. 24–25

G FURTHER ABLATION STUDIES

The ablations separate the contributions of balanced positive sampling and Multi-Positive InfoNCE. They also show that reducing similarity disparity further with an explicit loss does not improve final retrieval.

G FURTHER ABLATION STUDIES

  • σ falls from 0.055 to 0.009 with balanced positive sampling alone and to 0.004 after adding Multi-Positive InfoNCE.The authors report that data-level balance and objective-level coordination contribute complementarily to reducing modality preference.Source: G FURTHER ABLATION STUDIES · p. 25
  • Explicit Modality Balance Loss reduces σ from 0.004 to 0.003, but its final NDCG@10 is slightly lower than the default objective’s.Its early advantage disappears by the end of training.Source: G FURTHER ABLATION STUDIES · p. 25
  • Balancing positive probability mass does not require equal cosine similarities, since text, image, and fused views can differ in information content and granularity.The default objective slightly outperforms explicit similarity equality despite marginally greater dispersion, 0.004 versus 0.003.Source: G FURTHER ABLATION STUDIES · p. 25

H FULL 3D SURFACE COMPARISONS BEFORE AND AFTER TRIDENT

Across additional document benchmarks, Trident removes the pronounced performance valley associated with mixed modality composition. Its mixed-modality advantage persists across embedding dimensions, while residual composition sensitivity differs by backbone.

H FULL 3D SURFACE COMPARISONS BEFORE AND AFTER TRIDENT

  • After training, Qwen3-VL-Embedding-2B is nearly flat across modality ratios, while Jina-CLIP-v2 retains mild sensitivity at extreme ratios.The authors attribute the contrast to Qwen’s unified transformer versus CLIP’s separate modality towers.Source: H FULL 3D SURFACE COMPARISONS BEFORE… · pp. 26–27
  • Before training, four additional benchmark surfaces show a valley near balanced modality mixes; afterward, the valley disappears and performance stays stable across composition.The comparison plots Mean NDCG@10 against fused and text ratios before and after Trident.Source: H FULL 3D SURFACE COMPARISONS BEFORE… · pp. 25–26, Figure 16 · p. 27

I PERFORMANCE ACROSS MATRYOSHKA DIMENSIONS

Loading 2610.11816v1…