PaperlayerView exact revision on arXiv

Source-linked AI summary

Scaling Language-Centric Omnimodal Representation Learning

Chenghao Xiao, Hou Pong Chan, Hao Zhang, Weiwen Xu, Mahani Aljunied, Yu Rong

arXiv:2510.11693v1cs.CLcs.AIcs.CV

TL;DR

Why MLLM-based embedding approaches outperform traditional contrastive models remains underexplored. This paper proposes LCO-Emb, a language-centric contrastive refinement framework, and reports stronger benchmark performance alongside a positive relationship between generative capability and contrastive refinement.

  • Problem

    Why MLLM-based embedding approaches outperform traditional CLIP-based approaches on challenging tasks remains underexplored.

    Source: §1 · p. 1

  • Method

    LCO-EMB uses language-centric paired data for lightweight contrastive refinement, with LoRA minimally modifying pretrained MLLMs to preserve latent cross-modal alignment.

    Source: §1 · p. 2, §3 · p. 5

  • Results

    Across diverse backbones and benchmarks, LCO-EMB outperforms state-of-the-art multimodal embedding models trained on much larger multimodal sets, while GRSL shows a positive relationship between generative capacity and contrastive refinement.

    Source: §1 · p. 2, §7 · p. 14

  • Takeaways & Limitations

    The findings position generative pretraining as a central driver of scalable, efficient, and robust multimodal representation learning.

    Source: §7 · p. 14

    The study focuses on lightweight contrastive learning with minimally adjusted weights, while joint generative and contrastive training is also noted as an alternative.

    Source: Limitations · p. 14

Abstract

from arXiv · show

Recent multimodal embedding approaches leveraging multimodal large language models (MLLMs) fine-tuned with contrastive learning (CL) have shown promising results, yet the underlying reasons behind their superiority remain underexplored. This work argues that a crucial advantage of MLLM-based approaches stems from implicit cross-modal alignment achieved during generative pretraining, where the language decoder learns to exploit multimodal signals within a shared representation space for generating unimodal outputs. Through analysis of anisotropy and kernel similarity structure, we empirically confirm that latent alignment emerges within MLLM representations, allowing CL to serve as a lightweight refinement stage. Leveraging this insight, we propose a Language-Centric Omnimodal Embedding framework, termed LCO-Emb. Extensive experiments across diverse backbones and benchmarks demonstrate its effectiveness, achieving state-of-the-art performance across modalities. Furthermore, we identify a Generation-Representation Scaling Law (GRSL), showing that the representational capabilities gained through contrastive refinement scales positively with the MLLM's generative capabilities. This suggests that improving generative abilities evolves as an effective paradigm for enhancing representation quality. We provide a theoretical explanation of GRSL, which formally links the MLLM's generative quality to the upper bound on its representation performance, and validate it on a challenging, low-resource visual-document retrieval task, showing that continual generative pretraining before CL can further enhance the potential of a model's embedding capabilities. Codes, models, and resources are available at https://github.com/LCO-Embedding/LCO-Embedding.

1 Introduction

The introduction argues that MLLMs acquire implicit cross-modal alignment during generative pretraining, making contrastive learning a lightweight refinement for LCO-EMB. It also links generative capability to representation potential through a scaling law.

Introduction

  • CLIP-style alignment relies on massive paired-data contrastive learning [37] [50] [83], but scaling plateaus on complex tasks; MLLM-plus-CL approaches improve these tasks [8] [39] [85], though the reason remains underexplored.Prior scaling strategies increase model size, dataset volume, and batch size, yet performance tends to plateau on tasks requiring deeper cross-modal comprehension [9] [25] [50] [83].Source: §1 · p. 1
  • Text-only contrastive fine-tuning improves text and non-text embedding discriminability, supporting implicit cross-modal alignment learned during MLLM generative pretraining.The authors propose that learning to generate textual outputs lets MLLMs use multimodal information in a shared semantic space.Source: §1 · p. 2
  • LCO-EMB uses language-centric paired data for lightweight contrastive refinement of pre-aligned generative embeddings, with LoRA intended to preserve pretrained capabilities and alignment.Unlike computationally intensive CLIP alignment, this approach emphasizes preserving the cross-modal structure established during MLLM pretraining.Source: §1 · p. 2
  • LCO-EMB outperforms state-of-the-art multimodal embedding models using text-only training data, sets a new state of the art on MIEB, and performs competitively on audio and video.Adding minimal paired multimodal data in diverse formats further calibrates the embedding space for downstream tasks.Source: §1 · p. 2
  • The Generation-Representation Scaling Law links stronger pre-contrastive generative capability to greater representational potential, with a PAC-Bayesian bound relating generation quality to an upper bound on representation performance.On the low-resource Southeast Asian visual-document retrieval task SeaDoc, continual OCR-intensive pretraining improves retrieval after the same amount of text-only contrastive learning.Source: §1 · p. 2

2 Latent Cross-Modal Alignment in MLLMs

The section investigates latent cross-modal alignment in MLLMs through anisotropy and similarity-structure analyses, finding that text-only contrastive learning improves isotropy across modalities. The kernel analysis quantifies shared neighborhood structure across paired modalities.

2.1 Analysis of Anisotropy Degrees

  • Text-only contrastive learning made image, audio, and video embeddings more isotropic, indicating latent semantic alignment with text in Qwen2.5-Omni-3B.Before fine-tuning, embeddings showed collapsed structures and poorly separated distributions across modalities.Source: §2.1 · p. 3, Figure 1 · p. 3

2.2 Analysis of Kernel-level Similarity

  • Kernel alignment is measured with mNN, the average proportion of shared top-k neighbors for paired samples, and analyzed layer-wise before and after text-only contrastive learning on Qwen-VL 7B and 3B.Paired samples serve as semantic anchors for comparing neighborhood structures across modalities.Source: §2.2 · p. 4, Figure 2 · p. 4

3 Language-centric Omnimodal Representation Learning

MLLMs develop latent cross-modal alignment during generative pretraining, which lightweight text-only contrastive fine-tuning can unlock to improve representations across modalities. LCO-EMB leverages language-centric data while seeking to preserve that alignment.

3 Language-centric Omnimodal Representation Learning

  • LCO-EMB uses lightweight text-only contrastive fine-tuning to unlock latent cross-modal alignment that emerges in intermediate layers despite suboptimal initial similarity embeddings.The framework leverages language-centric data to improve representation quality across textual and non-textual modalities.Source: §3 · p. 5
  • Text-only LCO-EMB fine-tunes only the language decoder while freezing modality encoders and projector, then re-plugs the decoder; minimal multimodal paired data calibrates multimodal variants.MLLM architectures use a projector to align modality-specific representations with the decoder’s embedding space [2] [40] [59].Source: §3 · p. 5
  • LoRA [27] minimally perturbs the pretrained model, preserving generative capabilities [4] and latent cross-modal alignment, especially in the unaffected language-decoder embedding layer.It adds low-rank trainable parameters to selected layers while the original model remains frozen.Source: §3 · p. 5

4 Experiments

LCO-EMB achieves strong benchmark performance with limited contrastive training data, and text-only variants also outperform leading embedding models. Dataset-specific strengths, model ensembling, and LoRA training further shape its representational performance.

4.1 Experimental Settings

  • Experiments use LLaVA-Next, Qwen2.5-VL, and Qwen2.5-Omni backbones, with all-NLI and Scale-1M text training plus synthetic multimodal pairs, evaluated on MIEB-Lite and audio/video retrieval benchmarks.MIEB-Lite covers 51 tasks; audio-text and video-text evaluations use AudioCaps, Clotho, MSR-VTT, and ActivityNet, with Recall@1 for audio and video tasks.Source: §4.1 · pp. 5–6

4.2 Performance Comparison on the MIEB Benchmark

  • With ∼0.37M training pairs—about 21× less data than GME’s ∼8M—LCO-EMB’s multimodal variants set a new MIEB-Lite state of the art against strong baselines.LCO-EMB also performs competitively on clustering, linear probing, and zero-shot classification, where MLLM-based representations typically lag behind CLIP-style models.Source: Figure 4 · p. 6, Table 1 · p. 7, §4.2 · p. 7

4.3 Representational Capability of LCO-EMB

  • On MIEB-Sub18, text-only LCO-EMB beats E5-V by 21.69 points and Voyage-Multimodal-3 by 13.00 points on average across categories.It also exceeds prior advanced methods by 21.02 points on Linear Probing, 10.26 on Cross-lingual Visual STS, and 15.35 on Multilingual Image Retrieval; it achieves comparable document-understanding results to Voyage-M3 despite using only textual training data.Source: §4.3 · p. 7, Figure 5 · p. 8
  • All-NLI-trained LCO-EMB excels in Visual STS and Document Understanding, Scale-1M leads on Linear Probing and Multilingual Image Retrieval, and their model-soup ensemble achieves the best overall performance.The authors suggest Scale-1M’s semantically rich scene descriptions emulate image-caption pretraining, activating image representations without explicit visual data.Source: Table 2 · p. 8, §4.3 · p. 8
  • Text-only LoRA outperforms CLIP-style training on 800K PixmoCaps pairs while requiring 50× less training time; full fine-tuning remains inferior to LoRA.The authors attribute full fine-tuning’s weaker results to contrastive loss diverging from the pretraining objective, perturbing parameters and potentially disrupting established cross-modal alignment; shallow projection yields only marginal gains over native embeddings.Source: §4.3 · pp. 8–9, Table 3 · p. 9

5 Generation-Representation Scaling Law

The section finds that stronger generative capability predicts better post-contrastive representations and derives a PAC-Bayesian bound linking generative quality to representation performance. SeaDoc experiments show that generative pretraining can improve retrieval, but outcomes depend on training data and resolution.

5.1 The Relationship between Generative and Representational Capabilities

  • Across OCR image-text, video-text, and audio-text tasks, baseline generative performance correlates positively with post-CL representation performance across MLLM backbones.This is the Generation-Representation Scaling Law (GRSL): representational abilities enhanced through contrastive refinement scale positively with original generative capability.Source: §5.1 · p. 9, Figure 6 · p. 10

5.2.1 Definitions

  • Generative quality is defined through mutual information captured by the pretrained prior; for a fixed dataset, lower generative cross-entropy corresponds to higher mutual information and quality.This relationship uses the stated approximation that generative cross-entropy estimates conditional entropy.Source: §5.2.1 · p. 10

5.2.2 Central Hypothesis

  • A stronger generative prior provides a warm start because higher mutual information makes pretrained representations more predictive and positive pairs closer, helping contrastive optimization reach lower empirical loss.The hypothesis further posits that a better prior reduces the inefficiency gap between finite-step fine-tuning and the information-theoretic optimum.Source: §5.2.2 · pp. 10–11

5.2.3 Main Theoretical Result

  • The PAC-Bayesian bound links population contrastive risk to a generative bottleneck, log N − I_P(X; Y), and a complexity penalty; lower prior generative loss tightens the representation bound.The paper argues parameter-efficient tuning such as LoRA keeps KL(Q∥P) small, limiting the cost of moving from the generative prior.Source: §5.2.3 · p. 11

5.3.1 Data Curation

  • SeaDoc evaluates English-query retrieval of visual document pages in low-resource Southeast Asian languages, using 1,001 English queries over 5,055 pages from 29 books in Thai, Vietnamese, Malay, and Lao.Queries are generated to map uniquely to ground-truth pages and then filtered by human annotators.Source: §5.3 · p. 12, §5.3.1 · p. 12

5.3.2 Experimental Settings

  • The SeaDoc study uses Qwen2.5-VL-3B with lightweight contrastive learning, comparing it with a variant receiving additional generative pretraining and low- versus high-resolution settings.The generative stage uses SEA-language OCR data and PixmoCaps captioning data; the resolution settings cap images at 262,144 and 2,621,440 pixels, respectively.Source: §5.3.2 · p. 12

5.3.3 Experimental Results

  • On SeaDoc, OCR-only low-resolution SFT causes capability collapse, higher resolution partly mitigates it, and adding PixmoCaps further boosts post-CL retrieval.The authors attribute the PixmoCaps gain to preserving pretrained image-text alignment for subsequent text-only contrastive fine-tuning.Source: Figure 7 · p. 12, §5.3.3 · p. 13

6 Related Work

Prior work studies omnimodal learning with modality-specific encoders and explores MLLMs for image–text alignment, but leaves the use and theory of MLLMs’ latent generative alignment unexplored. This work positions its language-centric omnimodal approach around a scaling law linking an MLLM’s own generation and representation capabilities.

Omnimodal Representation Learning

  • Omnimodal methods [22] typically train modality-specific encoders on large-scale cross-modal pairs, while recent MLLM work [8] [39] [85] highlights image–text alignment without exploring latent generative alignment for omnimodal learning.The paper identifies the effectiveness and theoretical basis of exploiting this latent alignment as unexplored.Source: §6 · p. 13

Modality-centric Representation Learning

  • ImageBind [22] anchors contrastive learning in images, Web-SSL [18] scales language-free vision learning to CLIP-parity OCR, and E5-V uses text-only learning to generalize to images and composed retrieval.The paper says it extensively studies a language-centric view for training omnimodal representation models.Source: §6 · p. 13

Representation Capabilities

  • Xiao et al. [76] analyzed 50 models across 130 tasks in 39 languages and reported that CLIP’s scaling gains largely plateau on advanced representation benchmarks.Cambrian-1 [63] shows downstream MLLM performance scaling with vision-encoder representation capability, while Yang et al. [80] formalize that direction’s scaling law.Source: §6 · p. 13
  • The paper instead studies the “Generation-Representation Scaling law,” in which an MLLM’s representation abilities scale with its own generation capabilities, aligning with Xiao et al. [75] on LLM embeddings’ strengths in instruction following and reasoning-oriented retrieval.This contrasts with the “Representation-Generation Scaling Law,” where MLLM generation performance scales with modality-specific encoder strength.Source: §6 · p. 13

7 Conclusion

The conclusion attributes MLLM-based embeddings’ performance to implicit cross-modal alignment from generative pretraining and presents LCO-EMB and GRSL as evidence that generative capacity supports effective contrastive refinement.

7 Conclusion

  • LCO-EMB treats contrastive learning as lightweight refinement of implicit cross-modal alignment learned during generative pretraining, enhancing representations while preserving generative structure.The language decoder integrates multimodal information in a unified representation space.Source: §7 · p. 13
  • Theoretical analysis using a PAC-Bayesian generalization bound and empirical validation on diverse, challenging benchmarks support LCO-EMB’s efficacy and GRSL’s generality.GRSL establishes a positive correlation between generative capacity and the effectiveness of contrastive refinement.Source: §7 · p. 14
  • The findings position generative pretraining, rather than merely expanding cross-modal data, as a central driver of scalable, efficient, and robust multimodal representation learning.Source: §7 · p. 14

Limitations

The study leaves joint generative- and contrastive-loss training [43] [46] for future work because of its high computational cost, despite its potential to preserve model knowledge and enhance representational power.

Limitations

  • Joint generative and contrastive training [43] [46] could preserve model knowledge and enhance representational power, but its high computational cost makes it future work for omnimodal representation learning.The study instead uses lightweight contrastive learning with minimal weight adjustments through low-rank adaptation.Source: Limitations · p. 14

A Details of Additional Multimodal Data

The training corpus combines text-only all-NLI data with approximately 94k multimodal samples to align embeddings with downstream tasks while preserving generalization. The multimodal mix balances limited visual-document and image–text data with diverse text and synthetic examples.

A Details of Additional Multimodal Data

B Details of MIEB-Lite Benchmark

MIEB-Lite spans 51 tasks across 8 categories, assessing multimodal embedding capabilities through task-specific metrics. Its categories cover semantic understanding, document and image retrieval, classification, and embedding structure.

B Details of MIEB-Lite Benchmark

  • MIEB-Lite comprises 51 tasks across 8 categories; Visual STS [74] renders text as images and compares image-text embedding similarities with human judgments using Spearman correlation.Visual STS includes English, cross-lingual, and multilingual subcategories.Source: B Details of MIEB-Lite Benchmark · p. 22
  • Document Understanding/Visual Document Retrieval selects 6 Vidore tasks [19] to test retrieval of relevant visual documents and layout-text understanding, using nDCG@5.The tasks also assess interleaved image-text alignment.Source: B Details of MIEB-Lite Benchmark · p. 22
  • Image Linear Probing uses 8 challenging datasets with 16-shot accuracy, while compositionality tests fine-grained image-text alignment on ARO-Benchmark and Winoground [61] using accuracy.The 16-shot protocol follows Xiao et al. [76] and closely preserves ranking compared with full-dataset probing.Source: B Details of MIEB-Lite Benchmark · p. 22
  • Vision-centric QA uses reasoning-as-retrieval [75] to retrieve answers to image-conditioned questions, including spatial, depth, and relative-distance tasks, with accuracy as its metric.The category primarily assesses vision-centric capabilities [63].Source: B Details of MIEB-Lite Benchmark · p. 22
  • Retrieval includes 11 image-only, image-text, and interleaved tasks, plus multilingual WIT [57] and XM3600 [60] coverage across 38 languages, evaluated primarily with nDCG@10.The tasks assess modality-specific, composed encoding, and image–multilingual-text alignment.Source: B Details of MIEB-Lite Benchmark · pp. 22–23
  • Zero-shot classification evaluates 7 fine-grained datasets by prompt-based similarity matching with accuracy; clustering uses 2 tasks, including Imagenet-Dog15, with NMI, where MLLM embeddings typically lag CLIP-style models [76].Zero-shot prompts follow common practices from Radford et al. [50] and Xiao et al. [76].Source: B Details of MIEB-Lite Benchmark · p. 23

C Details of MIEB-Sub18 Benchmark · D Impact of LoRA Hyperparameters for LCO-EMB

The appendices describe the challenging MIEB-Sub18 benchmark and analyze how LoRA rank and α affect LCO-EMB, finding task-dependent effects rather than a global optimum.

C Details of MIEB-Sub18 Benchmark

  • MIEB-Sub18 selects 18 MIEB tasks comprising 47 challenging subtasks, emphasizing visual text representation, multilingual understanding, interleaved encodings, and image linear probing.Its scope includes Visual STS, multilingual image retrieval, seven Vidore document-understanding tasks, and three image linear-probing tasks.Source: C Details of MIEB-Sub18 Benchmark · p. 23
  • Visual STS renders text as images and evaluates semantic similarity with Spearman correlation across English, cross-lingual, and multilingual categories; WIT retrieval spans 11 languages and uses nDCG@10.Visual STS assesses interleaved encoding, while multilingual image retrieval evaluates alignment between image and multilingual text embeddings.Source: C Details of MIEB-Sub18 Benchmark · p. 23
  • The seven Vidore tasks retrieve visual documents using queries and measure nDCG@5; Stanford Cars, BirdSnap, and Country211 use 16-shot linear probing and report accuracy.The linear-probing tasks are included because MLLM embeddings underperform relative to CLIP and self-supervised vision models on them.Source: C Details of MIEB-Sub18 Benchmark · p. 23

D Impact of LoRA Hyperparameters for LCO-EMB

  • LCO-EMB uses LoRA for lightweight contrastive refinement intended to minimally perturb MLLM knowledge and preserve intrinsic cross-modal alignment—the authors’ “learn less, forget less” rationale.The analysis focuses on LoRA rank r and α.Source: D Impact of LoRA Hyperparameters for… · p. 23
  • LoRA rank sets the dimensionality and capacity of trainable weight updates, while α scales their contribution to model weights and thus how much new knowledge is infused.The authors leave a fuller empirical and theoretical analysis of the likely size-specific balance between new knowledge and modification of pretrained weights to future work.Source: D Impact of LoRA Hyperparameters for… · pp. 23–24
  • No global LoRA optimum emerges: appropriately scaled α/r generally favors retrieval, document understanding, and image probing, but larger α degrades compositionality and vision-centric QA; r=256, α=512 causes unrecoverable loss spikes.The pattern varies by task category; V-STS shows minimal variation with LoRA hyperparameters.Source: Table 4 · p. 24, D Impact of LoRA Hyperparameters for… · p. 24

E Details of the data construction process of SeaDoc · F Relationship Between Generative Loss and Conditional Entropy

SeaDoc uses a filtered, human-reviewed query set and a separate OCR-intensive training set for low-resource visual documents. The theoretical section links lower generative loss to higher mutual information, under a strong-prior approximation.

E Details of the data construction process of SeaDoc

  • Gemini-2.5-Flash [12] annotated each PDF page with OCR, an English translation, and an English query answerable only from that page, yielding 5,055 triplets.Source: E Details of the data construction… · p. 24
  • Quality control reduced 4,491 content-page annotations to questions scoring at least 9 for quality and 10 for groundedness, followed by linguist review.Quality measured informativeness and query relevance; groundedness measured whether the answer was exclusive to the page.Source: E Details of the data construction… · p. 24
  • 1,001 high-quality queries were selected from a 5,055-page SeaDoc corpus for retrieval tasks.The quality-control process filtered functional pages, applied quality and groundedness thresholds, and included a final review by in-house linguists.Source: E Details of the data construction… · p. 24
  • The SeaDoc training set uses 4k seed images outside retrieval-test queries to create five SFT tasks per image and an approximately 20k-example set.Tasks cover OCR, query generation, translation, and answering generated queries; the experiments also combine this data with 710k PixmoCap examples to preserve general capability.Source: E Details of the data construction… · p. 24

F Relationship Between Generative Loss and Conditional Entropy

  • The prior’s generative quality is defined as the mutual information IP(X; Y) that its representations capture between input X and target output Y.Source: F Relationship Between Generative Loss… · p. 25
  • Generative cross-entropy loss Lg(P) estimates the unknown conditional entropy H(Y|X), enabling it to be substituted into the mutual-information definition.The mutual information is IP(X; Y) = H(Y) − H(Y|X).Source: F Relationship Between Generative Loss… · p. 25
  • For a strong prior, lower generative loss corresponds to higher mutual information and generative quality because target entropy is fixed for a given dataset.This follows from approximating conditional entropy H(Y|X) with generative loss Lg(P).Source: F Relationship Between Generative Loss… · p. 25
Loading 2510.11693v1…