PaperlayerView exact revision on arXiv

Source-linked AI summary

Breaking the Modality Barrier: Universal Embedding Learning with Multimodal LLMs

Tiancheng Gu, Kaicheng Yang, Ziyong Feng, Xingjun Wang, Yanzhao Zhang, Dingkun Long, Yingda Chen, Weidong Cai, Jiankang Deng

arXiv:2504.17432v4cs.CV

TL;DR

CLIP’s limitations and the underexplored use of MLLMs for transferable multimodal representations motivate UniME. UniME combines textual discriminative knowledge distillation with false-negative filtering and hard-negative instruction tuning, achieving consistent improvements across benchmark and retrieval tasks.

  • Problem

    CLIP is constrained by text truncation, isolated image-text encoding, and deficient compositionality, while MLLMs’ potential for transferable multimodal representation learning remains underexplored.

  • Method

    UniME uses textual discriminative knowledge distillation followed by hard negative enhanced instruction tuning with false-negative filtering and multiple hard negatives per batch.

  • Results

    UniME achieves consistent performance improvement across MMEB, short- and long-caption retrieval, and compositional retrieval tasks.

  • Takeaways & Limitations

    UniME exhibits robust discriminative power and superior compositional understanding across the evaluated downstream tasks.

Abstract

from arXiv · show

The Contrastive Language-Image Pre-training (CLIP) framework has become a widely used approach for multimodal representation learning, particularly in image-text retrieval and clustering. However, its efficacy is constrained by three key limitations: (1) text token truncation, (2) isolated image-text encoding, and (3) deficient compositionality due to bag-of-words behavior. While recent Multimodal Large Language Models (MLLMs) have demonstrated significant advances in generalized vision-language understanding, their potential for learning transferable multimodal representations remains underexplored.In this work, we present UniME (Universal Multimodal Embedding), a novel two-stage framework that leverages MLLMs to learn discriminative representations for diverse downstream tasks. In the first stage, we perform textual discriminative knowledge distillation from a powerful LLM-based teacher model to enhance the embedding capability of the MLLMś language component. In the second stage, we introduce hard negative enhanced instruction tuning to further advance discriminative representation learning. Specifically, we initially mitigate false negative contamination and then sample multiple hard negatives per instance within each batch, forcing the model to focus on challenging samples. This approach not only improves discriminative power but also enhances instruction-following ability in downstream tasks. We conduct extensive experiments on the MMEB benchmark and multiple retrieval tasks, including short and long caption retrieval and compositional retrieval. Results demonstrate that UniME achieves consistent performance improvement across all tasks, exhibiting superior discriminative and compositional capabilities.

1 Introduction

UniME addresses limitations in CLIP and underexplored multimodal representation learning by using MLLMs to learn transferable representations. Its two-stage design combines textual distillation with hard-negative instruction tuning and improves discriminative and compositional performance across evaluated tasks.

  • CLIP is limited by short text inputs, isolated image-text encoding, and weak compositional representations.These constraints hinder detailed contextual processing, cross-modal interaction, and compositional retrieval.
  • MLLMs have advanced generalized vision-language understanding, but their use for transferable multimodal representation learning remains underexplored.
  • UniME is a two-stage framework that enables MLLMs to learn universal representations for diverse downstream tasks.The stages are textual discriminative knowledge distillation and hard negative enhanced instruction tuning.
  • Textual discriminative knowledge distillation uses a powerful LLM-based teacher to enhance the embedding capability of the MLLM’s language component.
  • Hard negative enhanced instruction tuning filters false negatives and samples multiple challenging negatives per batch, improving discriminative representations and downstream instruction following.
  • UniME achieves performance improvements across MMEB, short- and long-caption retrieval, and compositional retrieval tasks.The reported evaluations indicate robust discriminative power and superior compositional understanding.

2 Related work

Related work develops MLLMs and multimodal embedding methods to overcome limitations of dual-tower approaches. These efforts broaden input flexibility and cross-modal processing, but hard-negative discrimination remains challenging.

  • Multimodal Large Language Models: MLLMs combine vision encoders, projection layers, and LLM backbones to process and integrate cross-modal information.
  • LLM-based Representation Learning: Decoder-only LLMs have been adapted for representation learning through prompt-based methods and scaling across model sizes.
  • Multimodal Embedding Learning: CLIP’s 77-token limit, disjoint dual encoders, and bag-of-words behavior constrain fine-grained alignment, cross-modal fusion, and compositionality.
  • Multimodal Embedding Learning: Recent methods address these constraints with lightweight dual encoders and contrastive training for varied image-text combinations, high-resolution images, and long text.
  • Multimodal Embedding Learning: Despite these improvements, current methods still face challenges in effectively discriminating hard negative samples during retrieval.

3 Method

UniME extracts unified multimodal embeddings with MLLMs and trains them through textual distillation followed by hard-negative instruction tuning. The method filters false negatives, emphasizes challenging candidates, and supports diverse retrieval inputs and tasks.

  • 3.1 Preliminary Definitions and Feature Extraction: UniME ranks candidates by similarity between query and candidate embeddings extracted by an MLLM from unimodal or interleaved multimodal inputs.
  • 3.1 Preliminary Definitions and Feature Extraction: The MLLM’s unified vision tower, projection layer, and LLM backbone support flexible processing of text, images, and interleaved image-text inputs.
  • 3.2 Textual Discriminative Knowledge Distillation: Textual discriminative knowledge distillation transfers embedding knowledge from NV-Embed V2 by minimizing KL divergence between teacher and student sample distributions.
  • 3.2 Textual Discriminative Knowledge Distillation: Distilling relationships among batch samples improves efficiency over direct contrastive learning under identical data and training conditions.
  • 3.3 Hard Negative Enhanced Instruction Tuning: Hard negative enhanced instruction tuning addresses limited visual sensitivity, image-text misalignment, and generic prompts after the first stage.
  • 3.3 Hard Negative Enhanced Instruction Tuning: The method filters candidate negatives whose query similarity exceeds a threshold, then selects top-k challenging candidates by similarity while excluding positives.

4 Experiments

UniME is evaluated across MMEB and multiple retrieval settings, showing consistent gains over existing multimodal embedding baselines. Its improvements extend to short-caption, long-caption, and compositional retrieval.

  • Evaluation Setup: UniME is evaluated on in-distribution and out-of-distribution MMEB test sets, alongside short-caption, long-caption, and compositional retrieval tasks.The evaluation uses three multimodal large language model settings and reports Precision for retrieval benchmarks.
  • MMEB Results: 4.2% average improvement over E5-V is achieved with Phi3.5-V, while LLaVA-1.6 yields a 4.1% average enhancement.The comparison uses identical training data and configuration settings.
  • Baseline Comparison: 1.3% and 10.3% performance improvements over VLM2Vec are obtained with Phi3.5-V and LLaVA-1.6, respectively.These gains are reported for the proposed model across the two foundation-model settings.
  • Long-Caption Retrieval: 14.8% and 18.1% improvements over EVA-CLIP(8B) are reported for long-caption retrieval on Urban1K.The paper attributes this advantage partly to EVA-CLIP(8B)'s 77-token text restriction.
  • Compositional Retrieval: 3.9%, 4.2%, and 9.1% improvements over VLM2Vec are achieved on SugarCrepe relation replacement, object swapping, and attribute addition tasks.UniME also improves over EVA-CLIP(8B) by 4.2%, 0.6%, and 6.6% on the same three task types.

5 Analysis

The analysis shows that hard negatives provide stronger training signals, while both training stages and threshold selection materially affect performance. Semantic predictions also become more concrete and distributed after training.

  • Hard-Negative Analysis: Hard negatives sustain higher training losses and substantially larger pre-clip gradient norms than easy or random negatives.Easy-negative loss rapidly approaches zero, whereas hard-negative gradients differ by orders of magnitude.
  • Training-Stage Ablation: 15%, 19.5%, 24.9%, and 19.9% gains follow textual discriminative knowledge distillation on MMEB, short-caption, long-caption, and compositional retrieval.Using only hard negative enhanced instruction tuning produces gains of 38.5%, 17.3%, 21.3%, and 14.0% on the same tasks.
  • Training-Stage Ablation: Combining both training stages achieves optimal performance across all evaluated downstream tasks.The second stage also markedly improves MMEB performance, partly through complex instruction-following capabilities.
  • Threshold Ablation: Filtering decreases from 81.7% to 22.9% as β increases from -0.1 to 0.1, improving performance by reducing hard negatives removed as false negatives.At β=0.3, only 13.1% of samples are filtered and performance slightly declines because false negatives persist.
  • Hard-Negative Count: Setting k=8 outperforms k=4, whereas larger k values introduce easy negatives and gradually reduce performance.The number of hard samples sampled for each batch instance is controlled by k.
  • Semantic Visualization: After distillation, predicted tokens shift from abstract descriptions toward concrete semantics; after hard-negative tuning, probabilities spread across multiple image-aligned tokens.Examples include “cow,” “waterfront,” and “house,” replacing predictions such as “Pastoral” and “Peaceful.”

6 Conclusion

UniME is introduced as a two-stage framework for learning discriminative multimodal representations across downstream tasks. The reported experiments show consistent improvements and stronger discriminative and compositional capabilities.

  • Conclusion: UniME combines textual discriminative knowledge distillation with hard negative enhanced instruction tuning for multimodal representation learning.The second stage filters false negatives and samples multiple hard negatives per instance.
  • Conclusion: The framework improves discriminative power and instruction-following ability across MMEB, short- and long-caption retrieval, and compositional retrieval.The conclusion reports consistent performance improvement across all evaluated tasks.

A.1 Training Details

Training uses offline teacher embeddings and QLoRA on 8×A100 GPUs, reducing training duration for the two backbone configurations.

  • Stage 1: Textual Discriminative Knowledge Distillation: Stage 1 trains on text-only data using offline teacher embeddings, QLoRA, and 8×A100 GPUs.Training completes in one hour for Phi3.5-V and two hours for LLaVA-1.6.

A.2 Retrieval Task Evaluation Benchmarks

UniME is evaluated on short-caption, long-caption, and compositional image-text retrieval benchmarks using their standard protocols, with Recall@1 as the primary metric.

  • The evaluation covers short-caption, long-caption, and compositional image-text retrieval tasks.
  • Retrieval experiments follow each benchmark’s standard evaluation protocol.
  • Recall@1 is the primary reported metric for retrieval tasks.
  • The benchmark summary specifies the numbers of test queries and test candidates per query.

B External Results

External-results analysis compares LoRA ranks across UniME’s two training stages on MMEB benchmarks, distinguishing in-domain and out-of-domain evaluation.

  • Table 8 reports performance across in-domain and out-of-domain settings.
  • LoRA rank 32 delivers optimal performance in Stage 1, while rank 16 is most effective in Stage 2.
  • The comparison evaluates different LoRA ranks on MMEB benchmarks using the Phi-3.5V backbone.

B.2 Specific results on the MMEB

The MMEB analysis compares UniME with baseline models and examines how UniME’s training stages change next-token prediction behavior.

  • The reported UniME, E5-V, and VLM2Vec results use their best-performing variants with the LLaVA-1.6 backbone.
  • The staged comparison distinguishes textual discrimination distillation alone from combining it with hard-negative enhanced instruction tuning.
  • Figure 8 presents examples of negative samples encountered during training.
  • Figure 7 compares top-k subsequent-token prediction probabilities across UniME training stages.

C.2 Schematic illustration of Negative Data

The negative-data analysis presents training examples and reports that false negatives occur more often in textual than visual data, primarily because of synonyms.

  • False negatives are more prevalent in textual data than in visual data.
  • The reported primary reason for textual false negatives is the presence of synonyms.
  • Figure 8 provides examples of negative samples encountered during training.
Loading 2504.17432v4…