PaperlayerView exact revision on arXiv

Source-linked AI summary

UniME-V2: MLLM-as-a-Judge for Universal Multimodal Embedding Learning

Tiancheng Gu, Kaicheng Yang, Kaichen Zhang, Xiang An, Ziyong Feng, Yueyi Zhang, Weidong Cai, Jiankang Deng, Lidong Bing

arXiv:2510.13515v3cs.CVcs.AI

TL;DR

Existing multimodal embedding methods struggle with diverse hard negatives and fine-grained discrimination among false and hard negatives. UniME-V2 uses global retrieval and MLLM semantic judgments to mine negatives and align embeddings with soft semantic labels, achieving state-of-the-art average performance across evaluated tasks.

  • Problem

    Existing in-batch negative mining often lacks negative-sample diversity and struggles to distinguish false negatives from hard negatives or capture their semantic differences.

  • Method

    UniME-V2 uses global retrieval, MLLM-based semantic matching scores, soft-label distribution alignment, and a pairwise-listwise reranker trained on mined hard negatives.

  • Results

    UniME-V2 achieves state-of-the-art performance on average across the MMEB benchmark and multiple retrieval tasks.

  • Takeaways & Limitations

    MLLM semantic judgments provide soft supervision for distinguishing candidates and support the mining of diverse, high-quality hard negatives.

Abstract

from arXiv · show

Universal multimodal embedding models are foundational to various tasks. Existing approaches typically employ in-batch negative mining by measuring the similarity of query-candidate pairs. However, these methods often struggle to capture subtle semantic differences among candidates and lack diversity in negative samples. Moreover, the embeddings exhibit limited discriminative ability in distinguishing false and hard negatives. In this paper, we leverage the advanced understanding capabilities of MLLMs to enhance representation learning and present a novel Universal Multimodal Embedding (UniME-V2) model. Our approach first constructs a potential hard negative set through global retrieval. We then introduce the MLLM-as-a-Judge mechanism, which utilizes MLLMs to assess the semantic alignment of query-candidate pairs and generate soft semantic matching scores. These scores serve as a foundation for hard negative mining, mitigating the impact of false negatives and enabling the identification of diverse, high-quality hard negatives. Furthermore, the semantic matching scores are used as soft labels to mitigate the rigid one-to-one mapping constraint. By aligning the similarity matrix with the soft semantic matching score matrix, the model learns semantic distinctions among candidates, significantly enhancing its discriminative capacity. To further improve performance, we propose UniME-V2-Reranker, a reranking model trained on our mined hard negatives through a joint pairwise and listwise optimization approach. We conduct comprehensive experiments on the MMEB benchmark and multiple retrieval tasks, demonstrating that our method achieves state-of-the-art performance on average across all tasks.

Introduction

Multimodal embedding methods face limitations in negative-sample diversity and semantic discrimination. UniME-V2 uses MLLMs to mine higher-quality negatives and supervise semantic distinctions with soft matching scores.

  • Introduction: Prior approaches commonly rely on in-batch negative mining, motivating questions about overcoming batch constraints and learning differences among negative samples.The introduction identifies limited negative diversity and insufficient semantic discrimination as central concerns.
  • Introduction: UniME-V2 constructs potential hard negatives through global retrieval, then uses MLLM semantic judgments to identify diverse, high-quality negatives.The judgments assess query-candidate semantic alignment and reduce interference from false negatives.
  • Introduction: Soft semantic matching scores supervise learning semantic differences among candidates while relaxing the rigid one-to-one mapping constraint.The model aligns its similarity matrix with the semantic score matrix.
  • Introduction: UniME-V2 achieves state-of-the-art performance on average across the MMEB benchmark and varied retrieval tasks.The evaluated tasks include short- and long-caption retrieval and compositional retrieval.
  • Introduction: The paper also proposes UniME-V2-Reranker, trained on mined hard negatives with joint pairwise and listwise optimization.This reranking model is presented as an additional performance-enhancement component.

Related Work

MLLM-based multimodal representation learning builds on earlier multimodal encoders but addresses limitations in text handling, cross-modal fusion, and language understanding. The paper situates its hard-negative mining pipeline within this progression.

  • Related Work: MLLMs extend traditional LLMs by processing and integrating information across multiple modalities.LLaVA is described as an early contribution using a visual encoder, language model, and projection layer for multimodal alignment.
  • Related Work: CLIP supports image-text retrieval through large-scale cross-modal contrastive learning but has a 77-token text limit and limited cross-modal fusion.These constraints particularly affect detailed descriptions and instruction-sensitive tasks.
  • Related Work: CLIP’s simplistic language modeling can produce bag-of-words representations, motivating later efforts to use MLLMs for multimodal representation learning.The related-work discussion presents MLLMs as a response to these representation limitations.
  • Related Work: The MLLM-as-a-Judge pipeline uses global retrieval to form a potential hard-negative set before scoring query-candidate semantic alignment.This pipeline is introduced as part of the paper’s approach to hard-negative mining.

Methodology

UniME-V2 combines global retrieval, MLLM-generated semantic matching scores, and distribution alignment to mine diverse hard negatives and learn distinctions among candidates. UniME-V2-Reranker then combines pairwise and listwise training to rerank initial embedding-based retrieval results.

  • UniME-V2: UniME-V2 first uses global retrieval to construct a potential hard-negative set, addressing the limited diversity of in-batch mining.VLM2Vec embeddings retrieve candidate negatives globally before semantic filtering.
  • Hard-negative mining: MLLM-as-a-Judge scores query-candidate semantic alignment, excluding likely false negatives and selecting diverse, high-quality hard negatives.Candidates above the threshold are excluded, while cyclical sampling promotes diversity and ensures at least ten selections when needed.
  • UniME-V2: UniME-V2 uses soft semantic matching scores as supervision and aligns similarity distributions with them using symmetric JS-divergence.This replaces a rigid one-to-one target with softer distinctions among candidates.
  • UniME-V2-Reranker: UniME-V2-Reranker trains on positives and mined negatives with joint pairwise and listwise objectives, combining the losses as L = Lpair + Llist.Pairwise training predicts YES for positives and NO for hardest negatives, while listwise training predicts the ground-truth candidate position.
  • Inference: During inference, UniME-V2 retrieves the top 10 candidates by cosine similarity before UniME-V2-Reranker selects the best candidate from the list.The reranker evaluates each candidate against the query and outputs its corresponding position.

Experiments and Results

The implementation trains UniME-V2 with MLLM-generated scores and multimodal backbones using memory-efficient distributed fine-tuning on high-resolution image inputs.

  • Implementation: UniME-V2 uses Qwen2-VL-7B for candidate-set construction, Qwen2.5VL-7B for semantic scores, and Qwen2-VL plus LLaVA-OneVision for training.LoRA with rank 16 and DeepSpeed ZeRO stage-2 reduce GPU-memory requirements during training.

Datasets and Evaluation

The experiments train and evaluate across diverse MMEB tasks, covering both in-distribution and out-of-distribution retrieval settings.

  • Training data: Training uses 20 in-distribution MMEB datasets spanning classification, visual question answering, multimodal retrieval, and visual grounding.The corpus contains 662k curated training pairs with unimodal and multimodal inputs.
  • Evaluation: Evaluation covers 20 in-distribution and 16 out-of-distribution MMEB test sets, reporting Precision for each dataset.The benchmark assesses multimodal embedding capabilities across diverse retrieval tasks.

Main Results

UniME-V2 improves universal multimodal retrieval and hard-negative discrimination across foundation models and retrieval settings, while UniME-V2-Reranker further strengthens reranking performance.

  • 66.7 on out-of-distribution datasets exceeds all prior approaches, while UniME-V2 improves over VLM2Vec by 3.5% and 2.2% on Qwen2-VL-2B and 7B.
  • 2.2%-9.7% higher image-to-text retrieval performance than UniME is achieved on Flickr30K and MS-COCO, with significant gains also reported for long-caption tasks.
  • 5.3%, 6.0%, and 4.5% gains over UniME on SugarCrepe rise to 9.0%, 9.2%, and 9.2% after scaling from 2B to 7B.
  • 0.5%, 0.4%, 0.3%, and 7.4% gains over LamRA are achieved by UniME-V2-Reranker across four tasks with Qwen2-VL-7B retrieval.
  • UniME-V2-Reranker is consistently superior to LamRA across four downstream tasks while using half the data with Qwen2-VL-2B retrieval.

Analysis

Analysis shows that MLLM-based hard-negative mining and judgment-based training improve retrieval, while judge quality and negative-set size materially affect performance.

  • 1.5%, 5.5%, 7.6%, and 3.7% improvements over direct contrastive learning occur on MMEB, short-, long-, and composed-retrieval tasks.
  • Qwen2.5-VL-7B produces higher-quality semantic matching scores than InternVL3-8B, reaching 63.6 versus 58.5 on MMEB.
  • +2.3% on MMEB, +2.9% on short retrieval, +2.4% on long retrieval, and +1.7% on composed retrieval result when hard negatives increase from 4 to 8.
  • Increasing hard negatives to 10 introduces easier negatives, diminishing discriminative learning and slightly reducing performance.
  • Qualitative results show UniME-V2 retrieves query-relevant candidates, while UniME-V2-Reranker refines their ordering, such as prioritizing brown bear over black bear.

Conclusion

UniME-V2 uses MLLM-based semantic judgments to mine diverse hard negatives and align embeddings with soft semantic labels. Experiments on MMEB and varied retrieval tasks report state-of-the-art average performance across tasks.

  • UniME-V2 aligns embedding similarities with MLLM-generated soft semantic matching scores to learn finer-grained distinctions among candidates.The scores also relax the rigid one-to-one matching constraint.
  • The method constructs globally retrieved potential hard negatives, then uses MLLM-as-a-Judge to reduce false-negative interference and identify diverse, high-quality negatives.
  • UniME-V2-Reranker applies joint pairwise and listwise optimization to mined hard negatives for further reranking improvement.
  • State-of-the-art average performance is reported across the MMEB benchmark and various retrieval tasks.

Detail Experiment Setting

The experiments specify training configurations, reranking prompts, evaluation protocols, benchmark summaries, and a temperature ablation. Qualitative mining examples illustrate the processed hard-negative candidates.

  • Reranker configuration: UniME-V2-Reranker uses Qwen2.5-VL-7B with LoRA rank 128, one epoch, approximately 2,000 steps, and a 4,096-token maximum sequence length.
  • Reranker prompts: Pairwise reranker training asks whether each candidate meets the query requirements, whereas listwise training selects the best candidate by its number.
  • Evaluation protocol: Retrieval evaluations cover short-caption, long-caption, and compositional image-text tasks, primarily reporting Recall@1 with “Represent the image/text” prompts.The evaluation benchmarks summarize test-query and candidate counts.
  • Qualitative analysis: Qualitative examples present queries and corresponding hard-negative candidates processed after the hard-negative mining pipeline.

External Results

External experiments examine temperature sensitivity and compare UniME-V2 with established multimodal embedding baselines on MMEB. The comparison distinguishes Qwen2-VL-7B and LLaVA-OneVision-7B backbones.

  • Temperature ablation: A temperature of 0.02 yields optimal performance across MMEB, short and long retrieval, and compositional retrieval metrics.
  • MMEB comparison: UniME-V2 uses Qwen2-VL-7B, while UniME-V2† uses LLaVA-OneVision-7B as its backbone.

Further Analysis

Further analysis visualizes semantic matching scores and qualitative retrieval outcomes. The examples show graded candidate relevance and additional refinement from UniME-V2-Reranker.

  • Semantic matching scores: Target candidates receive nearly 1.0 matching scores, partially relevant candidates receive intermediate scores, and irrelevant candidates receive near-zero scores.
  • Retrieval and reranking: UniME-V2 retrieves query-matched candidates, while UniME-V2-Reranker further refines results by selecting the optimally matched final candidate.
  • Benchmark coverage: The MMEB evaluation includes 20 in-distribution and 16 out-of-distribution datasets, with out-of-distribution datasets marked separately.
Loading 2510.13515v3…