PaperlayerView exact revision on arXiv

Source-linked AI summary

UniME-V2: MLLM-as-a-Judge for Universal Multimodal Embedding Learning

Tiancheng Gu, Kaicheng Yang, Kaichen Zhang, Xiang An, Ziyong Feng, Yueyi Zhang, Weidong Cai, Jiankang Deng, Lidong Bing

arXiv:2510.13515v3cs.CVcs.AI

TL;DR

Existing multimodal embedding methods struggle with diverse hard negatives and fine-grained discrimination among false and hard negatives. UniME-V2 uses global retrieval and MLLM semantic judgments to mine negatives and align embeddings with soft semantic labels, achieving state-of-the-art average performance across evaluated tasks.

Abstract

from arXiv · show

Universal multimodal embedding models are foundational to various tasks. Existing approaches typically employ in-batch negative mining by measuring the similarity of query-candidate pairs. However, these methods often struggle to capture subtle semantic differences among candidates and lack diversity in negative samples. Moreover, the embeddings exhibit limited discriminative ability in distinguishing false and hard negatives. In this paper, we leverage the advanced understanding capabilities of MLLMs to enhance representation learning and present a novel Universal Multimodal Embedding (UniME-V2) model. Our approach first constructs a potential hard negative set through global retrieval. We then introduce the MLLM-as-a-Judge mechanism, which utilizes MLLMs to assess the semantic alignment of query-candidate pairs and generate soft semantic matching scores. These scores serve as a foundation for hard negative mining, mitigating the impact of false negatives and enabling the identification of diverse, high-quality hard negatives. Furthermore, the semantic matching scores are used as soft labels to mitigate the rigid one-to-one mapping constraint. By aligning the similarity matrix with the soft semantic matching score matrix, the model learns semantic distinctions among candidates, significantly enhancing its discriminative capacity. To further improve performance, we propose UniME-V2-Reranker, a reranking model trained on our mined hard negatives through a joint pairwise and listwise optimization approach. We conduct comprehensive experiments on the MMEB benchmark and multiple retrieval tasks, demonstrating that our method achieves state-of-the-art performance on average across all tasks.

Introduction

Multimodal embedding methods face limitations in negative-sample diversity and semantic discrimination. UniME-V2 uses MLLMs to mine higher-quality negatives and supervise semantic distinctions with soft matching scores.

Introduction

  • Prior approaches commonly rely on in-batch negative mining, motivating questions about overcoming batch constraints and learning differences among negative samples.The introduction identifies limited negative diversity and insufficient semantic discrimination as central concerns.Source: Introduction · pp. 1–2
  • UniME-V2 constructs potential hard negatives through global retrieval, then uses MLLM semantic judgments to identify diverse, high-quality negatives.The judgments assess query-candidate semantic alignment and reduce interference from false negatives.Source: Introduction · p. 2
  • Soft semantic matching scores supervise learning semantic differences among candidates while relaxing the rigid one-to-one mapping constraint.The model aligns its similarity matrix with the semantic score matrix.Source: Figure 1 · p. 1, Introduction · p. 2
  • UniME-V2 achieves state-of-the-art performance on average across the MMEB benchmark and varied retrieval tasks.The evaluated tasks include short- and long-caption retrieval and compositional retrieval.Source: Introduction · p. 2
  • The paper also proposes UniME-V2-Reranker, trained on mined hard negatives with joint pairwise and listwise optimization.This reranking model is presented as an additional performance-enhancement component.Source: Introduction · p. 2

Related Work

MLLM-based multimodal representation learning builds on earlier multimodal encoders but addresses limitations in text handling, cross-modal fusion, and language understanding. The paper situates its hard-negative mining pipeline within this progression.

Related Work

  • MLLMs extend traditional LLMs by processing and integrating information across multiple modalities.LLaVA is described as an early contribution using a visual encoder, language model, and projection layer for multimodal alignment.Source: Multimodal Large Language Models · p. 2
  • CLIP supports image-text retrieval through large-scale cross-modal contrastive learning but has a 77-token text limit and limited cross-modal fusion.These constraints particularly affect detailed descriptions and instruction-sensitive tasks.Source: Multimodal Representation Learning · p. 2
  • CLIP’s simplistic language modeling can produce bag-of-words representations, motivating later efforts to use MLLMs for multimodal representation learning.The related-work discussion presents MLLMs as a response to these representation limitations.Source: Multimodal Representation Learning · pp. 2–3
  • The MLLM-as-a-Judge pipeline uses global retrieval to form a potential hard-negative set before scoring query-candidate semantic alignment.This pipeline is introduced as part of the paper’s approach to hard-negative mining.Source: Figure 2 · p. 3

Methodology

UniME-V2 combines global retrieval, MLLM-generated semantic matching scores, and distribution alignment to mine diverse hard negatives and learn distinctions among candidates. UniME-V2-Reranker then combines pairwise and listwise training to rerank initial embedding-based retrieval results.

UniME-V2

Hard-negative mining

  • MLLM-as-a-Judge scores query-candidate semantic alignment, excluding likely false negatives and selecting diverse, high-quality hard negatives.Candidates above the threshold are excluded, while cyclical sampling promotes diversity and ensures at least ten selections when needed.Source: MLLM-as-a-Judge for Hard Negatives… · pp. 3–4

UniME-V2

  • UniME-V2 uses soft semantic matching scores as supervision and aligns similarity distributions with them using symmetric JS-divergence.This replaces a rigid one-to-one target with softer distinctions among candidates.Source: MLLM Judgment Based Training Framework · p. 4

UniME-V2-Reranker

  • UniME-V2-Reranker trains on positives and mined negatives with joint pairwise and listwise objectives, combining the losses as L = Lpair + Llist.Pairwise training predicts YES for positives and NO for hardest negatives, while listwise training predicts the ground-truth candidate position.Source: MLLM Judgment Based Training Framework · p. 4

Inference

  • During inference, UniME-V2 retrieves the top 10 candidates by cosine similarity before UniME-V2-Reranker selects the best candidate from the list.The reranker evaluates each candidate against the query and outputs its corresponding position.Source: Inference Pipeline · pp. 4–5

Experiments and Results

The implementation trains UniME-V2 with MLLM-generated scores and multimodal backbones using memory-efficient distributed fine-tuning on high-resolution image inputs.

Implementation

  • UniME-V2 uses Qwen2-VL-7B for candidate-set construction, Qwen2.5VL-7B for semantic scores, and Qwen2-VL plus LLaVA-OneVision for training.LoRA with rank 16 and DeepSpeed ZeRO stage-2 reduce GPU-memory requirements during training.Source: Implementation · p. 5

Datasets and Evaluation

The experiments train and evaluate across diverse MMEB tasks, covering both in-distribution and out-of-distribution retrieval settings.

Training data

  • Training uses 20 in-distribution MMEB datasets spanning classification, visual question answering, multimodal retrieval, and visual grounding.The corpus contains 662k curated training pairs with unimodal and multimodal inputs.Source: Training Data · p. 5

Evaluation

  • Evaluation covers 20 in-distribution and 16 out-of-distribution MMEB test sets, reporting Precision for each dataset.The benchmark assesses multimodal embedding capabilities across diverse retrieval tasks.Source: Evaluation · p. 5

Main Results

UniME-V2 improves universal multimodal retrieval and hard-negative discrimination across foundation models and retrieval settings, while UniME-V2-Reranker further strengthens reranking performance.

Analysis

Analysis shows that MLLM-based hard-negative mining and judgment-based training improve retrieval, while judge quality and negative-set size materially affect performance.

Conclusion

UniME-V2 uses MLLM-based semantic judgments to mine diverse hard negatives and align embeddings with soft semantic labels. Experiments on MMEB and varied retrieval tasks report state-of-the-art average performance across tasks.

  • UniME-V2 aligns embedding similarities with MLLM-generated soft semantic matching scores to learn finer-grained distinctions among candidates.The scores also relax the rigid one-to-one matching constraint.Source: Conclusion · p. 7
  • The method constructs globally retrieved potential hard negatives, then uses MLLM-as-a-Judge to reduce false-negative interference and identify diverse, high-quality negatives.Source: Conclusion · p. 7
  • UniME-V2-Reranker applies joint pairwise and listwise optimization to mined hard negatives for further reranking improvement.Source: Conclusion · p. 7
  • State-of-the-art average performance is reported across the MMEB benchmark and various retrieval tasks.Source: Conclusion · p. 8

Detail Experiment Setting

The experiments specify training configurations, reranking prompts, evaluation protocols, benchmark summaries, and a temperature ablation. Qualitative mining examples illustrate the processed hard-negative candidates.

Reranker configuration

Reranker prompts

Evaluation protocol

  • Retrieval evaluations cover short-caption, long-caption, and compositional image-text tasks, primarily reporting Recall@1 with “Represent the image/text” prompts.The evaluation benchmarks summarize test-query and candidate counts.Source: Retrieval Task Evaluation Benchmarks · p. 10, Table 9 · p. 10

Qualitative analysis

  • Qualitative examples present queries and corresponding hard-negative candidates processed after the hard-negative mining pipeline.Source: Figure 6 · p. 11

External Results

External experiments examine temperature sensitivity and compare UniME-V2 with established multimodal embedding baselines on MMEB. The comparison distinguishes Qwen2-VL-7B and LLaVA-OneVision-7B backbones.

Temperature ablation

MMEB comparison

Further Analysis

Further analysis visualizes semantic matching scores and qualitative retrieval outcomes. The examples show graded candidate relevance and additional refinement from UniME-V2-Reranker.

Semantic matching scores

  • Target candidates receive nearly 1.0 matching scores, partially relevant candidates receive intermediate scores, and irrelevant candidates receive near-zero scores.Source: Visualization of the Training Data · p. 11

Retrieval and reranking

Benchmark coverage

  • The MMEB evaluation includes 20 in-distribution and 16 out-of-distribution datasets, with out-of-distribution datasets marked separately.Source: Table 11 · p. 12

Related papers

Loading 2510.13515v3…