PaperlayerView exact revision on arXiv

Source-linked AI summary

UMER: Unifying Embedding and Ranking via Pair-Aware Discriminative Reasoning for Universal Multimodal Retrieval

Libiao Chen, Xiyang Liu, Yanheng Wei, Tao Wang, Zhenyu Tang

arXiv:2608.18504v2cs.AI

TL;DR

Existing item-wise CoT lacks evidence explaining why positives match and confusable hard negatives fail. UMER combines pair-aware reasoning, embeddings, and discriminative ranking with complementary mutual distillation in one MLLM. On MMEB-V2, UMER-H leads overall and across modality averages, surpassing RIME by 1.4 points overall under comparable experimental settings.

  • Problem

    Existing item-wise CoT enriches isolated queries and candidates but lacks interaction evidence explaining why positives match and semantically confusable hard negatives fail.

    Source: Introduction · p. 2

  • Method

    UMER jointly learns multimodal embeddings and pair-aware discriminative ranking in one MLLM, using query–candidate matching and discrepancy evidence plus complementary mutual distillation.

    Source: Conclusion · p. 7

  • Results

    UMER-H leads MMEB-V2 overall and across image, video, and visual-document averages, surpassing RIME by 1.4 points overall under comparable experimental settings.

    Source: Introduction · p. 2, Qwen2-VL-2B Results · p. 10

  • Takeaways & Limitations

    Embedding retrieval and pair-aware ranking have complementary task strengths: global matching favors embeddings, while answer and category verification favor ranking.

    Source: Main Results on MMEB-V2 · p. 6

    The discussed explicit-CoT embedding approaches incur substantial decoding cost because they generate a rationale before each embedding.

    Source: Reasoning-Enhanced Multimodal Retrieval · p. 3

Abstract

from arXiv · show

Universal multimodal retrieval aims to support diverse instruction-aware retrieval tasks, demanding both efficient corpus-scale matching and fine-grained semantic reasoning. Recent MLLM-based embedding methods typically derive representations from hidden states, while Chain-of-Thought (CoT) reasoning is emerging as a promising strategy for embedding enhancement by encoding intermediate semantic evidence into the representation space. However, existing CoT methods typically use item-wise reasoning over queries and candidates in isolation, providing no explicit evidence to distinguish a positive from a semantically confusable hard negative. Moreover, contrastive embeddings capture global similarity but struggle with meta-tasks requiring answer verification, category judgment or fine-grained reasoning. In this paper, we propose UMER, a Unified Multimodal Embedding and Ranking framework for universal multimodal retrieval. UMER replaces item-wise reflection with Pair-Aware Discriminative Reasoning, which compares query--candidate pairs to identify instruction-relevant matching and discrepancy evidence. UMER jointly learns contrastive embeddings for efficient global matching and discriminative ranking for explicit pairwise relevance judgment within a single MLLM. A complementary mutual distillation strategy further transfers reliable pairwise preferences between the embedding and ranking functions. On the MMEB-V2 benchmark, UMER achieves state-of-the-art performance under comparable experimental settings while supporting budget-adjustable inference.

Introduction

UMER unifies multimodal embeddings and discriminative ranking through pair-aware reasoning and complementary mutual distillation. It supports efficient vector retrieval with optional reasoning-based reranking.

Introduction

  • Item-wise CoT enriches isolated item semantics but lacks query–candidate interaction evidence explaining why positives match and semantically similar hard negatives fail.Existing methods (Lan et al. 2026; Jiang et al. 2026; He et al. 2026) independently describe queries or candidates. Autoregressive cross-entropy supervision also injects reasoning only indirectly into embedding tokens, creating a mismatch between generation quality and embedding discriminability.Source: Figure 1 · p. 1, Introduction · p. 2
  • Embedding retrieval and interaction-based ranking offer complementary capabilities because universal multimodal tasks require both global similarity matching and fine-grained relevance reasoning.The retrieval scope spans images, texts, videos, and documents. Content-matching tasks emphasize global similarity, whereas question answering and classification require answer evidence or category-specific conditions that a single contrastive objective cannot adequately capture.Source: Introduction · pp. 1–2, Figure 1 · p. 1
  • Pair-Aware Discriminative CoT compares positive and hard-negative query–candidate pairs to identify matching evidence, expose discrepancies, and produce deterministic ranking judgments.The authors also construct pair-aware CoT and ranking data to support this reasoning paradigm.Source: Introduction · p. 2
  • Complementary Mutual Distillation transfers discriminative ranking knowledge into the embedding space while using global embedding structure to stabilize ranking.This bidirectional transfer co-optimizes the framework’s embedding and ranking capabilities.Source: Introduction · p. 2
  • Budget-adjustable inference uses one-pass embedding extraction and vector indexing for efficiency, optionally adding pair-aware CoT and relevance scoring for reranking.The additional reasoning stage targets accuracy-oriented settings or hard samples.Source: Introduction · p. 2
  • UMER achieves state-of-the-art performance on MMEB-V2 under comparable experimental settings, combining efficient vector retrieval with improved accuracy through explicit reasoning and ranking.The unified MLLM jointly learns contrastive embeddings for corpus-scale matching and discriminative ranking for query–candidate relevance judgment.Source: Introduction · p. 2

Related Work

Multimodal embedders provide limited explicit candidate-comparison reasoning, reasoning-enhanced approaches remain item-wise, and rerankers are usually separate second-stage modules. UMER instead co-optimizes embedding, pair-aware reasoning, and ranking within one MLLM.

Multimodal Embedding Models

  • Instruction-following MLLM embedders advance multimodal retrieval but largely retain in-batch contrastive objectives with limited explicit reasoning about subtly different candidates.VLM2Vec (Jiang et al. 2024b) and VLM2Vec-V2 (Meng et al. 2025) use contrastive training on MMEB and MMEB-V2. UniME-V2 (Gu et al. 2026) adds MLLM-as-a-Judge hard-negative mining and soft-label alignment, while Qwen3-VL-Embedding (Li et al. 2026) combines large-scale contrastive training with reranker distillation.Source: Multimodal Embedding Models · p. 2

Reasoning-Enhanced Multimodal Retrieval

  • UME-R1 (Lan et al. 2026) extracts embeddings after generating rationales, while Embed-RL (Jiang et al. 2026) optimizes rationales using retrieval-oriented reinforcement learning.UME-R1 derives representations from hidden states after autoregressive reasoning; Embed-RL aligns reasoning traces with downstream matching signals.Source: Reasoning-Enhanced Multimodal Retrieval · pp. 2–3
  • Reasoning-enhanced retrieval remains item-wise, enriching queries and candidates independently rather than explicitly identifying their matching and discrepancy evidence.Explicit CoT requires costly rationale decoding before each embedding. PLUME (He et al. 2026) reduces overhead through continuous latent rollouts, while LaME (Wu et al. 2026b) uses an information bottleneck with learnable reason tokens.Source: Reasoning-Enhanced Multimodal Retrieval · p. 3

Multimodal Reranking Models

  • MLLM rerankers jointly encode query–candidate pairs to estimate fine-grained relevance after initial retrieval.RagVL (Chen et al. 2024) instruction-tunes image filtering for retrieval-augmented generation, while UniME-V2-Reranker (Gu et al. 2026) uses mined hard negatives with pairwise and listwise optimization. Qwen3-VL-Reranker (Li et al. 2026) uses a crossencoder to complement Qwen3-VL-Embedding in a two-stage pipeline.Source: Multimodal Reranking Models · p. 3
  • UMER unifies embedding, Pair-Aware Discriminative Reasoning, and ranking within one MLLM, unlike separately trained and deployed rerankers.Its branches share a backbone and are co-optimized end-to-end; separate rerankers instead transfer ranking knowledge through discrete offline targets such as soft labels or mined pseudo-negatives.Source: Multimodal Reranking Models · p. 3

Methodology

UMER jointly learns independently computable embeddings and pair-aware relevance judgments within one shared MLLM. It uses verified reasoning supervision, selective mutual distillation, and three-stage training, with embedding-only, ranking, and hybrid inference modes.

Unified Embedding and Ranking Architecture

  • UMER shares one MLLM across embedding and ranking functions, using attention masks and task-specific tokens to preserve distinct information flows.Branch-wise attention keeps query and candidate encodings independent for corpus indexing and approximate nearest-neighbor search. Learnable embedding-token states are mean-pooled, ℓ2-normalized, and trained with in-batch contrastive learning augmented by mined hard negatives.Source: Methodology · p. 3, Unified Embedding and Ranking… · p. 4
  • Pair-aware reasoning grounds relevance judgments in jointly visible query–candidate evidence, and a rationale-conditioned [RANK] token produces the relevance logit.After independently encoding inputs, the model generates a reasoning trace with full pairwise attention, supervised by token-level cross-entropy. Binary cross-entropy supplies absolute relevance labels, while a logistic pairwise objective encourages positive logits to exceed hard-negative logits.Source: Unified Embedding and Ranking… · p. 4

Pair-Aware Discriminative CoT Data Construction

  • Three highest-similarity non-relevant candidates per query supply hard negatives for structured CoT generation and evidence-sufficiency filtering.A frozen multimodal embedder mines candidates; Qwen3.5-9B generates Query Intent and Target Observations describing required and supporting or contradicting evidence, without explicit Yes/No answers. Qwen3.5-0.8B must infer the correct relevance label from each rationale alone, and both positive and negative rationales must pass for triplet retention.Source: Pair-Aware Discriminative CoT Data… · pp. 4–5

Complementary Mutual Distillation

  • Selective bidirectional distillation transfers only correctly ordered reference preferences, using embedding teachers for Retrieval and Grounding/MR and ranking teachers for Classification and QA.A frozen reference produces temperature-scaled positive–negative preference distributions, with gates activated only for correctly ordered triplets in the designated task group. Embedding-to-ranking transfer supports content matching, while ranking-to-embedding transfer refines reasoning-intensive retrieval.Source: Complementary Mutual Distillation · p. 5

Progressive Training and Flexible Inference

  • Three-stage training first learns embeddings, then adds verified CoT and ranking supervision, and finally introduces complementary mutual distillation.Stage 1 uses only Lemb on the full MMEB training set; Stage 2 adds Lcot, Lbce, and Lmargin on verified pair-aware triplets. Stage 3 freezes the Stage 2 checkpoint as its reference and adds Lcmd.Source: Progressive Training and Flexible… · p. 5
  • UMER-E retrieves with embeddings alone, UMER-R reranks retrieved candidates using pair-aware logits, and UMER-H combines normalized embedding scores and ranking logits.Precomputed candidate embeddings support approximate nearest-neighbor retrieval; autoregressive pair-aware reasoning is restricted to the retrieved top-K candidates. Hybrid scores normalize both signals within that list, with α controlling the ranking contribution.Source: Progressive Training and Flexible… · p. 5

Experiments and Results

UMER-H achieves the best overall MMEB-V2 score by fusing complementary embedding and ranking scores. Pair-aware supervision and selective mutual distillation improve the branches, while embedding-only retrieval and adjustable reranking offer different accuracy–efficiency tradeoffs.

Experimental Setup

  • 78 image, video, and visual-document tasks in MMEB-V2 (Meng et al. 2025) cover classification, question answering, retrieval, grounding, and moment retrieval.Hit@1 measures image/video tasks, NDCG@5 measures visual-document retrieval, meta-task scores macro-average constituent datasets, and All macro-averages all 78 tasks. Evaluation uses corrected ViDoSeek-page and MMLongBench-page versions and training data from the same public source datasets as VLM2Vec-V2.Source: Experimental Setup · p. 5

Main Results on MMEB-V2

  • 65.5 overall makes UMER-H best on MMEB-V2, exceeding UMER-E’s 63.1 and UMER-R’s 63.9 while leading all three modality-level results.UMER-H achieves Image 70.4, Video 45.0, and VisDoc 73.6. UMER-R favors I-QA, V-CLS, V-QA, and VisDoc-VR, whereas UMER-E favors I-RET, I-GD, VDRv1, and VDRv2.Source: Main Results on MMEB-V2 · pp. 5–6, Table 1 · p. 6

Ablation Studies

  • 65.4 in hybrid mode results from combining pair-aware CoT and ranking supervision, supporting their complementary rather than interchangeable roles.CoT alone raises UMER-E to 62.9 but leaves UMER-R at 45.4 because its [RANK] head is uncalibrated. Ranking losses alone raise UMER-E from 60.7 to 62.0 and achieve hybrid 65.0; combining both signals restores UMER-R to 63.4.Source: Ablation Studies · p. 6, Table 2 · p. 7
  • 63.1 for UMER-E and 63.9 for UMER-R result from selective bidirectional distillation, outperforming unconditional bidirectional transfer.r→e raises UMER-E from 62.4 to 62.7 with UMER-R unchanged; e→r raises UMER-R from 63.4 to 63.9 while shifting UMER-E to 62.2. Unconditional transfer yields UMER-E 62.6 and UMER-R 63.6 because agreement is enforced even when the teacher is incorrect.Source: Ablation Studies · p. 6, Table 2 · p. 7

In-Depth Analysis

  • 6.80 versus 5.24 points is UMER-E’s versus UME-R1’s task-macro-averaged positive–hard-negative cosine margin, favoring pair-aware over item-wise supervision.Across 3,600 queries from 36 image tasks, the margin subtracts the highest-scoring non-positive similarity from the highest-scoring positive similarity. Both produce independently indexable embeddings and comparable positive-margin coverage, but UMER-E retains 5.0 and 6.8 percentage points more queries at thresholds 10 and 20, respectively.Source: In-Depth Analysis · p. 6, Figure 3 · p. 7
  • 1.6 and 0.8 points are ranking’s CMD gains on Retrieval and Grounding/MR; embedding gains 0.3 and 0.5 on CLS and QA.Embedding initially favors content-matching Retrieval and Grounding/MR, whereas ranking favors reasoning-intensive CLS and QA. Positive gains across all groups indicate that CMD narrows capability gaps without eliminating this specialization.Source: In-Depth Analysis · pp. 6–7, Figure 4 · p. 7

Accuracy–Efficiency Tradeoff

  • 63.1 is the best embedding-only score, achieved by UMER-E with zero reasoning tokens and without sequential reasoning before embedding extraction.Query/indexing speedups are 118.6×/99.6× over UME-R1 and 3.9×/3.1× over PLUME. Measurements use one NVIDIA A100 GPU with batch size 1.Source: Accuracy–Efficiency Tradeoff · p. 7, Table 3 · p. 7
  • 65.5 at K = 5 captures 80% of UMER-H’s maximum observed gain over UMER-E using only 24% of the K = 20 latency.UMER-H reuses the UMER-E index and adds computation only for online top-K reranking. Increasing K from 3 to 20 raises the score from 64.8 to 66.1, while tokens and latency grow nearly linearly.Source: Accuracy–Efficiency Tradeoff · p. 7

Conclusion

UMER unifies multimodal embedding retrieval and reasoning-based ranking, establishing a new state of the art on MMEB-V2 under comparable experimental settings.

Conclusion

  • UMER establishes a new state of the art on MMEB-V2 under comparable experimental settings, demonstrating the complementary value of embedding retrieval and reasoning-based ranking.The experiments support combining efficient embedding retrieval with explicit reasoning-based ranking for universal multimodal retrieval.Source: Conclusion · p. 7
  • Pair-Aware Discriminative Reasoning explicitly compares queries with candidates to identify matching or discrepancy evidence, addressing the limited discriminative supervision of item-wise CoT.Source: Conclusion · p. 7
  • A shared MLLM jointly learns independently indexable embeddings, pair-aware CoT generation, and discriminative ranking.Source: Conclusion · p. 7
  • Complementary mutual distillation cooptimizes global metric matching and fine-grained relevance judgment.Source: Conclusion · p. 7

Supplementary Material · Implementation and Evaluation Details

UMER trains on image, video, and visual-document data using retrieved hard negatives for contrastive and pairwise supervision. Its pair-aware CoT targets are generated and audited offline before training.

Implementation and Evaluation Details

Pair-Aware CoT Data Construction

UMER constructs offline pair-aware rationales through hard-negative mining, structured evidence generation, and text-only verification. The resulting supervision targets details that distinguish labeled positives from plausible distractors rather than describing queries in isolation.

Stage 1: hard-negative mining

  • Hard-negative mining uses frozen Qwen3-VL-Embedding-8B representations and cosine similarity within source-local galleries, masking annotated positives before selecting plausible distractors.Candidates with similarity at least 0.75 times the annotated positive’s similarity are discarded to reduce missing-positive contamination, leaving a ranked pool of at most three. Independently reviewed non-relevant candidates take precedence over this embedding-mined fallback.Source: Pair-Aware CoT Data Construction · p. 9

Stage 2: structured CoT generation

  • Label-blind Qwen3.5-9B generates structured query intent and target observations for positive and hard-negative pairs, recording overlaps and discrepancies without a final relevance decision.Image/retrieval prompts request three to eight target-side facts. Generation samples three rationales per image/retrieval pair and one per video/document pair, using temperature 0.7, top-p 0.9, and respective limits of 384 and 192 tokens.Source: Pair-Aware CoT Data Construction · p. 9

Stage 3: text-only evidence verification

  • Text-only verification asks Qwen3.5-0.8B to infer match, not_match, or unknown solely from generated notes, without original inputs, dataset information, or relevance labels.A strict audit requires agreement with the pair label, valid schema, and no leaked final decision. The verifier requires concrete supporting or conflicting observations; unsupported decisions are insufficient.Source: Pair-Aware CoT Data Construction · p. 9

Training record selection

  • 6,632,206 canonical pair records populate four source sidecars, with training retrieving CoT records through normalized query–target pair keys.The released configuration selects the first schema-valid image/retrieval trace and one video/document trace per pair. Training uses one offline hard negative per query—the first candidate for embedding and the first with CoT for reranking—and forms reranking examples only when both positive and negative records exist.Source: Pair-Aware CoT Data Construction · p. 9

Complete MMEB-V2 Results

UMER-H achieves the best overall and modality averages in the reported Qwen2-VL-2B comparison, exceeding RIME by 1.4 points overall. The Qwen2-VL-7B hybrid configuration reaches 70.6 overall with K = 5.

Qwen2-VL-2B Results

  • 65.5 overall, 70.4 image, 45.0 video, and 73.6 visual-document averages place UMER-H first in the Qwen2-VL-2B comparison, exceeding RIME by 1.4 overall points.Image and video results use Hit@1; visual-document results use NDCG@5.Source: Table 2 · p. 10, Qwen2-VL-2B Results · p. 10

Qwen2-VL-2B Results; Qwen2-VL-7B Results

  • Both comparisons cover all 78 MMEB-V2 tasks and include only baselines with published task-level results at the corresponding model size.E, R, and H denote embedding retrieval, pair-aware ranking, and hybrid inference, respectively.Source: Table 2 · p. 10, Table 3 · p. 11

Qwen2-VL-7B Results

  • 70.6 overall, 74.3 image, 50.5 video, and 80.0 visual-document scores are achieved by Qwen2-VL-7B UMER-H using hybrid inference with K = 5.Image and video results use Hit@1; visual-document results use NDCG@5.Source: Table 3 · p. 11, Qwen2-VL-7B Results · p. 11

Further Inference Analysis

Embedding retrieval and pair-aware ranking provide complementary correct decisions, while hybrid inference remains robust across a broad range of ranking-score weights.

Fusion Configuration and Sensitivity

  • 0.8 points or less separates every tested UMER-H score from the 65.5 peak across ranking-score weights of 0.5–5.0 at K = 5.Default fusion standardizes both scores within the retrieved candidate set and assigns ranking scores weight 2.0. Candidates, normalization, and decoding remain fixed without retraining; the small decline at larger weights suggests neither branch should dominate every query.Source: Fusion Configuration and Sensitivity · p. 12, Table 4 · p. 12

Embedding–Ranking Complementarity

  • 9.0% of queries are correct only with embedding retrieval, 10.3% only with pair-aware ranking, and 47.2% with both, showing neither branch is uniformly dominant.The comparison covers the same 83,530 MMEB-V2 queries with unchanged retrieved candidates and evaluation labels.Source: Table 5 · p. 12, Embedding–Ranking Complementarity · p. 12
  • 12.1% of reasoning/semantic queries are correct only with ranking, versus 9.0% only with embedding retrieval.Content-matching queries have nearly symmetric exclusive sets (8.9% and 8.3%). These findings establish complementary decision signals, not that fusion corrects every exclusive error.Source: Embedding–Ranking Complementarity · p. 12, Table 5 · p. 12

Qualitative Results and Failure Cases

Pair-aware reranking corrects three examples where embedding similarity misses instruction-specific constraints, but regresses when foreground salience overrides a required spatial relation. Each comparison fixes one query and examines independently generated reasoning for the positive and a competing candidate.

Cross-Case Analysis

  • Cases 1–3 show pair-aware ranking correcting broad-category or topical embedding matches by enforcing compositional, visual, and document-level constraints.Each case contrasts the positive with the top-1 non-positive responsible for the correction or regression, showing independently generated pair-aware CoT and ranks before and after reranking.Source: Cross-Case Analysis · p. 12, Qualitative Results and Failure Cases · p. 12
  • Rank 5 to rank 1: pair-aware ranking promotes a head-focused Cavalier King Charles Spaniel portrait over an outdoor running-dog image.The negative’s CoT identifies an action shot with motion-blurred facial features, whereas the positive’s CoT describes a centered head, visible facial features, and close-up composition.Source: Cross-Case Analysis · pp. 12–13
  • Rank 4 to rank 1: pair-aware ranking promotes two ray-like animals without humans over the retrieved reference image, which violates the requested composition.The query requires two animals of a different species and no person; the positive candidate’s CoT explicitly records both animals and the absence of humans.Source: Cross-Case Analysis · pp. 12, 14
  • Rank 3 to rank 1: pair-aware ranking promotes the sinking-car metaphor slide over a topically related video slide about data.The negative candidate’s CoT reframes intent around a YouTube video, while the positive candidate’s CoT identifies a small sinking-car icon but qualifies the text’s explicit connection to the metaphor.Source: Cross-Case Analysis · pp. 12, 15
  • Pair-aware ranking regresses on spatial relations by favoring the salient foreground motorcycle over the requested motorcycle behind the red bike on the right.The correct candidate remains in the top-2, and its own CoT describes the requested background relation; the distractor’s reasoning instead prioritizes foreground salience.Source: Cross-Case Analysis · pp. 12, 16

Related papers

Loading 2608.18504v2…