PaperlayerView exact revision on arXiv

Source-linked AI summary

UMER: Unifying Embedding and Ranking via Pair-Aware Discriminative Reasoning for Universal Multimodal Retrieval

Libiao Chen, Xiyang Liu, Yanheng Wei, Tao Wang, Zhenyu Tang

arXiv:2608.18504v1cs.AI

TL;DR

Universal multimodal retrieval needs both efficient corpus-scale matching and fine-grained pairwise reasoning, but existing item-wise reasoning does not distinguish positives from hard negatives. UMER addresses this with pair-aware discriminative reasoning, jointly learned embedding and ranking functions, and mutual distillation, achieving state-of-the-art MMEB-V2 performance with a best overall score of 65.5.

  • Problem

    Existing item-wise CoT and contrastive embeddings lack explicit query–candidate evidence for distinguishing hard negatives and handling verification, classification, and fine-grained reasoning.

  • Method

    UMER jointly learns independently indexable contrastive embeddings and pair-aware discriminative ranking, with mutual distillation between the two functions.

  • Results

    UMER achieves state-of-the-art performance on MMEB-V2, with a best overall score of 65.5 and complementary strengths across embedding- and ranking-oriented tasks.

  • Takeaways & Limitations

    UMER supports efficient vector retrieval alongside explicit reasoning-based ranking for universal multimodal retrieval.

  • Takeaways & Limitations

    Pair-aware ranking can regress on spatial relations by focusing on foreground salience rather than preserving the query’s relation.

Abstract

from arXiv · show

Universal multimodal retrieval aims to support diverse instruction-aware retrieval tasks, demanding both efficient corpus-scale matching and fine-grained semantic reasoning. Recent MLLM-based embedding methods typically derive representations from hidden states, while Chain-of-Thought (CoT) reasoning is emerging as a promising strategy for embedding enhancement by encoding intermediate semantic evidence into the representation space. However, existing CoT methods typically use item-wise reasoning over queries and candidates in isolation, providing no explicit evidence to distinguish a positive from a semantically confusable hard negative. Moreover, contrastive embeddings capture global similarity but struggle with meta-tasks requiring answer verification, category judgment or fine-grained reasoning. In this paper, we propose UMER, a Unified Multimodal Embedding and Ranking framework for universal multimodal retrieval. UMER replaces item-wise reflection with Pair-Aware Discriminative Reasoning, which compares query--candidate pairs to identify instruction-relevant matching and discrepancy evidence. UMER jointly learns contrastive embeddings for efficient global matching and discriminative ranking for explicit pairwise relevance judgment within a single MLLM. A complementary mutual distillation strategy further transfers reliable pairwise preferences between the embedding and ranking functions. On the MMEB-V2 benchmark, UMER achieves state-of-the-art performance under comparable experimental settings while supporting budget-adjustable inference.

Introduction

Universal multimodal retrieval must combine efficient shared-space matching with reasoning-intensive relevance judgment across heterogeneous modalities and instruction-aware tasks. UMER addresses this gap with pair-aware discriminative reasoning, jointly learned embedding and ranking functions, mutual distillation, and budget-adjustable inference.

  • Background: Universal multimodal retrieval spans images, texts, videos, and documents, requiring both shared-space retrieval over large corpora and fine-grained instruction-aware relevance judgment.MLLM-based methods address complex instruction-aware retrieval, while CoT-derived hidden states incorporate intermediate reasoning into representations.
  • Motivation and contribution: UMER replaces item-wise CoT with Pair-Aware Discriminative Reasoning that compares query–candidate pairs using matching evidence for positives and discrepancy evidence for hard negatives.The paradigm takes positive and hard-negative pairs as reasoning inputs and produces a deterministic ranking judgment.
  • Framework: UMER jointly learns contrastive multimodal embeddings and discriminative ranking in one MLLM, targeting the complementary capabilities required by different retrieval meta-tasks.The framework is designed for heterogeneous universal multimodal retrieval requirements, with embedding and ranking serving complementary roles.
  • Framework: Complementary mutual distillation lets the ranking branch refine the embedding space while the embedding branch stabilizes ranking with global semantic structure.The contribution explicitly transfers knowledge between embedding and ranking functions.
  • Inference: UMER supports budget-adjustable inference: embedding-only vector indexing enables efficient large-scale retrieval, while additional pair-aware CoT ranking targets accuracy-oriented or hard-sample settings.The pipeline is Embedding-then-CoT-Ranking, adding relevance scores when greater discrimination is needed.

Related Work

Related work spans instruction-aware MLLM embedders, reasoning-enhanced representations, and multimodal rerankers. UMER addresses their separation and efficiency limitations by unifying embedding, pair-aware reasoning, and ranking in one end-to-end co-optimized MLLM.

  • Universal Multimodal Embedding: Universal multimodal embedding has progressed from dual encoders such as CLIP, ALIGN, and SigLIP (Radford et al. 2021; Jia et al. 2021; Zhai et al. 2023) to instruction-aware MLLM encoders.VLM2Vec (Jiang et al. 2024b) and VLM2Vec-V2 (Meng et al. 2025) use contrastive training on MMEB and MMEB-V2, while UniME-V2 (Gu et al. 2026) improves discriminability through hard-negative mining.
  • Reasoning-Enhanced Embedding: Reasoning-enhanced embedders generate intermediate rationales before extracting representations, but explicit CoT incurs decoding cost before each embedding.UME-R1 (Lan et al. 2026) derives representations after autoregressive rationale generation, Embed-RL (Jiang et al. 2026) optimizes rationales with retrieval-oriented rewards, and PLUME (He et al. 2026) plus LaME (Wu et al. 2026b) reduce overhead using latent reasoning.
  • MLLM Rerankers: Multimodal rerankers jointly encode query–candidate pairs for fine-grained relevance estimation, including RagVL (Chen et al. 2024), UniME-V2-Reranker (Gu et al. 2026), and Qwen3-VL-Reranker (Li et al. 2026).These models are typically separate second-stage modules, so ranking knowledge reaches embeddings through offline soft labels or mined pseudo-negatives.
  • Unified Framework: UMER unifies embedding, Pair-Aware Discriminative Reasoning, and ranking within one MLLM, sharing a backbone and enabling end-to-end co-optimization.This design replaces separate reranking modules and discrete knowledge transfer with jointly optimized embedding and ranking branches.

Methodology

UMER jointly learns independently computable embeddings for corpus-scale retrieval and pair-aware ranking for explicit relevance verification within one shared MLLM. It combines contrastive learning, discriminative pair reasoning, verified hard-negative supervision, selective mutual distillation, and staged training.

  • Method overview: UMER uses a shared MLLM to learn independently computable embeddings for corpus-scale candidate retrieval alongside pair-aware ranking for explicit verification.The embedding and ranking functions share the backbone while serving complementary retrieval roles.
  • Embedding encoding phase: Learnable query and candidate embedding tokens form fixed-dimensional representations trained with in-batch contrastive learning and mined hard negatives for independent corpus indexing.Branch-wise attention masks prevent embeddings from using unavailable cross-item information at retrieval time.
  • Pair-aware reasoning and ranking phase: Pair-aware discriminative CoT jointly exposes query–candidate evidence, then a rationale-conditioned [RANK] token predicts relevance and ranks positives above hard negatives.The losses supervise pair-conditioned evidence, absolute relevance labels, and positive-over-negative logit ordering.
  • Supervision construction: Training data uses hard-negative mining, structured CoT with Query Intent and Target Observations, and verifier filtering that retains only evidence-sufficient positive–negative triplets.A text-only verifier must infer the ground-truth Yes/No relevance label from both corresponding pair rationales.
  • Mutual distillation: Selective bidirectional CMD transfers embedding preferences to ranking on content-matching tasks and ranking preferences to embeddings on reasoning-intensive meta-tasks.Retrieval and Grounding/MR favor embedding supervision, while Classification and the remaining reasoning-intensive tasks favor ranking supervision.
  • Training and inference: Three-stage training first stabilizes embeddings, then adds verified pair-aware reasoning and ranking objectives, and finally distills preferences across capabilities.The hybrid score combines normalized embedding distance with normalized ranking distance, weighted by α.

Experiments and Results

UMER is evaluated on the 78-task MMEB-V2 benchmark using official modality-specific metrics and achieves its strongest overall performance with the hybrid UMER-H configuration. Ablations and efficiency studies show complementary embedding/ranking capabilities, benefits from selective mutual distillation, and budget-adjustable inference.

  • Capability specialization: Embedding is stronger on global content-matching retrieval and grounding, whereas ranking is stronger on reasoning-intensive classification and question answering.This task-dependent specialization motivates combining the two scoring functions within UMER-H.
  • Pair-aware reasoning and ranking: Pair-aware CoT raises UMER-E to 62.9, while ranking supervision alone raises UMER-E from 60.7 to 62.0 and produces 65.0 in hybrid mode.Pair-aware reasoning benefits the embedding space, whereas ranking supervision requires calibration of the ranking head before improving UMER-R.
  • Complementary mutual distillation: Selective bidirectional mutual distillation improves UMER-E and UMER-R to 63.1 and 63.9, while unconditional bidirectional transfer reaches only 62.6 and 63.6.One-way transfer improves the intended branch, but unconditional agreement can propagate teacher errors; the selective gate preserves complementary specialization while narrowing capability gaps.
  • Embedding efficiency: UMER-E reaches 63.1 with zero reasoning tokens and delivers 118.6×/99.6× query/indexing speedups over UME-R1 and 3.9×/3.1× over PLUME.UMER-H reuses the UMER-E index, avoiding sequential reasoning during embedding extraction; this is especially beneficial as indexing cost scales with corpus size.
  • Budget-adjustable inference: Increasing reranking K from 3 to 20 raises the score from 64.8 to 66.1 with nearly linear token and latency growth, while K = 5 achieves 65.5 at 24% of K = 20 latency.UMER-H reuses the UMER-E index and adds computation only for online top-K reranking; the default K = 5 captures 80% of the maximum observed gain.

Conclusion

UMER unifies multimodal embedding and ranking for universal multimodal retrieval through Pair-Aware Discriminative Reasoning. A shared MLLM learns independently indexable embeddings, pair-aware CoT generation, and discriminative ranking.

  • Conclusion: UMER presents a unified multimodal embedding and ranking framework for universal multimodal retrieval.
  • Conclusion: Pair-Aware Discriminative Reasoning compares queries with candidates to identify matching or discrepancy evidence, overcoming item-wise CoT’s limited discriminative supervision.
  • Conclusion: A shared MLLM jointly learns independently indexable embeddings, pair-aware CoT generation, and discriminative ranking.

Supplementary Material · Implementation and Evaluation Details

UMER’s supplementary implementation and evaluation details consolidate its multimodal training setup, hard-negative supervision, and offline preparation of pair-aware CoT targets. The settings are summarized in Table 1.

  • Implementation and Evaluation Details: The training mixture spans image, video, and visual-document sources, using curated positives instead of raw targets when available.
  • Implementation and Evaluation Details: Each training instance pairs a retrieved hard negative with contrastive and pairwise supervision, while pair-aware CoT targets are generated and audited offline before training.
  • Implementation and Evaluation Details: Table 1 consolidates UMER’s complete implementation and evaluation settings across the supplementary material.The table presents the overall configuration used for implementation and evaluation.

Pair-Aware CoT Data Construction

UMER constructs pair-aware rationales by contrasting each query with a relevant positive and a semantically close non-relevant candidate. The three-stage pipeline mines hard negatives, generates structured pair notes, and verifies those notes using text-only auditing.

  • Overview: Pair-aware supervision contrasts each query–positive pair with a plausible non-relevant candidate, teaching discriminative evidence rather than isolated query descriptions.The pipeline creates separate rationales for positive and negative pairs so supervision captures details distinguishing the labeled target from a distractor.
  • Stage 1: hard-negative mining: Hard-negative mining embeds source-local galleries, masks annotated positives, and retains up to three candidates after filtering likely missing positives by a 0.75 similarity threshold.Independently reviewed non-relevant candidates are preferred when available, with embedding-mined candidates as fallback; the final configuration uses one offline hard negative.
  • Stage 2: structured CoT generation: Structured CoT generation independently annotates positive and hard-negative pairs with query_intent and target_observations, using objective target-side facts without revealing pair labels or final decisions.The notes capture query-specific constraints and relevant entities, attributes, actions, relations, or text cues, including salient overlaps and missing or conflicting evidence.
  • Stage 3: text-only evidence verification: A text-only auditor accepts a rationale only when its match, not_match, or unknown prediction agrees with the pair label, the schema is valid, and no decision leaks into the notes.The auditor sees only the generated fields; across four source sidecars, the data contain 6,632,206 canonical pair records.

Complete MMEB-V2 Results

UMER achieves the strongest overall performance across the complete 78-task MMEB-V2 benchmark for both Qwen2-VL-2B and Qwen2-VL-7B models. Its hybrid configuration leads overall and across image, video, and visual-document task averages under the reported evaluation settings.

  • Qwen2-VL-2B results: 65.5 overall, 70.4 on image, 45.0 on video, and 73.6 on visual-document tasks make UMER-H best among Qwen2-VL-2B models, surpassing RIME overall by 1.4 points.Table 2 covers every MMEB-V2 task and reports detailed task-level outcomes for published 2B baselines.
  • Evaluation scope: The complete comparison spans 78 MMEB-V2 tasks, uses Hit@1 for image and video and NDCG@5 for visual documents, and distinguishes embedding, ranking, and hybrid inference.The tables include only baselines with published task-level results at the corresponding model size.
  • Qwen2-VL-7B results: 70.6 overall, 74.3 on image, 50.5 on video, and 80.0 on visual-document tasks are achieved by UMER-H with the same K = 5 hybrid configuration.Table 3 compares Qwen2-VL-7B models with published 78-task breakdowns.

Further Inference Analysis

UMER-H remains robust across a wide ranking-weight range, while embedding and pair-aware ranking provide complementary correct decisions across MMEB-V2 queries. Their comparable exclusive successes support hybrid inference rather than dominance by either branch.

  • Ranking-weight sensitivity: A ranking-score weight from 0.5–5.0 keeps UMER-H within 0.8 points of the peak score of 65.5, indicating robust hybrid inference without retraining.The default weight is 2.0; larger weights cause a small decline, suggesting neither branch should dominate every query.
  • Decision overlap: Across 83,530 MMEB-V2 queries, embedding-only and ranking-only successes are 9.0% and 10.3%, while both branches are correct for 47.2%.The comparable exclusive sets show that neither branch is uniformly dominant, establishing complementary decision signals for hybrid inference.

Qualitative Results and Failure Cases

Qualitative cases show pair-aware ranking correcting embedding failures on fine-grained composition, attribute conjunctions, and visual metaphors, while exposing a spatial-relation regression. The examples compare positive candidates with top-1 hard negatives and report rank changes before and after reranking.

  • Case 2: Attribute-sensitive composed retrieval: For conjunctive attributes, ranking moves the positive from rank 4 to rank 1 by selecting two different-species animals with no human instead of the visually similar underwater reference.The embedding result preserves the scene but violates both explicit requirements.
  • Case 3: Visual-metaphor document retrieval: For visual-metaphor retrieval, ranking moves the positive from rank 3 to rank 1 by selecting the slide combining a sinking-car icon with the consequence of blindly following data.Embedding instead selects a semantically related video slide whose visual evidence is a video player rather than the requested metaphor.
  • Case 4: Spatial-relation ranking regression: Pair-aware ranking regresses on a spatial relation, replacing the correct background motorcycle at embedding rank 1 with the salient foreground motorcycle at rank 2.Its candidate-specific reasoning emphasizes foreground salience and fails to preserve the requested behind/on-the-right relation, although the correct candidate remains in the top-2.
  • Case 1: Fine-grained image composition: For dog retrieval, ranking corrects an embedding result that matches the breed but misses the requested head-focused close-up composition, moving the positive from rank 5 to rank 1.The target is a Cavalier King Charles Spaniel portrait with the head centered, whereas the embedding top result is a running dog image.
Loading 2608.18504v1…