Source-linked AI summary
UMER: Unifying Embedding and Ranking via Pair-Aware Discriminative Reasoning for Universal Multimodal Retrieval
Libiao Chen, Xiyang Liu, Yanheng Wei, Tao Wang, Zhenyu Tang
TL;DR
Universal multimodal retrieval must combine efficient corpus-scale matching with fine-grained reasoning, but item-wise CoT lacks explicit evidence distinguishing positives from hard negatives. UMER jointly learns pair-aware discriminative ranking and contrastive embeddings with mutual distillation, achieving a best overall MMEB-V2 score of 65.5 and surpassing RIME by 1.4 points.
Problem
Existing item-wise CoT does not model query–candidate interactions or explain why positives match and hard negatives fail, limiting discriminative evidence for retrieval.
Method
UMER uses Pair-Aware Discriminative Reasoning to jointly learn independently indexable embeddings and pairwise ranking, with complementary mutual distillation between them.
Results
65.5 overall, 1.4 points above RIME: UMER-H achieves the best overall MMEB-V2 score.
Takeaways & Limitations
UMER combines efficient vector retrieval with explicit reasoning-based ranking for universal multimodal retrieval.
Takeaways & Limitations
Explicit CoT reasoning introduces substantial decoding cost because a rationale must be generated before each embedding.
Abstract
from arXiv · showhide
Universal multimodal retrieval aims to support diverse instruction-aware retrieval tasks, demanding both efficient corpus-scale matching and fine-grained semantic reasoning. Recent MLLM-based embedding methods typically derive representations from hidden states, while Chain-of-Thought (CoT) reasoning is emerging as a promising strategy for embedding enhancement by encoding intermediate semantic evidence into the representation space. However, existing CoT methods typically use item-wise reasoning over queries and candidates in isolation, providing no explicit evidence to distinguish a positive from a semantically confusable hard negative. Moreover, contrastive embeddings capture global similarity but struggle with meta-tasks requiring answer verification, category judgment or fine-grained reasoning. In this paper, we propose UMER, a Unified Multimodal Embedding and Ranking framework for universal multimodal retrieval. UMER replaces item-wise reflection with Pair-Aware Discriminative Reasoning, which compares query--candidate pairs to identify instruction-relevant matching and discrepancy evidence. UMER jointly learns contrastive embeddings for efficient global matching and discriminative ranking for explicit pairwise relevance judgment within a single MLLM. A complementary mutual distillation strategy further transfers reliable pairwise preferences between the embedding and ranking functions. On the MMEB-V2 benchmark, UMER achieves state-of-the-art performance under comparable experimental settings while supporting budget-adjustable inference.
Introduction
Universal multimodal retrieval combines efficient shared-space matching with reasoning-intensive relevance judgment across heterogeneous media and instruction-aware tasks. UMER addresses this gap through pair-aware discriminative reasoning and a unified framework that jointly learns embeddings and ranking with adjustable inference cost.
- Introduction: Universal multimodal retrieval targets instruction-aware tasks across images, texts, videos, and documents by encoding heterogeneous inputs into a shared embedding space.MLLM-based methods extend this setting, while CoT-based approaches derive embeddings from hidden states informed by textual rationales and intermediate reasoning.
- Introduction: Existing item-wise CoT lacks pair-aware evidence to distinguish positives from semantically confusable hard negatives, while embedding and ranking provide complementary capabilities.Figure 1 identifies these as overlooked issues in universal multimodal retrieval.
- Introduction: UMER jointly learns multimodal embeddings and discriminative ranking in one MLLM to support heterogeneous universal retrieval requirements.The framework combines contrastive embedding learning with discriminative ranking, while complementary mutual distillation transfers knowledge between the two functions.
- Introduction: Pair-Aware Discriminative CoT compares positive and hard-negative query–candidate pairs to identify matching and discrepancy evidence and produce deterministic ranking judgments.This replaces independent item-wise reflection with explicit pairwise comparison for representation learning.
- Introduction: UMER supports budget-adjustable inference, using one-pass embeddings and vector indexing for efficient retrieval or adding pair-aware CoT ranking for accuracy-oriented and hard-sample settings.The Embedding-then-CoT-Ranking pipeline enables staged computation according to the inference budget.
Related Work
Prior work spans instruction-aware multimodal embedders, reasoning-enhanced representations, and pairwise rerankers, but these approaches leave efficiency or integration gaps. UMER addresses this gap by unifying embedding, pair-aware reasoning, and ranking in one end-to-end optimized MLLM.
- Universal Multimodal Embedding: Universal multimodal embedding has progressed from dual encoders to instruction-following MLLM-based embedders trained contrastively on MMEB benchmarks.This progression includes CLIP, ALIGN and SigLIP (Radford et al. 2021; Jia et al. 2021; Zhai et al. 2023), followed by VLM2Vec (Jiang et al. 2024b) and VLM2Vec-V2 (Meng et al. 2025).
- Reasoning-Enhanced Embedding: Reasoning-enhanced embedders encode intermediate rationales before representation extraction, but explicit decoding adds substantial cost for every embedding.UME-R1 (Lan et al. 2026) generates rationales autoregressively, Embed-RL (Jiang et al. 2026) optimizes them with retrieval-oriented reinforcement learning, and latent approaches such as PLUME (He et al. 2026) and LaME (Wu et al. 2026b) reduce this overhead.
- MLLM Reranking: MLLM rerankers jointly encode query–candidate pairs for fine-grained relevance estimation, but are typically separate second-stage modules.Examples include RagVL (Chen et al. 2024), UniME-V2-Reranker (Gu et al. 2026), and Qwen3-VL-Reranker (Li et al. 2026).
- UMER’s Unification: UMER unifies embedding, Pair-Aware Discriminative Reasoning, and ranking within a single MLLM, allowing shared-backbone end-to-end co-optimization instead of discrete offline knowledge transfer.Existing rerankers transfer ranking knowledge to embeddings through soft labels or mined pseudo-negatives, whereas UMER jointly optimizes the two branches.
Methodology
UMER jointly trains independently computable embeddings for corpus-scale retrieval and pair-aware ranking for explicit relevance verification within one shared MLLM. It combines contrastive hard-negative learning, structured evidence-grounded reasoning, selective cross-capability distillation, and staged optimization.
- Method overview: UMER jointly learns an independently computable embedding function and a pair-aware ranking function with a shared MLLM for retrieval and verification.The embedding supports corpus-scale candidate retrieval, while ranking explicitly verifies query–candidate relevance.
- Embedding encoding phase: Contrastive learning maps queries and candidates into an independently indexable shared metric space using learnable aggregation tokens, in-batch candidates, and mined hard negatives.Branch-wise attention prevents embeddings from using cross-item information unavailable during retrieval.
- Pair-aware reasoning and ranking phase: Pair-aware discriminative CoT compares each query–candidate pair, grounds reasoning in jointly visible evidence, and feeds a rationale-conditioned representation to binary and pairwise ranking losses.The objectives supervise pair-conditioned evidence, absolute relevance, and higher positive than hard-negative logits.
- Supervision construction: Verified training triplets combine frozen-embedder hard-negative mining, structured Query Intent and Target Observations CoT, and text-only verification of both pair rationales.A triplet is retained only when the verifier correctly infers the ground-truth Yes/No label from both the positive and corresponding hard-negative CoTs.
- Mutual distillation and training: Selective bidirectional CMD transfers embedding preferences to ranking on content-matching tasks and ranking preferences to embeddings on reasoning-intensive tasks, preserving complementary roles.UMER trains progressively: embeddings first, verified-pair reasoning and ranking second, then cross-capability distillation from a frozen reference model.
Experiments and Results
UMER is evaluated on the 78-task MMEB-V2 benchmark using modality-specific retrieval metrics and achieves strong overall and modality-level performance. Ablations show complementary embedding–ranking capabilities, while mutual distillation and adjustable reranking improve capability transfer and efficiency.
- Main results: UMER-H achieves the best overall MMEB-V2 score of 65.5, with best modality-level results on Image (70.4), Video (45.0), and VisDoc (73.6).MMEB-V2 spans 78 tasks across image, video, and visual-document domains, covering classification, question answering, retrieval, grounding, and moment retrieval.
- Pair-aware reasoning and ranking: Pair-aware CoT raises UMER-E to 62.9, ranking supervision raises hybrid performance to 65.0, and combining both mechanisms further benefits the unified system.Pair-aware CoT directly improves the embedding space, while ranking losses provide effective pairwise supervision; the [RANK] head remains uncalibrated with CoT alone.
- Complementary mutual distillation: Selective bidirectional distillation improves UMER-E and UMER-R to 63.1 and 63.9, outperforming unconditional bidirectional transfer because it avoids enforcing agreement with incorrect teachers.One-way transfer improves its intended branch, while unconditional bidirectional transfer captures only part of the benefit at 62.6 / 63.6.
- Capability specialization and transfer: Embedding excels at content-matching retrieval and grounding, whereas ranking is stronger for reasoning-intensive classification and question answering.This capability split reflects distinct inductive biases, and complementary mutual distillation improves both functions across task groups without collapsing their specialization.
- Efficiency and budget-adjustable inference: UMER-E achieves 63.1 with zero reasoning tokens, delivering 118.6×/99.6× speedups over UME-R1 and 3.9×/3.1× over PLUME for query/indexing.UMER-H reuses the UMER-E index, and increasing reranking K from 3 to 20 raises the score from 64.8 to 66.1; default K = 5 reaches 65.5 at 24% of K = 20 latency.
Conclusion
UMER unifies multimodal embedding and ranking for universal multimodal retrieval by explicitly comparing query–candidate pairs through Pair-Aware Discriminative Reasoning. A shared MLLM jointly learns independently indexable embeddings, pair-aware CoT generation, and discriminative ranking.
- UMER presents a unified multimodal embedding and ranking framework for universal multimodal retrieval.
- Pair-Aware Discriminative Reasoning compares queries with candidates to identify matching or discrepancy evidence, addressing item-wise CoT’s limited discriminative supervision.
- A shared MLLM jointly learns independently indexable embeddings, pair-aware CoT generation, and discriminative ranking.
Supplementary Material · Implementation and Evaluation Details
The supplementary implementation and evaluation details consolidate UMER’s complete settings, covering multimodal training data, curated positives, retrieved hard negatives, and offline-audited pair-aware CoT targets.
- Implementation and Evaluation Details: The training mixture spans image, video, and visual-document sources.
- Implementation and Evaluation Details: Curated positives replace raw targets when such positives are available.
- Implementation and Evaluation Details: Each training instance is paired with a retrieved hard negative.
- Implementation and Evaluation Details: Retrieved hard negatives provide supervision for both contrastive and pairwise learning.
- Implementation and Evaluation Details: Pair-aware CoT targets are generated and audited offline before training.
- Implementation and Evaluation Details: UMER’s complete implementation and evaluation settings are consolidated in Table 1.
Pair-Aware CoT Data Construction
The pair-aware CoT data pipeline constructs supervision from query–positive and hard-negative pairs, generating structured evidence for each without revealing labels. It then audits the generated notes text-only for schema validity, label agreement, and decision leakage.
- Overview: Pair-aware rationales compare each query with a relevant target and a plausible distractor, teaching models which details distinguish the labeled target rather than describing queries alone.The pipeline obtains a semantically close non-relevant candidate and creates separate rationales for positive and negative pairs.
- Stage 1: hard-negative mining: Hard-negative mining masks annotated positives, searches source-local galleries by cosine similarity, filters likely missing positives, and retains up to three candidates.Reviewed non-relevant candidates are preferred when available; otherwise, the embedding-mined pool provides a fallback.
- Stage 2: structured CoT generation: Qwen3.5-9B independently annotates each pair with query intent and target observations, recording salient overlaps, missing evidence, or conflicts without issuing a match decision.The structured output uses two fields and includes relevant entities, attributes, actions, relations, or text cues in neutral wording.
- Stage 3: text-only evidence verification: A Qwen3.5-0.8B auditor verifies notes using only the generated fields, accepting traces only when predictions agree with labels, schemas are valid, and decisions do not leak.The verifier returns match, not_match, or unknown, and requires concrete supporting observations or conflicts; four sidecars contain 6,632,206 canonical pair records.
Complete MMEB-V2 Results
Across complete 78-task MMEB-V2 evaluations, UMER-H achieves the best reported averages for Qwen2-VL-2B and reaches strong results with Qwen2-VL-7B under the same K = 5 hybrid configuration.
- Qwen2-VL-2B: 65.5 overall, 70.4 image, 45.0 video, and 73.6 visual-document average make UMER-H best overall and 1.4 points ahead of RIME.These results cover every MMEB-V2 task for Qwen2-VL-2B models.
- Evaluation scope: The complete comparisons report 78 tasks, using Hit@1 for image and video and NDCG@5 for visual-document retrieval.Baselines are limited to models with published task-level results at the corresponding parameter scale.
- Qwen2-VL-7B: 70.6 overall and 74.3/50.5/80.0 on image/video/visual-document tasks are achieved by UMER-H with the same K = 5 hybrid configuration.The comparison covers Qwen2-VL-7B models with published 78-task breakdowns.
Further Inference Analysis
Further inference analysis shows that embedding retrieval and pair-aware ranking provide complementary decision signals, while hybrid inference remains robust across a broad ranking-weight range. The default fusion configuration therefore benefits from combining both branches without retraining.
- Weight sensitivity: 65.5 is the peak UMER-H score, and all ranking-score weights from 0.5–5.0 remain within 0.8 points of it under fixed candidates, normalization, and decoding.The default hybrid mode assigns ranking-score weight 2.0; larger weights produce a small decline, indicating neither branch should dominate every query.
- Decision overlap: The overlap analysis compares both branches on the same 83,530 MMEB-V2 queries using unchanged candidates and evaluation, separating Both, E only, R only, and Neither outcomes.Percentages are reported separately for reasoning/semantic and content-matching query families to preserve their different failure profiles.
- Decision overlap: 10.3% of queries are correct only with pair-aware ranking versus 9.0% only with embedding retrieval, while 47.2% are solved by both, demonstrating complementary branches.Across 83,530 MMEB-V2 queries, neither branch is uniformly dominant; ranking-only successes reach 12.1% for reasoning/semantic queries versus 9.0% embedding-only, while content-matching exclusives are 8.9% and 8.3%.
Qualitative Results and Failure Cases
Qualitative cases show pair-aware ranking correcting embedding errors by enforcing fine-grained composition, attribute, and visual-metaphor constraints. A spatial-relation case exposes a ranking regression where foreground salience overrides the requested background relation.
- Corrections: Cases 1–3 show pair-aware ranking correcting embedding retrieval by promoting evidence for exact compositional, attribute, visual, or document-level constraints.The relevant candidate was already retrieved, but embedding similarity overemphasized broad category or topic cues.
- Case 2: Attribute-sensitive composed retrieval: Rank 4 to rank 1: ranking replaces a visually similar underwater image with two ray-like animals and no human, satisfying conjunctive constraints.The embedding top result preserved the scene but violated the requested species and no-person requirements.
- Case 3: Visual-metaphor document retrieval: Rank 3 to rank 1: ranking replaces a related video slide with the requested sinking-car metaphor illustrating the consequence of blindly following data.The promoted slide combines the car-in-water icon with the stated metaphor, whereas the embedding result shares only the broad topic.
- Case 4: Spatial-relation ranking regression: Ranking regresses by promoting the foreground motorcycle instead of the requested motorcycle behind the red bike, although the correct candidate remains in the top-2.Candidate-specific reasoning focuses on foreground salience and fails to preserve the query’s behind/on-the-right spatial relation.