PaperlayerView exact revision on arXiv

Source-linked AI summary

Learning from Failures: Retrieval-Centric CoT via Hard Negatives for Unified Multimodal Retrieval

Zelong Sun, Jun Wang, Kaicheng Yang, Tiancheng Gu, Ziyong Feng, Zhiwu Lu

arXiv:2608.06060v1cs.CV

TL;DR

Unified multimodal retrieval can miss fine-grained cues needed to distinguish semantically similar candidates. UniME-R1 uses retrieval feedback to diagnose confusion, refine queries, and consistently improve performance across multimodal benchmarks.

  • Problem

    Unified multimodal retrieval must distinguish candidates using fine-grained cues distributed across heterogeneous inputs, but raw-input encoders can miss this evidence.

  • Method

    UniME-R1 uses an embedder–adviser framework that analyzes initially retrieved candidates, generates retrieval-centric CoT, and adaptively reranks or re-retrieves.

  • Results

    UniME-R1 consistently improves multimodal retrieval across MMEB-V2 and diverse general tasks, reaching 69.9 (2B) and 70.3 (4B) on MMEB-V2.

  • Takeaways & Limitations

    Retrieval feedback is an effective basis for multimodal retrieval reasoning across image, video, and visual-document tasks.

  • Takeaways & Limitations

    Applying retrieval-centric reasoning indiscriminately can introduce redundant or noisy semantics when initial retrieval is already reliable.

Abstract

from arXiv · show

Unified multimodal retrieval aims to identify candidates that satisfy complex user intent expressed through heterogeneous inputs. Although Large Vision-Language Model (LVLM)-based retrievers are efficient and scalable, directly encoding raw multimodal inputs often misses fine-grained discriminative cues, leading to confusion among semantically similar candidates. Recent methods mitigate this limitation by generating Chain-of-Thought (CoT) rationales to enrich the query representation. However, such reasoning is typically derived from the query alone: it explains what the query describes, but not what the retriever misunderstands. We argue that effective retrieval reasoning should instead be conditioned on retrieval feedback. Based on this insight, we introduce UniME-R1, an embedder-adviser framework that learns to reason over initially retrieved candidates and generate Retrieval-Centric Chain-of-Thought (RC-CoT). The adviser analyzes candidates individually to identify the discriminative cues confused by the embedder. If the target appears in the initial top-k set, UniME-R1 directly reranks the candidates; otherwise, it generates RC-CoT to refine the retrieval direction and performs full-corpus re-retrieval with a dual-mode embedder. To train the framework, we mine hard negatives to simulate realistic retrieval failures, jointly optimize direct retrieval and RC-CoT-augmented retrieval, and align the adviser with retrieval outcomes through supervised learning and retrieval-oriented reinforcement learning. Extensive experiments on MMEB-V2 and a diverse set of general multimodal retrieval benchmarks demonstrate that UniME-R1 consistently improves retrieval performance over strong baselines.

1 Introduction

Unified multimodal retrieval must distinguish fine-grained cues across heterogeneous inputs, but query-only reasoning cannot diagnose the retriever’s model-specific confusions. UniME-R1 addresses this with retrieval-feedback-conditioned RC-CoT and an adaptive rerank-or-retrieve framework.

  • Unified multimodal retrieval must resolve user intent across text, images, videos, and visual documents using fine-grained attributes, relations, events, and document details.
  • Existing Reasoner–Embedder methods generate captions, expansions, or CoT from the query alone, enriching semantics without observing actual retrieval outputs (Lan et al., 2025; Cui et al., 2025; Jiang et al., 2026).This can miss the specific distinctions causing retrieval confusion and may add redundant or noisy semantics when initial retrieval is reliable.
  • RC-CoT analyzes initially retrieved candidates to diagnose embedder-specific confusion and produce targeted attributes, relations, events, or temporal cues that correct retrieval direction.Retrieved candidates expose which semantically similar results are plausible yet insufficient, including events presented in the wrong temporal order.
  • UniME-R1 uses a dual-mode embedder and retrieval-aware adviser to adaptively rerank the initial top-k set when a match is present or perform RC-CoT-enhanced full-corpus re-retrieval otherwise.The framework predicts whether a match exists among current candidates, independently ranks them, and retains a reusable candidate index.
  • Hard negatives jointly strengthen embedder contrastive learning, create realistic adviser-failure contexts, and support retrieval-oriented rewards, with adviser training followed by structured supervision and GRPO optimization (Shao et al., 2024).Rewards assess format validity, path judgment, candidate ranking, and whether RC-CoT improves re-retrieval.

2 Related Work

Related work develops shared multimodal representations and stronger LVLM reasoning. Universal multimodal embedding builds on cross-modal contrastive learning while addressing dual-encoder modality gaps, and recent LVLM methods use CoT prompting and reinforcement learning to improve reasoning.

  • Universal Multimodal Embedding: Universal multimodal embedding maps diverse modalities into a shared feature space while learning semantically meaningful representations.CLIP (Radford et al., 2021) is an early representative that achieves strong image-text retrieval through large-scale cross-modal contrastive learning.
  • Universal Multimodal Embedding: CLIP’s dual-encoder architecture inevitably suffers from a modality gap (Gu et al., 2025).
  • LVLM Reasoning: LVLM reasoning methods use CoT prompting for step-by-step rationales and reinforcement learning to optimize reasoning trajectories across diverse tasks.DeepEyes (Zheng et al., 2025) relies solely on reinforcement learning to encourage thinking with images and achieves strong reasoning performance.

3 UniME-R1

UniME-R1 combines a dual-mode multimodal embedder with a retrieval-aware adviser that diagnoses initial retrieval failures using candidate feedback. It adaptively reranks existing candidates or performs RC-CoT-enhanced full-corpus re-retrieval, supported by hard-negative-based joint contrastive training and adviser optimization.

  • Architecture and inference: When a target is already in the initial top-k set, UniME-R1 reranks candidates; otherwise, it appends RC-CoT to the query and searches the full corpus.The separate rerank_judge and rerank_list fields control path selection and candidate ordering, respectively.
  • Architecture and inference: UniME-R1 uses a dual-mode embedder and retrieval-aware adviser to analyze initial top-k candidates and adaptively choose reranking or RC-CoT-enhanced re-retrieval.The adviser predicts candidate relevance, ordering, and whether a matching candidate exists in the current top-k set.
  • Retrieval failure diagnosis: RC-CoT conditions reasoning on retrieved candidates to identify weakly encoded intent cues and move the query representation away from observed confusions.Its cot_focus field specifies the needed retrieval adjustment, while cot_answer expresses the corrected intent for the embedder.
  • Representation design: Candidate representations are always encoded through dis_emb, so they can be precomputed and indexed once while both query modes use the same candidate index.The embedder constructs dedicated discriminative and RC-CoT-enhanced query representations for initial retrieval and re-retrieval.
  • Training: Hard negatives strengthen direct and RC-CoT-enhanced retrieval training while creating realistic retrieval-failure contexts for adviser supervision.They are mined by retrieving candidates, filtering potential false negatives using similarity gaps, and verifying relevance with an LVLM judge.
  • Training: After supervised fine-tuning, GRPO freezes the embedder and rewards output format, path decisions, candidate ranking, and RC-CoT retrieval effectiveness.The RC-CoT reward favors refined queries that rank the target above in-batch and mined hard negatives while increasing its similarity margin over confusing candidates.

4 Experiment

UniME-R1 achieves state-of-the-art performance on MMEB-V2 at both model scales and improves general multimodal retrieval across multiple benchmarks. Its evaluation uses MMEB-V2-derived training data and compares against representative embedder-only and reasoner–embedder baselines.

  • Experimental Setup: Training derives all UniME-R1 data from MMEB-V2, including 1.73M RC-CoT-augmented samples for <gen_emb>, 643K SFT samples, and 13K GRPO examples for the adviser.The dual-mode embedder trains <dis_emb> on original MMEB-V2 data and <gen_emb> on RC-CoT-augmented data.
  • Baselines: UniME-R1 is compared with representative embedder-only methods and reasoner–embedder methods, including UME-R1, TTE, and Embed-RL.The embedder-only baselines include VLM2Vec, VLM2Vec-V2, GME, ColPali, CAFe, B3, Unite, LamRA, and RzenEmbed-v1.
  • MMEB-V2 Results: 69.9 (2B) and 70.3 (4B) are UniME-R1’s best overall MMEB-V2 scores, surpassing the strongest baselines by 3.1 and 1.4 points, respectively.Against UME-R1 and TTE, gains are 9.8 and 6.8 points for 2B, and 5.8 and 1.7 points for 4B.
  • General Multimodal Retrieval: UniME-R1-2B improves over UniME-V2-7B by 3.1&1.1, 7.3&3.2, 2.7&2.8, and 2.3&1.6 points on Flickr30K, COCO, ShareGPT4V, and Urban1K, while UniME-R1-4B gains 1.6 points over Embed-RL-4B on UVRB.These consistent gains indicate that retrieval-feedback RC-CoT exploits both concise visual cues and richer descriptions.

5 Ablation Study

Ablations show that retrieval feedback, adaptive routing, dual-mode training, and retrieval-oriented reinforcement learning each materially improve UniME-R1. Actual top-k feedback and complete training produce the strongest gains, while the components address complementary retrieval failures.

  • RC-CoT Format: Actual top-k feedback yields 68.5 overall, exceeding query-only CoT at 65.6 and random-candidate conditioning at 66.5 by 2.9 and 2.0 points.The comparison isolates retrieval feedback under a fixed adviser, generation budget, and dual-mode embedder.
  • Inference Strategy: Adaptive routing reaches 69.9 overall, outperforming always reranking at 69.1 and always re-retrieving at 69.0 by 0.8 and 0.9 points.Reranking recovers targets already in the top-k, whereas re-retrieval corrects failed retrieval directions.
  • Dual-Mode Embedder: The complete embedder improves 1.2 points overall over discriminative-only training, as Lgen enables RC-CoT-conditioned representations and hard negatives sharpen both embedding modes.Adding Lgen alone raises the overall score from 62.3 to 62.6, while video and visual-document performance increase by 0.7 and 0.9 points; hard negatives add 1.0, 0.6, and 1.0 points on image, video, and visual-document tasks.
  • Reinforcement Learning: Full GRPO improves the SFT adviser from 68.8 to 69.9 overall, while removing judge, rerank, or RC-CoT rewards reduces performance by 0.4, 0.4, or 0.3 points.The full model obtains the best image, video, and overall results.

6 Analysis

UniME-R1’s retrieval-oriented reinforcement learning calibrates routing, while query-only RC-CoT generation substantially reduces candidate-side latency. Iterative inference further improves retrieval, especially for video queries with fine-grained temporal ambiguity.

  • Routing calibration: GRPO reduces the routing–GT@Top-5 gap from 17.3 to 5.3 points on image tasks and from 18.7 to 4.7 points on video tasks.On VisDoc, the gap instead increases from 2.4 to 5.7 points, indicating mild over-routing to reranking.
  • Inference efficiency: 0.01 s per candidate for UniME-R1 versus 0.27 s for Embed-RL, making candidate processing 27× faster.UniME-R1 generates RC-CoT only for queries and encodes each candidate once, whereas Embed-RL generates CoT for both queries and candidates.
  • Iterative retrieval feedback: 69.9 to 70.5 overall score after scaling test time, with gains of 0.3, 1.3, and 0.5 points on image, video, and visual-document tasks, respectively.The largest gain occurs on video retrieval, where an additional round better resolves fine-grained temporal ambiguity.

7 Conclusion

UniME-R1 is an embedder–adviser framework that generates Retrieval-Centric Chain-of-Thought from initial retrieval feedback. It diagnoses candidate confusion to adaptively rerank or perform RC-CoT-enhanced re-retrieval, while hard-negative training and retrieval-oriented reinforcement learning align both components with retrieval outcomes.

  • 7 Conclusion: UniME-R1 generates Retrieval-Centric Chain-of-Thought (RC-CoT) from initial retrieval feedback by diagnosing confusion exposed by retrieved candidates.
  • 7 Conclusion: The framework adaptively reranks current candidates or performs RC-CoT-enhanced re-retrieval with a reusable index.
  • 7 Conclusion: Hard-negative training and retrieval-oriented reinforcement learning align UniME-R1’s embedder and adviser with retrieval outcomes.

A Detailed Dataset Construction · A.1 Hard Negative Mining

UniME-R1 constructs hard negatives by combining embedder retrieval with LVLM-based soft relevance judgments. Candidates that are challenging yet incorrect are retained for contrastive learning, teacher annotation, and retrieval-oriented rewards.

  • A.1 Hard Negative Mining: Hard negatives begin with top-ranked candidates retrieved by Qwen3-VL-Embedder 8B for each query–positive pair.The mining process operates over a query–positive pair (q, c+) and the candidate corpus C.
  • A.1 Hard Negative Mining: Candidates whose embedder similarity is too close to or exceeds the positive are removed using threshold τh to reduce false negatives.The filtering condition is s(q, c) − s(q, c+) > τh.
  • A.1 Hard Negative Mining: An LVLM judge evaluates whether each remaining candidate satisfies the query and must output exactly “Yes” or “No.”The decision rules reject candidates with missing attributes, wrong objects, or complete irrelevance.
  • A.1 Hard Negative Mining: Rather than using decoded judgments, the method defines soft relevance as logit(yes) − logit(no) for each query–candidate pair.The target receives the same score, a(q, c+), enabling comparison against candidate relevance.
  • A.1 Hard Negative Mining: Candidates ranked below the target by LVLM relevance, satisfying a(q, c) < a(q, c+), form the challenging-but-incorrect hard-negative pool H(q).This criterion preserves candidates that remain difficult for the judge while not satisfying the query as well as the target.
  • A.1 Hard Negative Mining: The resulting hard-negative pool is shared across contrastive learning, teacher annotation, and retrieval-oriented rewards.Table 8 reports statistics of the training data.

A.2 Data Sources and Modality-Balanced Sampling … B.1 Effect of Retrieved Context Size

The paper balances multimodal training data, mitigates false negatives and unreliable reasoning supervision, and shows that broader retrieved context improves retrieval-oriented reasoning, reaching a best overall score of 70.5. Ablations further examine candidate-wise analysis and context size across image, video, and visual-document tasks.

  • A.2 Data Sources and Modality-Balanced Sampling: The multimodal retrieval corpus combines LLaVA-Hound video-language data, ViDoRe and VisRAG visual-document data, and MMEB-train image-task data.Together, these sources cover images, videos, visual documents, and diverse retrieval tasks.
  • A.2 Data Sources and Modality-Balanced Sampling: Modality-aware stratified sampling caps each image dataset at 50,000, visual-document dataset at 100,000, and video dataset at 300,000 examples before filtering.Sampling within every modality and constituent dataset preserves broad task coverage instead of letting large video datasets dominate.
  • A.3 Merging Image Classification Datasets: Merging N24News, HatefulMemes, VOC2007, and SUN397 before batch construction increases semantic diversity and reduces false negatives in contrastive learning.The original query-target correspondences and dataset-specific task instructions are preserved.
  • A.4 Data Assignment for the Dual-Mode Embedder: The discriminative embedder is trained on all sampled examples, whereas the generative embedder receives contrastive supervision only from examples with valid, quality-filtered RC-CoT.This preserves broad coverage for <dis_emb> while preventing low-quality reasoning traces from degrading <gen_emb>; the SFT data contain 643,080 structured examples.
  • A.5 RL Data Sampling: Equidistant sampling within difficult sub-datasets controls their contributions to the RL set, exposing GRPO to diverse retrieval failures rather than dominant sources.The procedure addresses substantial size differences among challenging sub-datasets.
  • B Additional Ablation Studies: The ablations include candidate-wise analysis in <rerank_think> on MMEB-V2.Table 10 isolates the effect of candidate-wise analysis.
  • B.1 Effect of Retrieved Context Size: 70.5 is the best overall score when the adviser sees Top-7 retrieved candidates, while image performance saturates at Top-5 and video and VisDoc continue improving.Expanding context from Top-1 to Top-3 improves the overall score by 3.2 points and video performance by 6.7 points.

B.2 Effect of Candidate-Wise Reranking Analysis · C Detailed Routing Behavior Before and After GRPO

Candidate-wise reranking analysis materially improves retrieval, especially on video tasks, while GRPO changes routing toward the GT@Top-5 pattern across most task groups. These findings show the value of explicit candidate comparison and reveal how routing behavior shifts after GRPO.

  • B.2 Effect of Candidate-Wise Reranking Analysis: The <rerank_think> field requires relevance analysis for each retrieved candidate before predicting candidate order and the routing decision.This design tests candidate-wise reasoning as an intermediate step rather than only as an output format.
  • B.2 Effect of Candidate-Wise Reranking Analysis: 69.9 versus 67.3: removing candidate-wise analysis lowers the overall score by 2.6 points.The comparison removes <rerank_think> while retaining the adviser’s other outputs.
  • B.2 Effect of Candidate-Wise Reranking Analysis: The full model gains 1.8 image points, 0.9 visual-document points, and 6.4 video points over the no-analysis variant.The larger video improvement indicates that explicit candidate comparison is particularly useful for fine-grained temporal and event-level differences.
  • C Detailed Routing Behavior Before and After GRPO: GRPO raises the reranking rate for every task group, but the increase represents a policy shift rather than an improvement by itself.Table 11 provides the task-level routing statistics underlying Figure 4.
  • C Detailed Routing Behavior Before and After GRPO: In nine of twelve task groups, GRPO routing frequency moves closer to GT@Top-5; Video-CLS, Video-RET, and VisDoc-VDRv1 are exceptions.The comparison concerns routing frequency relative to the GT@Top-5 reference.
  • C Detailed Routing Behavior Before and After GRPO: At the modality level, GRPO substantially reduces the aggregate gap on image and video tasks.The supplied passage contrasts this with movement in the VisDoc modality but does not specify its direction fully.

D Gains After RL · E Comprehensive Performance on Video Retrieval · E.1 UVRB Evaluation Metrics

RL improves UniME-R1 most when retrieval-failure diagnosis and RC-CoT query refinement are enabled, while the UVRB evaluation measures video retrieval across tasks, domains, and sub-domains using standardized aggregate metrics.

  • D Gains After RL: RL produces its largest improvement in RC-CoT-only inference, gaining 1.2 overall points, 1.8 image points, and 1.7 video points.This pattern indicates that RL primarily strengthens retrieval-failure diagnosis and query refinement.
  • D Gains After RL: Rerank-only inference improves by 0.7 overall points but slightly declines on video and VisDoc, showing that RL does not uniformly improve candidate ordering.The full adaptive model reaches 69.9 under the combined inference configuration.
  • E.1 UVRB Evaluation Metrics: UVRB evaluates video retrieval across 16 datasets organized into non-overlapping Tasks, Domains, and Sub-domains categories.The benchmark uses unweighted arithmetic means to compare heterogeneous datasets fairly.
  • E.1 UVRB Evaluation Metrics: UVRB reports R@1 by default, R@10 for CMRB and LoVR-TH, and P@1 for the multi-positive MS-TI and MS-TV datasets.These metric choices follow the benchmark protocol.
  • E.1 UVRB Evaluation Metrics: The overall AVG in Table 2 averages the three task groups and three domain groups arithmetically rather than averaging directly over datasets.The groups are TXT, CMP, VIS and CG, FG, LC, respectively.

E.2 Per-Dataset Results … H Prompt for Inference

UniME-R1 improves per-dataset video retrieval over corresponding Embed-RL models, with especially large gains on composed retrieval. Qualitative cases show that candidate-wise RC-CoT and RL-trained adviser judgments provide targeted retrieval refinement and better query-intent alignment, using the illustrated inference prompt.

  • E.2 Per-Dataset Results: 58.2 and 58.7 average scores with the 2B and 4B embedders outperform corresponding Embed-RL models by 1.0 and 1.3 points across 16 datasets.On composed retrieval, UniME-R1-2B reaches 50.5 on MS-TI and 43.8 on MS-TV, versus 19.3 and 21.0 for Embed-RL-2B.
  • F Qualitative Case Studies: Candidate comparisons reveal corrective cues absent from the query, such as the target containing no solid object, and guide re-retrieval toward the relevant white object.Figure 6 illustrates how encoding these cues in RC-CoT provides a more targeted retrieval direction.
  • G Comparative Examples of Adviser Before and After RL: RL changes the adviser’s RC-CoT from broad and incidental reasoning to shorter, more precise reasoning centered on retrieval-critical evidence.The comparison uses the same inputs for SFT and RL advisers.
  • G Comparative Examples of Adviser Before and After RL: RL improves reranking intent alignment by distinguishing general passenger seating from the purpose-oriented phrase to ride directly.This distinction produces improved alignment between candidate relevance judgments and query intent.
  • H Prompt for Inference: The inference process uses a dedicated Adviser prompt, illustrated with the query “Recognize the breakfast type that the person is cooking in the video.”Figure 10 presents the prompt used during inference.
  • F Qualitative Case Studies: Candidate-level relevance analysis separates the ground-truth target from a strong distractor, yielding a more accurate ranking through adviser-guided reranking.Figure 7 demonstrates the reranking path on a candidate that partially matches the queried relation.
Loading 2608.06060v1…