Source-linked AI summary
Reason to Contrast: A Cascaded Multimodal Retrieval Framework
Xuanming Cui, Hong-You Chen, Hao Yu, Hao Yuan, Zihao Wang, Shlok Kumar Mishra, Hanchao Yu, Yonghuan Yang, Jun Xiao, Ser-Nam Lim, Jianpeng Cheng, Qi Guo, Xiangjun Fan
TL;DR
Multimodal retrieval traditionally scales through embedding models, while TTE shows that reasoning tokens can improve embeddings. TTE-v2 adds query-aware reranking and reranker feedback, reaching state-of-the-art MMEB-V2 performance with test-time token scaling.
Problem
TTE restricts reasoning to query or candidate sides independently, limiting joint query–candidate understanding and fine-grained ranking.
Method
TTE-v2 cascades initial retrieval with Query-Aware Reasoning, ECR-based reranking, and reranker-guided hard-negative mining and false-negative filtering.
Results
75.7% overall accuracy makes TTE-v2 state of the art on MMEB-V2, while its 2B variant surpasses previous 7B models trained with extensive external data.
Takeaways & Limitations
TTE-v2 demonstrates token-wise test-time scaling as an alternative to increasing model or embedding size for multimodal retrieval.
Takeaways & Limitations
The reported MMEB-V2 evaluation excludes the ViDoSeek-page and MMLongBench-page datasets because of dataset errors.
Abstract
from arXiv · showhide
Traditional multimodal retrieval systems rely primarily on bi-encoder architectures, where performance is closely tied to embedding dimensionality. Recent work, Think-Then-Embed (TTE), shows that incorporating multimodal reasoning to elicit additional informative tokens before embedding can further improve retrieval. In this paper, we extend this paradigm with TTE-v2, a hybrid multimodal retrieval framework that introduces reasoning-driven performance scaling based on additional input token budget rather than model or embedding size. Our approach augments the initial multimodal retrieval with additional reasoning steps for reranking, enabling more expressive query-candidate interactions at test time. The reranking stage further provides fine-grained supervision for hard negative mining and false negative filtering, creating a feedback loop that effectively strengthens the upstream retriever. This cascaded design delivers substantial test-time improvements based on intermediate reasoning token scaling. Experiments on the MMEB-V2 benchmark demonstrate that TTE-v2-7B achieves a new state-of-the-art accuracy of 75.7%, and that TTE-v2-2B matches or surpasses leading 7B models trained with significantly larger external data. Our results highlight the promise of token-wise scaling as an alternative scaling paradigm for multimodal retrieval.
1 Introduction
Multimodal retrieval has moved from independent embedding toward reasoning-enhanced and cascaded designs that add query-aware reranking and feedback. TTE-v2 combines these ideas to scale test-time reasoning and improve retrieval quality.
- Background: TTE adds reasoning tokens before embedding, enabling test-time scaling through reasoning budget rather than only encoder parameters or embedding dimensionality.This reasoning step produces Embedding-Centric Reasoning that conditions the embedder for retrieval.
- Reason-to-Contrast: TTE-v2 introduces a cascaded framework that adds query-aware reasoning and ECR-based reranking to test-time multimodal retrieval.Its design extends TTE beyond independent query or candidate reasoning by incorporating joint query–candidate information.
- Reason-to-Contrast: ECRR reuses candidate reasoning for lightweight text-only reranking, while QAR generates query-aware reasoning that supplies more discriminative pairwise information.ECRR requires no additional reasoning calls, whereas QAR performs joint reasoning over queries and retrieved candidates.
- Contributions: Reranker judgments are fed back into training to mine hard negatives, filter false negatives, and distill improved supervision into the retriever.The feedback loop is motivated by stronger joint query–candidate matching than the underlying embedder provides.
- Contributions: Over 10% absolute improvement in zero-shot retrieval is reported over the strong TTE baseline through QAR and ECRR.The two stages provide additional reasoning and reranking beyond the initial embedding-based retrieval.
- Results: 75.7% overall accuracy establishes TTE-v2 as state of the art on MMEB-V2, outperforming proprietary models trained on substantially larger private datasets.In video retrieval, the method improves over the previous best by 6%, while TTE-v2-2B surpasses earlier 7B models trained with extensive external data.
2 Related Work
Related work spans universal multimodal embedding, cascaded retrieval and reranking, and automated hard-negative mining. TTE-v2 differs by using pre-generated reasoning for lightweight reranking and reranker feedback.
- Universal Multimodal Retrieval: Universal multimodal retrieval unifies heterogeneous retrieval tasks under a shared representation space, including visual question answering, classification, grounding, and document retrieval.These tasks require embedding models to interpret task instructions when producing representations.
- Multimodal Embedding Models: Recent multimodal embedding models use MLLMs with contrastive training and improve performance through architecture, hard-negative mining, synthetic data, or auxiliary objectives.This work builds on that MLLM-based embedding direction while adding cascaded reasoning and reranking.
- Cascaded Multimodal Retrieval and Re-ranking: Cascaded retrieval commonly pairs a fast encoder for broad candidate retrieval with a slower, more expressive reranker for refinement.Earlier multimodal rerankers often rely on computationally expensive query–target fusion.
- Cascaded Multimodal Retrieval and Re-ranking: ECRR uses pre-generated Embedding-Centric Reasoning for text-only reranking, making it lighter than multimodal reranking while achieving better performance.The approach avoids additional multimodal reranking calls after ECRs have been generated.
- Hard Negative Mining: Hard-negative mining methods typically select difficult samples from embedding nearest neighbors, but contrastive datasets often lack annotated hard negatives.False-negative filtering is therefore an additional concern in automated mining approaches.
3 Preliminary
The preliminary formulation represents queries and targets with a shared MLLM encoder and trains their pooled embeddings using a weighted contrastive objective. TTE additionally supplies embedding-centric reasoning before encoding.
- MLLM-Based Universal Multimodal Retrieval: Queries and targets independently pass through the same MLLM encoder, whose hidden-state sequence is pooled into a final embedding.The paper adopts the final token’s hidden state as the pooled representation.
- MLLM-Based Universal Multimodal Retrieval: Training uses a uni-directional InfoNCE objective in which each query treats its matched target as the positive sample.The batch contains N matched query–target pairs.
- MLLM-Based Universal Multimodal Retrieval: The contrastive loss incorporates extra hard negatives weighted by w_k, with cosine similarity scaled by temperature τ.K denotes the number of extra hard negatives for each training sample.
- Think Then Embed: TTE adds Embedding-Centric Reasoning before final embedding generation to bridge asymmetric gaps between query and target representations.Each input combines visual content, text, task instruction, and an ECR sequence that can describe the referred object or provide task-specific reasoning.
4 Reason-to-Contrast
Reason-to-Contrast extends TTE with a cascaded coarse-to-fine pipeline that reranks retrieved candidates using ECRs, rewrites them with query-aware reasoning, and feeds reranking judgments back into hard-negative mining.
- Framework: TTE-v2 combines ECR-based reranking, Query-Aware Rewriting, and reranking-feedback hard-negative mining in a cascaded framework.The framework applies these components after initial retrieval to refine candidate ordering and training supervision.
- Motivation: Video retrieval remains difficult because TTE often finds relevant items within top-k candidates but does not consistently rank the best item first.The reported top-1 versus top-k gap is especially pronounced for video tasks, whose content contains more information than the query.
- ECR-based Reranking: ECR-based reranking reuses embedding-centric reasoning traces, with pairwise scoring or listwise reasoning over the query and retrieved candidate traces.Pairwise reranking assigns a query–candidate score, while listwise reranking jointly processes the query and top-k ECRs.
- Query-Aware ECR: Query-Aware Rewriting generates candidate ECRs that explicitly consider the query, adding discriminative details for reranking and hard-negative separation.The rewritten ECR can expose query-matching details absent from an over-general original ECR.
- Hard Negative Mining: Reranking feedback improves negative supervision by filtering false negatives and using reranker scores to select or weight hard negatives.The procedure first retrieves a candidate pool, then removes candidates ranked above the ground truth and scores the remaining samples by hardness.
5 Experiments
TTE-v2 is evaluated on MMEB-V2 across multimodal retrieval tasks, with reranking, query-aware reasoning, and hard-negative mining producing gains over baselines.
- Video Retrieval: +5.4% for TTE-v2-2B over the previous best model establishes consistent state-of-the-art performance across video retrieval tasks.The evaluation covers MMEB-V2 video retrieval tasks, with Table 1 presenting the main comparison.
- Multilingual Image Retrieval: +16.1% absolute Recall@1 on CrossModal-3600 shows that 2B ECRR improves out-of-distribution multilingual image retrieval.Both ECRR and QAR improve over baselines on CrossModal-3600 and XTD10.
- Full MMEB-V2 Benchmark: 75.7 overall score on MMEB-V2 makes TTE-v2-7B the strongest reported model, while TTE-v2-2B ranks third and improves over TTEt by 2.7%.TTE-v2-7B exceeds IFM-TTE-7B at 75.1; the 7B improvement over TTEt is 3.2%.
- Effect of Reranking Methods: ECRR pairwise raises TTE-2B performance on DiDeMo from 37.2 to 47.9, while listwise ECRR reaches 47.1.Both variants provide gains exceeding 10%, with larger improvements for the smaller 2B model.
- Effect of Query-Aware Reranking: QAR improves difficult video retrieval tasks, exceeding 10% gains for TTE-7B on QVHighlight and Charades-STA and over 5% on VATEX.The gains target cases requiring complex joint reasoning between query and target.
- Effect of ECRR-based Hard Negative Mining: ECRR-based hard-negative mining achieves the highest overall performance and reduces false negatives relative to naive top-k and embedder-based filtering.Random negatives can harm accuracy, while B3 provides marginal gains below 0.2%.
6 Conclusion
TTE-v2 combines QAR, ECRR, and ECRR-based hard-negative mining for coarse-to-fine retrieval and achieves state-of-the-art MMEB-V2 performance without additional data.
- Conclusion: TTE-v2 achieves state-of-the-art performance on MMEB-V2 without relying on additional data.The framework combines test-time reranking with hard-negative mining and false-negative filtering to strengthen retrieval.
- Conclusion: QAR and ECRR improve retrieval in a coarse-to-fine manner, while ECRR-based hard-negative mining distills reranking knowledge into the embedding model.The conclusion presents these components as the framework’s main contributions.
A More ablations
Additional ablations show that ECRR remains effective with weaker ECR generators, whereas QAR benefits substantially from stronger reasoning models and carefully chosen candidate pools.
- Effect of ECR quality on reranking: 36.8 overall with Qwen2.5-VL 32B ECRs and ECRR exceeds 27.0 from zero-shot reranking with Qwen2.5-VL 72B.ECRR outperforms MLLM-based zero-shot reranking across all tested ECR generators.
- Effectiveness of Query-Aware Reasoning: More capable QAR models improve performance: Gemini 2.5 Pro exceeds ECRR by more than 7% on VATEX.Qwen2.5-VL 32B can underperform ECRR, while Qwen2.5-VL 72B begins to surpass it.
- Reranking performance on reranker size: Larger rerankers and larger top-k pools generally improve ECRR, but top-k scaling can hurt with a 0.6B reranker or when extra candidates introduce distractors.QVHighlight degrades when top-k increases from 5 to 10 because its top-5 accuracy is already high.
B Qualitative examples
The qualitative analysis emphasizes that QAR adds query-relevant discriminative details, while reranking behavior depends on candidate-pool size and reranker capacity.
- Reranking behavior: ECRR’s top-k and reranker-size behavior should be read together: larger pools help stronger rerankers but can add distractors for weaker ones.The reported trend is especially relevant when interpreting the Figure 9 ablation.
- Qualitative examples: QAR supplements an accurate but overgeneral video description with details aligned to the query, such as people walking alongside the main character.The example shows QAR recovering caption content omitted by the original ECR.
C Implementation Details
The implementation uses Qwen3 as a text-only pairwise reranker for ECRR, with prompts specified across the accompanying figures and an MLLM reranking baseline for comparison.
- Reranking: Qwen3 reranker assigns a scalar matching score to each query–target ECR pair in ECRR.When query-side ECRs exist, the reranker receives the query ECR instead of the original query text.
- Reranking: MLLM-based zero-shot reranking uses an off-the-shelf multimodal model to judge whether each candidate video matches the query.The baseline passes query text and candidate video to the MLLM and prompts for a yes-or-no decision.
- Reranking: Figure 10 provides the Qwen3 reranker prompt template used for ECRR.