PaperlayerView exact revision on arXiv

Source-linked AI summary

V-Retrver: Evidence-Driven Agentic Reasoning for Universal Multimodal Retrieval

Dongyang Chen, Chaoyang Wang, Dezhao Su, Xi Xiao, Zeyu Zhang, Jing Xiong, Qing Li, Yuzhang Shang, Shichao Kan

arXiv:2602.06034v2cs.CV

TL;DR

Existing multimodal retrieval methods rely on static visual representations or language-only reasoning, limiting fine-grained visual verification. V-Retrver enables agentic visual inspection during reranking and consistently improves retrieval across benchmarks, reaching 69.7% average Recall on M-BEIR.

  • Problem

    Existing retrieval methods rely on fixed visual representations or language-only reranking, limiting reliable use of fine-grained visual evidence in ambiguous cases.

  • Method

    V-Retrver combines coarse-to-fine candidate retrieval with an agentic reranker that iteratively invokes visual tools, reasons, and revises rankings.

  • Results

    Across M-BEIR and out-of-domain datasets, V-Retrver consistently outperforms strong baselines, achieving 69.7% average Recall on M-BEIR, +4.9% over U-MARVEL-7B (64.8%).

  • Takeaways & Limitations

    The results support interleaved visual reasoning as an effective approach for improving retrieval accuracy, perception-grounded reasoning, and generalization.

  • Takeaways & Limitations

    Its visual toolset is limited to image selection and zoom-in operations, which may be insufficient for object-level manipulation or multi-step spatial analysis.

Abstract

from arXiv · show

Multimodal Large Language Models (MLLMs) have recently been applied to universal multimodal retrieval, where Chain-of-Thought (CoT) reasoning improves candidate reranking. However, existing approaches remain largely language-driven, relying on static visual encodings and lacking the ability to actively verify fine-grained visual evidence, which often leads to speculative reasoning in visually ambiguous cases. We propose V-Retrver, an evidence-driven retrieval framework that reformulates multimodal retrieval as an agentic reasoning process grounded in visual inspection. V-Retrver enables an MLLM to selectively acquire visual evidence during reasoning via external visual tools, performing a multimodal interleaved reasoning process that alternates between hypothesis generation and targeted visual verification.To train such an evidence-gathering retrieval agent, we adopt a curriculum-based learning strategy combining supervised reasoning activation, rejection-based refinement, and reinforcement learning with an evidence-aligned objective. Experiments across multiple multimodal retrieval benchmarks demonstrate consistent improvements in retrieval accuracy (with 23.0% improvements on average), perception-driven reasoning reliability, and generalization.

1. Introduction

V-Retrver reframes universal multimodal retrieval as evidence-driven agentic reasoning, enabling MLLMs to verify fine-grained visual evidence instead of relying only on static representations and language. A curriculum-based training strategy aligns reasoning, ranking, and visual-tool use, yielding stronger accuracy, reliability, and generalization.

  • Motivation: Existing MLLM retrieval methods rely on fixed visual embeddings or textual descriptions, making visually ambiguous fine-grained distinctions difficult to verify.This limitation is especially pronounced when candidate images share semantic content but differ in appearance, style, or local context.
  • Method: V-Retrver enables MLLMs to selectively acquire visual evidence through external tools while alternating hypothesis generation with targeted visual verification.The framework uses multimodal interleaved Chain-of-Thought reasoning to ground ranking decisions in inspected candidate images.
  • Training: A three-stage curriculum combines supervised cold-start training, rejection-sampling refinement, and evidence-aligned reinforcement learning for reasoning and tool-use alignment.The stages initialize reasoning and operation formatting, consolidate high-quality trajectories, and align retrieval performance with visual-tool usage.
  • Results: Experiments on M-BEIR and out-of-domain datasets show consistent gains in retrieval accuracy, perception-grounded reasoning reliability, and generalization across diverse settings.These results support the effectiveness of interleaved visual reasoning for universal multimodal retrieval.

2. Related Work

Recent MLLMs integrate visual perception with language reasoning through high-performing open-source and holistic vision-language models. Multimodal retrieval has also advanced across text–image, composed-image, and multimodal-document tasks, while some models still lack Chain-of-Thought reasoning and test-time scalability.

  • Multi-modal Large Language Models: MLLMs have driven deep integration of visual perception and language reasoning, producing high-performing models such as LLaVA, Qwen-VL, and InternVL.The passage identifies these as notable open-source models.
  • Multi-modal Large Language Models: Holistic vision-language models, including GPT-4V, incorporate mixture-of-experts architectures and image-generation components but generally lack Chain-of-Thought and test-time scalability.These limitations are attributed to the cited model line of work in the passage.
  • Multimodal Retrieval: Deep-learning advances have propelled multimodal retrieval across text–image cross-modal retrieval, composed image retrieval, and multimodal document retrieval.The passage lists these as broad retrieval-task categories supported by recent advances.

3. Method

V-Retrver reformulates universal multimodal retrieval as coarse-to-fine, evidence-grounded reasoning, combining embedding-based candidate proposal with agentic reranking and targeted visual inspection. Its MIER paradigm and three-stage curriculum train the agent to acquire evidence selectively, improve ranking reliability, and reduce speculative reasoning.

  • Retrieval Framework: V-Retrver first retrieves top-K candidates using shared-space embedding similarity, then applies an agentic reasoning model for fine-grained reranking over the reduced candidate set.This coarse-to-fine pipeline separates efficient candidate proposal from computationally intensive evidence-based reasoning.
  • Multimodal Interleaved Evidence Reasoning: MIER interleaves textual hypothesis generation with on-demand visual evidence acquisition, allowing the agent to revise reasoning and terminate with a final candidate ranking.Tool requests and returned visual observations are appended to the reasoning context as an interleaved multimodal trajectory.
  • Multimodal Interleaved Evidence Reasoning: MIER explicitly grounds intermediate reasoning in dynamically acquired visual evidence, mitigating speculative inference and hallucination in visually ambiguous cases.The approach supports more reliable ranking decisions by linking reasoning steps to visual observations obtained on demand.
  • Visual Tools: The visual interface provides SELECT-IMAGE for choosing similar candidates and ZOOM-IN for inspecting localized discriminative attributes such as objects, textures, and spatial configurations.These tools let the agent control what to observe and where to focus rather than encoding all visual information upfront.
  • Curriculum-Based Training: The curriculum progresses from supervised reasoning activation, through rejection fine-tuning for reliable trajectories, to EAPO for evidence-aligned reinforcement learning.EAPO evaluates ranking quality and evidence utilization, encouraging visual inspection when it supports correct decisions while discouraging redundant tool use.
  • Curriculum-Based Training: EAPO teaches the agent not only what to rank, but also how and when to acquire visual evidence for reliable and efficient retrieval decisions.Its evidence acquisition behavior rewards inspection that contributes to correct decisions and penalizes redundant or excessive tool usage.

4. Experiments

V-Retrver is evaluated across M-BEIR, unseen datasets, and held-out retrieval tasks, where it achieves strong retrieval performance and generalization. Ablations show that staged training and visual-tool integration substantially improve evidence-grounded reasoning and recall.

  • Performance on M-BEIR: 69.7% average Recall establishes V-Retrver-7B as state-of-the-art on M-BEIR, improving 4.9 percentage points over U-MARVEL-7B’s 64.8%.On fine-grained visual-detail tasks, V-Retrver reaches 51.2% on FIQ and 73.5% on CIRR, versus 38.2% and 63.2% for U-MARVEL-7B.
  • Robustness on Held-out Tasks: 61.1% average Recall on held-out tasks surpasses LamRA-7B’s 50.9% by 10.2 percentage points without exposure to the excluded formats.The results support adaptability across modality combinations and the use of interleaved evidence in zero-shot retrieval.
  • Impact of Training Stages: 45.8% performance without SFT, RSFT, and RL falls below Qwen2.5-VL-7B’s 47.2%, while SFT alone raises average recall to 59.4%.These ablations indicate that zero-shot tool invocation without alignment is ineffective and that staged training activates basic tool-use ability.
  • Effectiveness of Visual Tool: 67.2% average recall with visual tools exceeds the text-only RL variant’s 61.8%, showing that visual inspection supplies information unavailable from static representations alone.Both variants use the same training dataset and RL framework, isolating the effect of tool use.
  • RL Training Dynamics: RL training increases retrieval accuracy while reducing redundant tool use, response length, and tool frequency as the agent learns to seek visual evidence selectively.The training dynamics indicate improved perception-driven reasoning, stricter tool-protocol execution, and greater focus on resolving critical visual ambiguities.

5. Conclusion · A. Prompt Template

V-Retrver is presented as an evidence-driven MLLM framework for universal multimodal retrieval that interleaves reasoning with visual inspection to verify candidate images and refine ranking. The paper also provides prompt templates for system, user, and CoT annotation processes used in training and inference.

  • 5. Conclusion: V-Retrver uses multimodal interleaved CoT reasoning to dynamically inspect and verify candidate images through visual tool invocation.This evidence-driven process supports finer-grained ranking of candidate result lists.
  • 5. Conclusion: The framework is trained through a three-stage pipeline designed to develop multimodal interleaved CoT reasoning abilities.
  • A.1. System Prompt: The system prompt template is used for both training and inference.
  • A.1. System Prompt: Figure 4 presents the system prompt template for training and inference.
  • A.2. User Prompt: The user prompt template is used for both training and inference.
  • A.2. User Prompt: Figure 5 presents the user prompt template for training and inference.
  • A.3. Annotation Prompt: The annotation prompt guides CoT annotation by being inserted into the user prompt during generation.This prompt is specifically used for the CoT annotation process.
  • A.3. Annotation Prompt: Figure 6 presents the annotation prompt template used in the CoT annotation process.

B. Details about M-BEIR Dataset

M-BEIR is summarized in Table 7 and applies additional processing to its component datasets, causing differences from their standard evaluations. For example, including training data in CIRR’s candidate pool makes evaluation more difficult than on the original dataset.

  • M-BEIR applies additional processing to incorporated datasets, which can produce differences from standard individual-dataset evaluations.
  • Including training data in CIRR’s M-BEIR candidate pool increases evaluation difficulty compared with the original CIRR dataset.
  • Table 7 summarizes the M-BEIR benchmarks.

C. Details about Unseen Dataset

The unseen datasets include adaptations of MSCOCO and FashionIQ whose captions or query formats differ substantially from the originals. These format differences motivate treating them as unseen, as illustrated by CIRCO’s reference-image-plus-relative-caption queries.

  • Dataset construction: Many unseen datasets are adapted from MSCOCO or FashionIQ but use significantly different captions or query formats.These differences distinguish the adapted datasets from their source benchmarks.
  • Dataset construction: The paper treats adapted datasets as unseen because their modified captions or query formats create substantial differences from the original datasets.The passage explicitly notes this rationale for unseen-dataset classification.
  • Dataset construction: CIRCO combines a reference image with a relative caption, creating a query format substantially different from the original COCO dataset.CIRCO exemplifies how query-format changes produce a disparity from the source benchmark.

D. Exploration of RAG Applications

The framework’s practical utility is evaluated in Retrieval-Augmented Generation scenarios across three knowledge-based visual question answering benchmarks. Retrieval and VQA are trained simultaneously to align agentic visual reasoning with downstream generation needs.

  • The evaluation extends the framework to Retrieval-Augmented Generation scenarios to assess its practical utility.
  • The method is evaluated on three Knowledge-based Visual Question Answering benchmarks following LamRA’s experimental setup.
  • Retrieval and VQA tasks are trained simultaneously so the agentic visual reasoning process aligns with downstream generation needs.
  • Table 9 presents a comparison of RAG capabilities on KVQA tasks.

E. Algorithms and Detailed Analysis · E.1. Inference Process

This merged section introduces V-Retrver’s formal inference and training algorithms, followed by complexity analysis. Its inference procedure uses a coarse-to-fine pipeline with sliding-window agentic reasoning, as specified in Algorithm 1.

  • E. Algorithms and Detailed Analysis: The section presents formal algorithms for V-Retrver’s inference and training processes.These algorithms are followed by a complexity analysis.
  • E. Algorithms and Detailed Analysis: The section covers both inference and training procedures rather than inference alone.Its stated scope includes formal algorithms for both processes.
  • E. Algorithms and Detailed Analysis: A complexity analysis follows the presentation of the formal algorithms.The section introduction explicitly places complexity analysis after the inference and training algorithms.
  • E.1. Inference Process: V-Retrver’s inference process is organized as a coarse-to-fine pipeline.The supplied passage identifies this pipeline as the framework for inference.
  • E.1. Inference Process: The inference pipeline incorporates sliding-window agentic reasoning.This reasoning structure is part of the detailed inference process.
  • E.1. Inference Process: Algorithm 1 details the inference process of V-Retrver.The passage explicitly identifies Algorithm 1 as the location of the inference-process specification.

E.2. Training Pipeline · F. Qualitative Examples

V-Retrver uses a three-stage curriculum to train an evidence-driven retrieval policy, while its inference pipeline combines coarse embedding retrieval with agentic reranking and visual-tool verification. Qualitative examples show accurate retrieval through fine-grained, structured reasoning.

  • E.2. Training Pipeline: The training pipeline progressively aligns the model with evidence-driven retrieval objectives through a three-stage curriculum.The curriculum is presented in Algorithm 2.
  • E.2. Training Pipeline: Inference first computes cosine similarities between the query and all candidates, then selects the top-K candidates for agentic reranking.Candidates are ranked using sn = cos(Φ(q), Φ(cn)) before selecting Ctop.
  • E.2. Training Pipeline: The reranking agent processes candidate windows sequentially while maintaining an initialized context for each window.The windows use size W and stride S, and each window starts with H0 ← (q, wj, Instruction).
  • E.2. Training Pipeline: When the agent emits a tool action, the system executes the visual tool and appends its observations and output to the reasoning context.The context update is Ht+1 ← Ht ⊕ ot ⊕ vobs.
  • E.2. Training Pipeline: After receiving an answer, the system parses the local ranking for each window and updates the global ranking list.The local rank list r̂j is added to Lglobal before the window loop continues.
  • E.2. Training Pipeline: Training begins with supervised reasoning activation, filters synthesized Chain-of-Thought data for format compliance, and refines the model using valid, format-correct and rank-correct trajectories.The first stage minimizes LSFT, while rejection sampling retains trajectories satisfying IsFormatValid(τi) ∧ IsRankCorrect(τi).
  • E.2. Training Pipeline: The final stage applies evidence-aligned policy optimization with rewards combining format, ranking, and tool-use terms, followed by group-normalized advantage estimation and GRPO updates.Rewards are R(oi) = αrfmt(oi) + βrrank(oi) + rtool(oi).
  • F. Qualitative Examples: Qualitative examples demonstrate accurate retrieval through fine-grained and structured reasoning, illustrating the effectiveness of V-Retrver.The examples are presented in Fig. 7 through Fig. 11.

G. Limitations and Future Works

V-Retrver’s limitations concern both its restricted visual tools and its reliance on synthesized trajectories and curated rewards, which may reduce robustness in complex or diverse real-world settings.

  • The current visual toolset supports only image selection and zoom-in, limiting complex reasoning requiring object-level manipulation or multi-step spatial analysis.
  • Training uses synthesized reasoning trajectories and curated rewards, potentially introducing bias and limiting robustness under diverse or noisy real-world conditions.
  • The paper identifies these constraints as remaining limitations despite V-Retrver’s strong performance.
Loading 2602.06034v2…