PaperlayerView exact revision on arXiv

Source-linked AI summary

DeepMMSearch-R1: Empowering Multimodal LLMs in Multimodal Web Search

Kartik Narayan, Yang Xu, Tian Cao, Kavya Nerella, Vishal M. Patel, Navid Shiee, Peter Grasch, Chao Jia, Yinfei Yang, Zhe Gan

arXiv:2510.12801v1cs.CVcs.IR

TL;DR

Knowledge-intensive multimodal VQA requires current external information, but existing retrieval and search approaches use rigid or limited tool interactions. DeepMMSearch-R1 addresses this with on-demand multi-turn text and image search, cropped-image retrieval, dynamic query refinement, and two-stage SFT-plus-RL training. It surpasses RAG and prompt-based search-agent baselines while remaining competitive with OpenAI o3.

  • Problem

    Knowledge-intensive visual question answering requires background knowledge beyond static model training data, while existing multimodal search methods have limited or rigid tool interactions.

  • Method

    DeepMMSearch-R1 combines dynamic multi-turn text and image search, question-focused image cropping, DeepMMSearchVQA, supervised finetuning, and online GRPO reinforcement learning.

  • Results

    DeepMMSearch-R1-7B (RL) surpasses RAG workflows by 21.13 points and prompt-based search agents by 8.89 points, while achieving competitive performance with OpenAI o3.

  • Takeaways & Limitations

    Training multimodal LLMs to decide when and how to use search tools improves benchmark performance over fixed retrieval and test-time prompting.

  • Takeaways & Limitations

    Web-retrieved content can be biased, outdated, or misleading, while automatic summarization may amplify misinformation or create copyright concerns.

Abstract

from arXiv · show

Multimodal Large Language Models (MLLMs) in real-world applications require access to external knowledge sources and must remain responsive to the dynamic and ever-changing real-world information in order to address information-seeking and knowledge-intensive user queries. Existing approaches, such as retrieval augmented generation (RAG) methods, search agents, and search equipped MLLMs, often suffer from rigid pipelines, excessive search calls, and poorly constructed search queries, which result in inefficiencies and suboptimal outcomes. To address these limitations, we present DeepMMSearch-R1, the first multimodal LLM capable of performing on-demand, multi-turn web searches and dynamically crafting queries for both image and text search tools. Specifically, DeepMMSearch-R1 can initiate web searches based on relevant crops of the input image making the image search more effective, and can iteratively adapt text search queries based on retrieved information, thereby enabling self-reflection and self-correction. Our approach relies on a two-stage training pipeline: a cold start supervised finetuning phase followed by an online reinforcement learning optimization. For training, we introduce DeepMMSearchVQA, a novel multimodal VQA dataset created through an automated pipeline intermixed with real-world information from web search tools. This dataset contains diverse, multi-hop queries that integrate textual and visual information, teaching the model when to search, what to search for, which search tool to use and how to reason over the retrieved information. We conduct extensive experiments across a range of knowledge-intensive benchmarks to demonstrate the superiority of our approach. Finally, we analyze the results and provide insights that are valuable for advancing multimodal web-search.

1 Introduction

DeepMMSearch-R1 addresses limitations in multimodal web search by combining on-demand multi-turn retrieval, dynamic query refinement, and cropped image search. It is trained with DeepMMSearchVQA and a two-stage SFT-plus-RL pipeline, achieving state-of-the-art performance over previous open-source baselines.

  • Motivation: Prior methods are limited by rigid retrieval, weak interaction with noisy web results, text-only search, or single-call tool use.Whole-image search can also be distracted by background content and unrelated visual entities, reducing retrieval quality.
  • Proposed approach: DeepMMSearch-R1 enables on-demand, multi-turn searches with dynamic text queries and cropped image search to address noisy multimodal retrieval.It iteratively refines text queries through self-reflection and self-correction, while Grounding DINO selects a question-relevant image region before image search.
  • Training resources: DeepMMSearchVQA provides diverse multi-hop conversations that teach when and what to search, which tool to use, and how to reason over retrieved content.The dataset includes both search-required and search-free questions with multi-turn interactions.
  • Training: The approach uses cold-start supervised finetuning followed by online reinforcement learning with GRPO to improve web-search tool use.The paper presents this two-stage process as the basis for its reported performance improvement.

2 Proposed Data: DeepMMSearchVQA

DeepMMSearchVQA is an automatically constructed multimodal VQA dataset designed to train web-search tool use. Its multi-turn examples combine reasoning, structured tool calls, and web-retrieved information across balanced knowledge categories.

  • Dataset design: DeepMMSearchVQA contains diverse multi-hop conversations covering search-free and search-required questions across a broad knowledge taxonomy.The construction principles emphasize diversity and multiple conversational turns for reasoning, self-reflection, and self-correction.
  • Tool annotations: The dataset uses structured tags for image, cropped-image, and text search, enabling examples to record tool choice and query construction.Multiple refined text searches capture self-reflection and self-correction behavior.
  • Generation pipeline: The generation pipeline feeds question-image pairs to Gemini, validates generated answers, and invokes search tools when the action tag requires external information.Retrieved top-k results are summarized and returned to Gemini for subsequent reasoning turns.
  • Dataset statistics: The final 10,000-example corpus records knowledge categories, conversational-turn counts, and the proportions requiring image, text, or both search types.These statistics are presented in Figure 2(bottom).

3 DeepMMSearch-R1 Training Recipe

DeepMMSearch-R1 is trained in two stages: supervised finetuning teaches multimodal tool use and web-retrieval reasoning, while online GRPO refines tool selection and search behavior. Its training combines multi-turn rollouts, masked web content, and rewards for factual accuracy and structural compliance.

  • 3.1 Supervised Finetuning Stage: The two-stage recipe first uses supervised finetuning, then online GRPO to refine tool selection and search efficiency.The initial stage provides grounding, image-search, and text-search capabilities; GRPO further optimizes search behavior.
  • 3.1 Supervised Finetuning Stage: Supervised training predicts each token in multi-turn reasoning traces, tool calls, and final answers from the multimodal question and image.Web-retrieved information is masked during loss computation so learning focuses on reasoning and structured tool calls.
  • 3.2 Reinforcement Learning Stage: GRPO compares candidate rollouts for the same prompt using advantages centered around the group’s mean reward.This group-relative centering removes dependence on absolute reward scale and targets responses better than the group average.
  • 3.2 Reinforcement Learning Stage: The GRPO objective uses a clipped importance-weighted surrogate with KL regularization against a frozen reference model.The clipping range and KL term are used to stabilize optimization under noisy reward signals.
  • 3.2 Reinforcement Learning Stage: Rollouts use grounding, image-search, and text-search tools interactively, continuing until a final response or the maximum turn limit.The model chooses full-image or cropped-image search when it cannot confidently identify a visual entity.
  • 3.2 Reinforcement Learning Stage: The composite reward combines binary factual correctness with format compliance for valid tags and tool-call structure.The reward is Rtotal = (1 − λfmt) s + λfmt sfmt, balancing answer correctness and schema adherence.

4 Experiments

Experiments evaluate DeepMMSearch-R1 against multimodal-search baselines, analyze its search and tool-use behavior, and test design choices affecting performance. The results show gains from trained tool use, iterative search, cropped image search, and RL-refined tool selection while preserving general VQA capabilities.

  • Main Results: DeepMMSearch-R1-7B (RL) surpasses RAG workflows by 21.13 points and prompt-based search agents by 8.89 points, while remaining competitive with OpenAI o3.The comparison uses multiple baseline categories and shows the largest performance boost for the web-search-equipped model.
  • Ablation Analysis: Cropped image search improves performance by an average of 1.75 points across six datasets, while self-reflection and self-correction enable iterative query refinement.Cropped search reduces background noise, especially for questions about a single visual entity rather than an entire scene.
  • Ablation Analysis: Search-balanced SFT data improves behavior by avoiding excessive search on datasets where most questions can be answered without retrieval.High proportions of search-required examples induce excessive search and poorer performance on OKVQA and A-OKVQA.
  • Tool-Use Analysis: Tool use matches dataset demands, with tools used on 87.7% of DynVQA samples versus 43.5% of OKVQA samples.DynVQA contains more questions requiring external information, whereas OKVQA requires fewer search calls.
  • Tool-Use Analysis: RL increases query refinement by 1.54% on DynVQA and 2.64% on OKVQA while reducing cropped-image searches by 36.81% and 34.86%, respectively.The model becomes more selective, cropping only when necessary while still achieving overall performance gains.
  • General VQA Evaluation: SFT with LoRA and online GRPO with a KL penalty maintains performance on general VQA and reasoning benchmarks.The authors attribute this preservation to limited parameter updates and regularized policy changes.

5 Conclusion

DeepMMSearch-R1 integrates on-demand, multi-turn web search into multimodal question answering through iterative query refinement and cropped image search. Its two-stage SFT-and-RL approach outperforms prior baselines across six benchmarks.

  • DeepMMSearch-R1 outperforms prior baselines across six benchmarks.
  • The model performs on-demand, multi-turn web searches with dynamic query refinement, self-reflection, and self-correction.
  • Cropped image search complements iterative text-query refinement by addressing limitations of prior multimodal retrieval methods.
  • Training uses supervised finetuning with DeepMMSearchVQA followed by online GRPO reinforcement learning to improve tool-use efficiency and effectiveness.

Ethics Statement

The paper highlights ethical and deployment risks of real-time web-search capabilities, including unreliable retrieved content, misinformation amplification, copyright concerns, and infrastructure dependence. It also situates these risks within prior search and multimodal-LLM approaches.

  • Retrieved web content may be biased, outdated, or misleading, while automatic summarization can amplify misinformation and create copyright concerns.
  • Dependence on external infrastructure may limit accessibility for resource-constrained institutions.
  • The paper encourages source attribution, content filtering, and human oversight for high-stakes deployments.
  • Prior search agents and search-equipped MLLMs remain limited in handling noisy web information, generalizing beyond pretraining, or supporting image search.

C Datasets

The paper draws on diverse visual-question-answering datasets spanning information seeking, multimodal search, dynamic retrieval, factuality, commonsense, and knowledge-based reasoning. These datasets vary substantially in scale, language, domain, and external-knowledge requirements.

  • InfoSeek contains 8,900 human-written pairs and 1.35 million automatically generated QA triplets across unseen entities and questions.
  • FVQA combines visual-knowledge, text-knowledge, and manually annotated training examples for multimodal search VQA.
  • Encyclopedic VQA provides 221,000 question–answer pairs linked to up to five images and Wikipedia evidence.
  • SimpleVQA, OKVQA, and A-OKVQA cover factuality, external knowledge, commonsense, and rationale-based visual question answering.
  • DynVQA evaluates adaptive retrieval for changing, multimodal-knowledge, and multi-hop questions across English and Chinese.

D Implementation Details

Implementation uses Qwen2.5-VL-7B-Instruct with LoRA-based supervised finetuning and constrained tool interaction. Image search is limited to one call, while text search can be invoked multiple times within a rollout budget.

  • The base model is Qwen2.5-VL-7B-Instruct, finetuned with rank-8 LoRA across all target modules.
  • Supervised finetuning runs for 3 epochs at a 1e-4 learning rate with cosine scheduling, 0.1 warmup, and bf16 precision.
  • Image or cropped-image search can be called once, whereas text search can be called multiple times, with total tool calls capped at 10 per rollout.
  • Generated outputs are optimized by masking input tokens, with responses limited to 8192 tokens.

E Prompts

The prompts define how the model analyzes images, chooses direct answers or image/text searches, iterates over retrieved information, and produces final answers. Additional prompts specify image and text summarization plus semantic and numerical answer evaluation.

  • Initial Prompt: The initial prompt asks the model to analyze the image and question, then choose direct answering, image search, or text search.Direct answers require sufficient visual identification and internal knowledge; otherwise the model selects a search action.
  • Initial Prompt: Cropped image search is preferred for specific visual elements, while whole-image search is reserved for scene-level questions.The prompt gives examples including faces and logos for cropped searches and requires a special whole-image format.
  • Initial Prompt: After one image search, the model may perform multiple text searches and reassess subsequent results before answering.Search results are returned in later turns inside information tags, enabling iterative text-search decisions.
  • E.2 WebSearch Equipped MLLMs evaluation Prompt: The evaluation prompt uses a reverse-image-search summary to help formulate one focused text query for facts needed to answer the image question.It supplies the question, image, and a text summary identifying the main visual subject.
  • Final Answer Prompt: The final-answer prompt requires reasoning over text-search results followed by a concise answer, or a fixed insufficiency response.Reasoning is enclosed in reason tags, while the answer is enclosed in answer tags.
  • E.4 Prompt-based Search Agent Prompt: The prompt-based search agent uses the same prompts as the supervised-fine-tuning data-generation process.This equivalence is stated explicitly for the search-agent prompt.
  • LLM-as-judge Prompt: The LLM judge accepts semantically equivalent answers and alternate ground-truth answers, while ignoring trivial formatting differences.The judge treats synonyms, paraphrases, aliases, punctuation, word order, casing, and articles as non-substantive differences.
  • GPT-4o as reward model prompt: The reward evaluation allows numeric range inclusion, reasonable rounding, and unit conversion but rejects substantive entity or value differences.It outputs a Boolean match and concise reason; examples distinguish valid numeric flexibility from mismatched values.

F.1 Introduction Figure

Figure F.1 presents an image of a boat race.

  • F.1 Introduction Figure: Figure F.1 depicts a boat race.The caption identifies the scene as a boat race.
  • F.1 Introduction Figure: The figure provides visual content rather than a reported quantitative comparison.No axes, values, or comparative outcome are specified.
  • F.1 Introduction Figure: Its stated scope is limited to identifying the pictured event.The passage names the event but supplies no additional context.

G DeepMMSearchVQA Samples

Figures G.1 and G.2 are presented as sample examples from the DeepMMSearchVQA dataset.

  • G DeepMMSearchVQA Samples: Figure G.1 shows a sample in DeepMMSearchVQA.The passage identifies the figure as a dataset sample without describing its contents.
  • G DeepMMSearchVQA Samples: Together, the figures document sample instances from DeepMMSearchVQA rather than a reported comparison.Neither passage specifies axes, values, or an outcome comparison.
  • G DeepMMSearchVQA Samples: Figure G.2 shows another sample in DeepMMSearchVQA.The passage identifies the figure as a dataset sample without describing its contents.
Loading 2510.12801v1…