PaperlayerView exact revision on arXiv

Source-linked AI summary

MMSearch-R1: Incentivizing LMMs to Search

Jinming Wu, Zihao Deng, Wei Li, Yiding Liu, Bo You, Bo Li, Zejun Ma, Ziwei Liu

arXiv:2506.20670v1cs.CVcs.CL

TL;DR

LMMs struggle with knowledge-intensive and information-seeking VQA when complex, dynamic, or up-to-date knowledge is needed. MMSearch-R1 trains LMMs for on-demand multimodal search, outperforming same-sized RAG baselines while reducing search use.

  • Problem

    LMMs struggle on knowledge-intensive and information-seeking VQA when complex, dynamic, or post-training knowledge is needed, motivating access to external up-to-date knowledge.

    Source: §1 · p. 2

  • Method

    MMSearch-R1 trains LMMs with end-to-end reinforcement learning for on-demand multi-turn search, using image and text tools, a search-penalty reward, and search-required and search-free training samples.

    Source: §1 · pp. 2–3, §5 · p. 9

  • Results

    MMSearch-R1-7B outperforms same-sized RAG counterparts by an average of 3% in accuracy and reduces average search rate by 32.9%, while competing with a 32B RAG model.

    Source: §4.2 · p. 8

  • Takeaways & Limitations

    The framework, dataset, and empirical findings provide practical insights into training LMMs with real-world interaction capabilities.

    Source: §5 · p. 9

    The exact-string-match reward suits short factual questions with unambiguous answers but can penalize semantically correct phrasing and limit generalization to open-ended QA.

    Source: H Limitations · p. 25

Abstract

from arXiv · show

Robust deployment of large multimodal models (LMMs) in real-world scenarios requires access to external knowledge sources, given the complexity and dynamic nature of real-world information. Existing approaches such as retrieval-augmented generation (RAG) and prompt engineered search agents rely on rigid pipelines, often leading to inefficient or excessive search behaviors. We present MMSearch-R1, the first end-to-end reinforcement learning framework that enables LMMs to perform on-demand, multi-turn search in real-world Internet environments. Our framework integrates both image and text search tools, allowing the model to reason about when and how to invoke them guided by an outcome-based reward with a search penalty. To support training, We collect a multimodal search VQA dataset through a semi-automated pipeline that covers diverse visual and textual knowledge needs and curate a search-balanced subset with both search-required and search-free samples, which proves essential for shaping efficient and on-demand search behavior. Extensive experiments on knowledge-intensive and info-seeking VQA tasks show that our model not only outperforms RAG-based baselines of the same model size, but also matches the performance of a larger RAG-based model while reducing search calls by over 30%. We further analyze key empirical findings to offer actionable insights for advancing research in multimodal search.

1 Introduction

Static training struggles with dynamic, long-tail knowledge, while existing search approaches do not teach LMMs to search effectively on demand. MMSearch-R1 uses end-to-end reinforcement learning to train multimodal search in real-world internet environments.

Introduction

  • Static training misses dynamic or restricted long-tail knowledge, while RAG can over-retrieve from static corpora and prompt-engineered agents lack learned search behavior; prior end-to-end RL work mainly targets text search.These limitations motivate multimodal methods that teach on-demand search for real-world environments.Source: §1 · p. 2
  • MMSearch-R1 trains LMMs to decide when to search, what to search for, and how to reason over results, using GRPO with an outcome-based reward and search penalty without cold-start initialization.The reward encourages models to recognize their knowledge boundaries and search when necessary, including through multi-turn strategies.Source: §1 · pp. 2–3
  • The dataset construction method estimates familiarity with required visual and textual knowledge to create mixed search-required and search-free VQA samples, supplemented by manually annotated evaluation data.The test data spans diverse knowledge categories and difficulty levels.Source: §1 · p. 2
  • The real-world search pipeline integrates image search for related webpage thumbnails and titles with text search for precise queries and relevant webpages.Source: §1 · p. 3
  • Over 30% fewer search calls accompanied MMSearch-R1-7B’s stronger performance than same-size RAG baselines and competitive performance against a 32B RAG-based model.These results are reported for knowledge-intensive and information-seeking VQA tasks.Source: §1 · p. 3
  • The authors report extensive experiments to derive empirical insights and state that they will open-source the data and complete training framework.Source: §1 · p. 3

2 Building Iterative Multimodal Search-Integrated RL Framework

MMSearch-R1 integrates standard GRPO with image and text search in iterative rollouts, while a reward combining answer accuracy, a search penalty, and format compliance encourages on-demand search. Retrieved content is excluded from loss computation to prevent environment feedback from shaping gradient updates.

2.1 Group Relative Policy Optimization (GRPO)

  • Training uses standard GRPO, which estimates its baseline from group rewards rather than a value function, reducing computational burden.GRPO is a variant of PPO.Source: §2.1 · p. 3

2.2 Multimodal Search Tools

  • The toolkit pairs image search for visually matched webpages with text search that retrieves and summarizes page content relevant to the question.Image search returns five matched webpages as thumbnails and titles; text search returns five URLs, whose page content is fetched by Jina Reader and summarized by Qwen3-32B.Source: §2.2 · p. 3

2.3 Rollout with Multi-turn Multimodal Search

  • Rollouts iterate through reasoning and one action per turn, passing retrieved information into the next dialogue turn until an answer or turn limit.The model may invoke image search using <search><img></search> or generate a natural-language query within <text_search> tags; retrieved content is masked during loss computation.Source: §2.3 · pp. 3–4

2.4 Reward Modeling

  • A weighted reward combines exact-match accuracy, a penalty for correct answers reached using search, and a strict format score.The final reward is (1 − α) · Acc_Score · Search_Penalty + α · Format_Score; penalizing searched answers encourages the model to use internal knowledge first and search only when needed.Source: §2.4 · p. 4

3 Curating Search-balanced VQA Datasets

FactualVQA is designed to train and evaluate on-demand multimodal search, combining search-required and search-free questions with concise verifiable answers and broad knowledge and difficulty coverage.

3 Curating Search-balanced VQA Datasets

  • The dataset design covers visual- and textual-knowledge search needs, concise factual answers, and diverse knowledge categories and difficulty levels.Existing datasets often use familiar images and emphasize textual knowledge, while MMSearch [26] uses recent-news images but is too small to provide sufficient visual-knowledge training data.Source: §3 · p. 5
  • FactualVQA combines automated pipelines and manual annotation to provide multimodal search VQA data for both training and evaluation.The collection process includes VQA Collection and Search Balancing.Source: §3 · p. 5, §3.1 · p. 5

3.1 Training Dataset Construction

  • The collection yields 6,000 FVQA-auto-vc samples (5,400 training, 600 testing), 7,000 FVQA-auto-txt samples, and 800 FVQA-manual-train samples.For visual-concept data, MetaCLIP concepts spanning common to niche items guide web image–page retrieval and GPT-4o question generation and taxonomy classification; InfoSeek questions are mapped to that taxonomy, and annotators add factual real-user-style queries.Source: §3.1 · pp. 5–6
  • The 5,000-example FVQA-train set contains approximately 3,400 search-required and 1,600 search-free questions, classified using eight Qwen2.5-VL-Instruct-7B rollouts per question.Questions with eight failed rollouts were discarded; search-required labels distinguish image, text, or mixed search, while any correct answer without search yields a search-free label.Source: §3.1 · p. 6

3.2 Test Dataset Annotation

  • FVQA-test contains 1,800 manually verified or human-annotated examples, with 600 each from FVQA-auto-vc, the InfoSeek Human Split, and manual annotation.The FVQA-auto-vc portion does not overlap training and was manually checked; InfoSeek answers were manually annotated because the original human-labeled answers were unavailable.Source: §3.2 · p. 6

4 Experiments

MMSearch-R1 improves answer accuracy while using search more selectively than fixed-search RAG, and RL strengthens retrieval, internal-knowledge use, and data efficiency. Balanced training data and a search penalty help maintain selective search behavior.

4.1 Setups

  • Experiments train Qwen2.5-VL-7B-Instruct with up to two searches over three dialogue rounds and evaluate five VQA benchmarks, including three OOD sets, against direct-answer and fixed two-search RAG baselines.Accuracy is assessed by GPT-4o as judge, and search ratio is also recorded; MMSearch, SimpleVQA, and LiveVQA are the OOD test sets.Source: §4.1 · pp. 6–7

4.2 Findings

  • MMSearch-R1-7B exceeds same-size RAG counterparts by an average of 3% in accuracy, cuts average search rate by 32.9%, and performs competitively with RAG-based Qwen2.5-VL-32B.The comparison covers in-domain and out-of-domain test sets.Source: Table 1 · p. 7, §4.2 · p. 8
  • Under fixed image-then-text RAG search, MMSearch-R1-7B consistently improves over the base model across in-domain and out-of-domain tasks, with gains in query generation and retrieved-information extraction.Because both searches are executed for every question, this comparison isolates query generation and reasoning over retrieved content from decisions about whether to search.Source: Figure 4 · p. 8, §4.2 · p. 8
  • On InfoSeek and SimpleVQA, RL increases the proportion of answers correct without search, indicating stronger internal recall and reasoning and more selective use of external tools.The model shifts toward relying on its own knowledge when sufficient and reserving search for novel or long-tail queries.Source: Figure 4 · p. 8, §4.2 · p. 8
  • RL-trained Qwen2.5-VL-7B outperforms SFT across all downstream tasks despite using about half as much training data, and invokes search more often on information-seeking MMSearch and LiveVQA.RL uses 5k FVQA samples; the SFT set is formed from 13.2k examples and filtered to 8k after removing incorrect GPT-4o answers.Source: §4.2 · pp. 8–9, Figure 5 · p. 9
  • Removing either data balancing or the search penalty yields slightly higher rewards but drives search ratios toward 100%, whereas using both maintains comparable reward with a lower, more stable ratio.The comparison indicates that balanced data and the penalty support selective on-demand search.Source: Figure 5 · p. 9, §4.2 · p. 9

5 Conclusion … C Multimodal Search Tools

MMSearch-R1 is an RL-based framework that lets LMMs recognize knowledge gaps, selectively search images or text, and reason over retrieved content; it outperforms same-sized RAG baselines and approaches larger models with significantly fewer search calls. The accompanying sections contextualize external-knowledge methods, describe GRPO, and detail the multimodal search services.

5 Conclusion

  • MMSearch-R1 learns to identify knowledge gaps, selectively invoke image or text search, and reason over results, outperforming same-sized RAG baselines and approaching larger models with significantly fewer search calls.The paper presents the framework, dataset, and findings as groundwork for adaptive, interactive LMMs.Source: §5 · p. 9

A.1 Large Multimodal Models (LMMs)

  • LMMs’ static training knowledge can leave them unable to answer long-tail or newly emerging questions reliably, motivating retrieval and search tool use.The passage notes that these knowledge gaps can lead to hallucinations and undermine reliability in real-world applications.Source: §A.1 · p. 15

A.2 Large Models with External Knowledge Access

  • External-knowledge approaches primarily use retrieval-based methods or search-based methods to improve factual reliability and knowledge coverage.This distinction frames the related-work discussion that follows.Source: §A.2 · p. 15

A.2.1 Retrieval-Augmented Generation (RAG)

  • RAG retrieves information from external knowledge bases to ground generation, but its reliance on static corpora limits it in dynamic, open-ended settings.Visual retrieval methods extend this paradigm to multimodal knowledge, including MuRAG [8], REVEAL [22], RagVL [10], and VisRAG [64].Source: §A.2.1 · p. 15

A.2.2 Search-Augmented Approaches

  • Search-augmented approaches use web tools or prompted agents for dynamic information, but multimodal agents are typically not trained with supervision or reinforcement to use search reliably.Prior multimodal systems include AVIS [21], VSA [70], and MMSearch [26], which retrieve and synthesize visual and textual evidence.Source: §A.2.2 · p. 16

A.3 Reinforcement Learning-powered Search Agents

  • End-to-end RL has advanced text-based multiturn search agents, while suitable training data and RL frameworks for real-world multimodal search remain underexplored.The passage cites DeepResearcher [72], Search-R1 [27], and ReSearch [7] as text-focused efforts.Source: §A.3 · p. 16

B Group Relative Policy Optimization (GRPO) Algorithm

  • GRPO updates a policy from grouped rollouts using a reference-policy constraint and group-normalized rewards to form advantages.The policy ratio and hyperparameters control update stability and constraint strength; reward normalization mitigates variance across samples.Source: B Group Relative Policy Optimization… · p. 17

C Multimodal Search Tools

  • The multimodal search pipeline combines independent image and text services with parallel processing, caching, and monitoring to support training and inference.Image search returns up to five relevant thumbnail-and-title results; text search parses and summarizes up to five webpages per round, with layered caches, Redis-based LRU eviction, and distributed rate limiting.Source: C Multimodal Search Tools · p. 17

D Prompts

Appendix D documents prompts for generating grounded visual QA pairs, structuring multimodal search interactions, summarizing webpages, and evaluating answers. It also specifies the Direct Answer and RAG baseline workflows.

D.1 Prompts for FVQA-auto-ac VQA Generation

  • The FVQA-auto-vc prompt guides GPT-4o to create concise, grounded QA pairs from image–webpage pairs, focusing on visual concepts answerable from observable or webpage-associated information.The prompt details are referenced in Table 2.Source: §D.1 · p. 18

D.2 Prompts for RL Training

  • Training uses three context-specific prompts—at conversation start, after image search, and after text search—to structure tool interactions and align model actions with the intended workflow.The prompt details are referenced in Table 3.Source: §D.2 · p. 18, Table 3 · p. 19

D.3 Prompts for Webpage Summarization Model

  • The text-search pipeline’s webpage summarization prompt guides extraction and summarization based on the question, aiming for focused, concise content that reduces token consumption.The prompt is shown in Table 4.Source: §D.3 · pp. 19–20, Table 4 · p. 20

D.4 Prompt for Direct Answer Baseline

  • The Direct Answer baseline prompt asks the model to answer concisely using only the image’s visual content.The prompt template is shown in Table 5.Source: §D.4 · p. 20, Table 5 · p. 20

D.5 Prompts for RAG Workflow Baseline

  • The RAG baseline performs exactly two searches per example—image search followed by text search—and uses up to three conversation rounds, reaching the maximum allowed search steps in experiments.After image results, the model generates a text query; after text results, it generates the final answer.Source: §D.5 · p. 20

D.6 Prompts for LLM-as-Judge Evaluation

  • The evaluation prompt uses GPT-4o as an LLM judge to classify responses as correct or incorrect against ground truth, with judgments used to compute accuracy.The full prompt is shown in Table 7.Source: §D.6 · p. 20

E Details of Benchmark Datasets · F Details of Experimental Implementation

The benchmark details describe curated and sampled datasets for evaluating multimodal knowledge and information-seeking, with dataset-specific scope and filtering choices.

E Details of Benchmark Datasets

  • FVQA-test contains 1,800 manually verified or newly annotated examples from three sources and spans visual and textual knowledge categories.Its sources are 600 FVQA-auto-vc samples without training overlap, 600 InfoSeek Human Split examples with re-annotated answers, and 600 newly collected examples.Source: E Details of Benchmark Datasets · pp. 20–21
  • InfoSeek [9] uses human-authored templates for 300 Wikidata relations, filters unsupported question-answer pairs, and balances sampling across entities and relations.Its image-question-answer triplets are grounded in real-world knowledge and multimodal contexts.Source: E Details of Benchmark Datasets · p. 21
  • The evaluation samples 2,000 examples from InfoSeek’s test split because of the dataset’s large size.Source: E Details of Benchmark Datasets · p. 21
  • MMSearch [26] has 300 manually collected examples across 14 subdomains, including recent news and rare knowledge; evaluation uses its 171 visual questions.The news examples cover events from August 2024, while the knowledge examples target facts that current leading LMMs fail to answer.Source: E Details of Benchmark Datasets · p. 21
  • SimpleVQA [13] contains 2,025 examples, of which 1,013 English question-answer pairs are selected to avoid language-related confounds.Its factual examples come from post-2023 VQA datasets and expert-generated internet-search samples, with difficulty filtering and human-in-the-loop quality control.Source: E Details of Benchmark Datasets · pp. 21–22
  • LiveVQA [18] contains 3,602 pairs across 14 news categories, with GPT-4o-generated questions ranging from basic visual queries to multi-hop reasoning over article text.Its content comes from six globally recognized news platforms.Source: E Details of Benchmark Datasets · p. 22

F.1 RL Training Setting

MMSearch-R1 is trained with GRPO using Qwen2.5-VL-7B on a search-balanced FVQA-train set, with bounded tool use and a search penalty. The reported configuration specifies the training scale, optimization settings, and checkpoint used for evaluation.

F.1 RL Training Setting

  • Training uses Qwen2.5-VL-7B with GRPO via veRL on FVQA-train, comprising approximately 1,600 search-free and 3,400 search-required samples.Training runs on 4 nodes with 8 Nvidia H100 GPUs each; total batch size is 512 and mini-batch size is 128.Source: §F.1 · p. 23
  • Each prompt generates 8 rollouts with up to 3 tool calls per rollout; the search penalty and format-score weight are both 0.1.The learning rate is 2e-6, KL divergence coefficient β is 0.001, and clip ratio ϵ is 0.2.Source: §F.1 · p. 23
  • MMSearch-R1-7B downstream evaluation uses the step-50 checkpoint, identified as the point when training converges.Source: §F.1 · p. 23

F.2 SFT Training Setting … G.1 Evaluation on General VQA Benchmarks

MMSearch-R1-7B retains general VQA performance comparable to Qwen2.5-VL-7B, while the supplementary settings describe the SFT comparison model and inference and judging procedures. The general VQA evaluation covers six standard benchmarks and finds slight gains on two.

F.2 SFT Training Setting

  • The SFT comparison model is Qwen2.5-VL-7B fine-tuned on 8,000 GPT-4o-generated samples with up to three dialogue rounds.GPT-4o responses provide supervision; the first-turn user question and later tool-returned content are masked during loss computation.Source: §F.2 · p. 23
  • SFT uses one machine with 8 Nvidia H100 GPUs, per-device batch size 1, gradient accumulation over 2 steps, and 1 epoch of training.The learning rate is 1e-5 with a cosine scheduler and 10% warm-up.Source: §F.2 · p. 23

F.3 Evaluation Setting

  • Inference uses veRL with vLLM, top_p 1.0, temperature 0, and one generated response per test sample.The trainer's val_only parameter is set to True.Source: §F.3 · p. 23
  • GPT-4o-20241120 judges inference results using a predefined prompt, with top_p 0.1 and temperature 0.Its binary Yes/No output determines the final accuracy metric.Source: §F.3 · p. 23

G.1 Evaluation on General VQA Benchmarks

  • MMSearch-R1-7B performs comparably to Qwen2.5-VL-7B across six general VQA benchmarks, slightly outperforming it on LLaVA-Wilder and MathVista.Results are similar on AI2D, ChartQA, OCRBench, and MME; experiments use LMMS-Eval.Source: §G.1 · p. 23
  • The general VQA suite includes AI2D, ChartQA, LLaVA-Wilder, MathVista, MME, and OCRBench.The authors interpret the comparable results as evidence that reinforcement learning enhances search-related behavior while preserving general visual understanding and reasoning.Source: §G.1 · p. 23

G.2 Ablation Study on Reward Modeling

GPT-4o reward modeling improves accuracy over exact-match reward, suggesting richer semantic supervision can help, though it brings costs and limitations.

G.2 Ablation Study on Reward Modeling

  • 59.5% average accuracy for ∗4o is 3.8 points higher than ∗EM across the evaluated datasets.GPT-4o provides semantic-level accuracy judgments that can avoid exact-match false negatives, but may introduce bias, increase training cost, and struggle beyond its knowledge scope. The experiment used a FVQA-train subset with relatively few search-free examples, and its model converged to a higher average search rate than in Table 1.Source: §G.2 · p. 24

H Limitations · I Broader Impacts

MMSearch-R1’s limitations center on unstable multimodal search-tool interactions and an exact-match reward that may not generalize beyond short factual QA. The paper also notes risks from autonomous web retrieval and recommends safeguards for content quality and provenance.

H Limitations

  • 0.2% of image searches and 1% of text searches failed end-to-end during training, with partial failures increasing the rates.Failure meant receiving no valid results; partial failures included retrieving fewer than the expected top five results.Source: H Limitations · pp. 24–25
  • Image search submits the full image, while the multi-component text pipeline can vary or fail during ranking, extraction, and summarization.The pipeline uses SerpAPI, Jina Reader, and a summarization model; its summarizer may hallucinate relevant information.Source: H Limitations · p. 24
  • Exact string-match rewards suit short factual questions but may penalize semantically correct answers with different phrasing, limiting generalization to open-ended QA.A preliminary comparison found GPT-4o-based rewards more tolerant of answer variation than EM-based rewards.Source: H Limitations · p. 25
  • Improving tool-interaction robustness and reward expressiveness is essential to scaling MMSearch-R1 into more adaptive, reliable multimodal reasoning agents.The authors identify stability and consistency of tool outputs as a nontrivial challenge as training scales up.Source: H Limitations · p. 25

I Broader Impacts

  • Autonomous web retrieval and summarization may surface outdated, biased, or misleading information and raise concerns about verifiability, copyright, and misinformation.Over-reliance on real-time search may also contribute to inadvertent propagation of misinformation.Source: I Broader Impacts · p. 25
  • The authors recommend filtering retrieved content, attributing sources, and enabling users to trace and verify response provenance.These measures are proposed to mitigate risks associated with search-based assistants.Source: I Broader Impacts · p. 25
  • Future work should investigate domain whitelisting and adaptive uncertainty calibration, especially for high-stakes applications.These are proposed as safer retrieval strategies.Source: I Broader Impacts · p. 25

J Examples of FVQA Dataset · K Case Studies

The section illustrates FVQA’s fact-oriented, knowledge-intensive examples and uses MMSearch-R1 case studies to show its search behavior on complex, real-world information-seeking VQA tasks.

J Examples of FVQA Dataset

K Case Studies

Related papers

Loading 2506.20670v1…