PaperlayerView exact revision on arXiv

Source-linked AI summary

MMSearch-R1: Incentivizing LMMs to Search

Jinming Wu, Zihao Deng, Wei Li, Yiding Liu, Bo You, Bo Li, Zejun Ma, Ziwei Liu

arXiv:2506.20670v1cs.CVcs.CL

TL;DR

LMMs struggle with dynamic and long-tail knowledge, while existing search approaches can encourage rigid or excessive retrieval. MMSearch-R1 trains LMMs with end-to-end reinforcement learning, multimodal search tools, and search-balanced data, outperforming same-sized RAG baselines and reducing search calls by over 30%.

  • Problem

    Static training leaves LMMs limited on dynamic, long-tail, or otherwise inaccessible knowledge, motivating external search for knowledge-intensive VQA.

  • Method

    MMSearch-R1 uses end-to-end reinforcement learning with modified GRPO, image and text search tools, and search-balanced VQA data containing search-required and search-free samples.

  • Results

    MMSearch-R1-7B outperforms same-sized RAG baselines, performs competitively with a larger 32B RAG-based model, and reduces search calls by over 30%.

  • Takeaways & Limitations

    The framework supports adaptive, interactive multimodal agents that recognize knowledge gaps, selectively search, and reason over retrieved content.

  • Takeaways & Limitations

    The search pipeline has stability and quality limitations, including variable rankings, webpage extraction failures, and possible summarizer hallucinations.

Abstract

from arXiv · show

Robust deployment of large multimodal models (LMMs) in real-world scenarios requires access to external knowledge sources, given the complexity and dynamic nature of real-world information. Existing approaches such as retrieval-augmented generation (RAG) and prompt engineered search agents rely on rigid pipelines, often leading to inefficient or excessive search behaviors. We present MMSearch-R1, the first end-to-end reinforcement learning framework that enables LMMs to perform on-demand, multi-turn search in real-world Internet environments. Our framework integrates both image and text search tools, allowing the model to reason about when and how to invoke them guided by an outcome-based reward with a search penalty. To support training, We collect a multimodal search VQA dataset through a semi-automated pipeline that covers diverse visual and textual knowledge needs and curate a search-balanced subset with both search-required and search-free samples, which proves essential for shaping efficient and on-demand search behavior. Extensive experiments on knowledge-intensive and info-seeking VQA tasks show that our model not only outperforms RAG-based baselines of the same model size, but also matches the performance of a larger RAG-based model while reducing search calls by over 30%. We further analyze key empirical findings to offer actionable insights for advancing research in multimodal search.

1 Introduction

MMSearch-R1 addresses the difficulty of answering knowledge-intensive VQA questions requiring dynamic or long-tail external knowledge by training LMMs to search on demand. It combines multimodal search, balanced data, and outcome-based reinforcement learning to improve performance while reducing unnecessary search.

  • Motivation: Static visual-text training struggles with dynamic, long-tail, private, copyrighted, or security-constrained knowledge, increasing hallucination risks on knowledge-intensive VQA.These limitations arise when questions require information beyond the model’s internal knowledge boundaries.
  • Approach: MMSearch-R1 trains LMMs to learn when to search, what to search for, and how to reason over retrieved results.It is presented as an end-to-end RL solution for on-demand search in real-world Internet environments.
  • Data: The dataset construction method estimates model familiarity and mixes search-required with search-free VQA samples to shape on-demand search behavior.Manually annotated test data covers diverse knowledge categories and difficulty levels.
  • Approach: The framework integrates image search for unfamiliar visual content with text search for precise textual knowledge retrieval.The two tools support complementary visual and textual information needs.
  • Results: MMSearch-R1-7B outperforms same-sized RAG baselines, performs competitively with a 32B RAG-based model, and reduces search calls by over 30%.The result is attributed to outcome-based reward with a search penalty and multi-turn search learning.
  • Impact: The authors share empirical findings and plan to open-source the data and complete training framework for further research.The stated goal is to facilitate research on search-augmented multimodal reasoning.

2 Building Iterative Multimodal Search-Integrated RL Framework

The framework extends GRPO with multimodal search interactions during iterative rollouts. Models can invoke image or text search across multiple turns, while rewards combine answer accuracy, search penalties, and format compliance.

  • RL Framework: GRPO is adapted to allow search interactions with the real-world environment during rollout without relying on a value function.Its group-based reward baseline reduces computational burden relative to PPO.
  • Search Tools: The search toolkit combines image retrieval of visually matched webpages with text retrieval, webpage reading, and question-focused summarization.Image search returns top-5 thumbnails and titles, while text search processes top-5 webpage URLs.
  • Rollout: Rollouts are multi-turn and iterative: the model reasons over new information, selects one action, and continues until answering or reaching the turn limit.Retrieved content is passed into the next dialogue round and masked during loss computation.
  • Reward Modeling: The reward combines an accuracy score with search penalty and a format score.Exact-match answer correctness, search reliance, and strict response formatting jointly determine the training reward.
  • Dataset Context: Figure 3 summarizes FVQA construction through automated visual-knowledge sample collection, a knowledge taxonomy, and combined automatic and manual sources.The figure presents the dataset’s collection and composition pipeline.

3 Curating Search-balanced VQA Datasets

FVQA is constructed to support reinforcement learning for on-demand search by covering both search-required and search-free multimodal questions. Its pipeline combines automated collection, search balancing, and manually verified evaluation data.

  • Dataset Criteria: The dataset includes search-free questions and search-required questions involving either visual knowledge or textual knowledge.Visual knowledge concerns recognizing entities, while textual knowledge concerns facts about those entities.
  • Construction: FVQA combines automated pipelines and manual annotation to support both training and evaluation.The dataset is designed as a multimodal search VQA resource.
  • Construction: Training-data selection uses two processes: VQA Collection and Search Balancing.These processes organize source selection and distinguish questions by search requirement.
  • VQA Collection: Visual Knowledge-required samples are collected by sampling common and niche concepts, retrieving image–webpage pairs, and generating factual VQA pairs with GPT-4o.The concepts are sampled from both head and tail regions of MetaCLIP metadata.
  • Search Balancing: Search Balancing trains a Qwen2.5-VL-Instruct-7B model and uses eight rollouts per question to classify training examples by search requirement.Questions for which all eight rollouts fail are discarded due to insufficient training signal.
  • Evaluation Data: FVQA-test contains 1,800 high-quality examples drawn from multiple sources and manually checked or human-annotated for accuracy.The test set includes 600 samples from FVQA-auto-vc and 600 from the InfoSeek Human Split among its sources.

4 Experiments

Experiments evaluate MMSearch-R1 across knowledge-intensive and information-seeking VQA benchmarks using accuracy and search ratio, comparing direct answering, fixed RAG workflows, and trained variants. Results show that reinforcement learning improves retrieval, internal-knowledge use, data efficiency, and selective search behavior.

  • Overall performance: MMSearch-R1-7B outperforms same-size RAG counterparts by 3% average accuracy while reducing average search rate by 32.9%.It also performs competitively with the larger RAG-based Qwen2.5-VL-32B model.
  • Retrieval abilities: Under the RAG workflow, MMSearch-R1-7B consistently improves over the base model across in-domain and out-of-domain tasks.The gains cover both image and text retrieval, including query generation and information extraction.
  • Internal knowledge use: RL training increases the proportion of answers that are correct without search on InfoSeek and SimpleVQA.The behavioral shift indicates greater use of internal knowledge when sufficient and more selective search for novel or long-tail queries.
  • RL versus SFT: RL-trained models consistently outperform SFT-trained models across five tasks despite using only about half as much training data.Their search-tool usage also aligns more closely with task requirements, increasing on the information-seeking MMSearch and LiveVQA datasets.
  • Search behavior: Balanced data and a search penalty maintain a lower, more stable search ratio than ablated strategies.Removing either component yields slightly higher rewards but search ratios that rapidly approach nearly 100%.

5 Conclusion

MMSearch-R1 combines reinforcement learning with multimodal search tools to equip LMMs for on-demand search in real-world environments. Its pipeline integrates search infrastructure and training components aimed at adaptive, efficient interaction with external information.

  • MMSearch-R1 equips LMMs with on-demand search in real-world Internet environments through reinforcement learning.
  • The framework combines image and text search tools so models can selectively retrieve visual or textual information and reason over retrieved content.
  • The multimodal search pipeline uses independent image- and text-search services supported by caching, parallel processing, and monitoring.
  • Search-service stability is supported by distributed rate limiting that smooths bursty requests and reduces overload risk.

D Prompts

The appendix documents prompts that structure VQA generation, reinforcement-learning interactions, webpage summarization, baseline workflows, and evaluation. These prompts constrain each model’s role and processing stage.

  • D.1 Prompts for FVQA-auto-ac VQA Generation: GPT-4o is prompted to generate concise, grounded VQA pairs from image–webpage pairs, focusing on observable or associated visual concepts.
  • D.2 Prompts for RL Training: RL training uses separate prompts at conversation start, after image search, and after text search to structure tool interactions.
  • D.3 Prompts for Webpage Summarization Model: The webpage summarization prompt directs the summarizer to extract content relevant to the user question.
  • D.3 Prompts for Webpage Summarization Model: The summarization prompt aims to make returned content more focused and concise, reducing overall token consumption.
  • D.4 Prompt for Direct Answer Baseline: The Direct Answer baseline asks the model to answer using only the image’s visual content.
  • D.5 Prompts for RAG Workflow Baseline: The RAG baseline requires exactly two searches per VQA example: image search followed by text search, within a maximum of three conversation rounds.
  • D.6 Prompts for LLM-as-Judge Evaluation: GPT-4o serves as the judge LLM, determining response correctness against ground-truth answers for accuracy computation.

E Details of Benchmark Datasets

The benchmark suite combines manually curated, semi-automatically constructed, and internet-sourced datasets spanning visual, textual, factual, and timely information-seeking questions. These datasets support broad evaluation of multimodal knowledge reasoning.

  • FVQA-test: FVQA-test contains 1800 examples from three sources, including verified training-derived samples, re-annotated InfoSeek examples, and newly collected human data.
  • InfoSeek: InfoSeek converts Wikidata triples into natural-language image-question-answer triplets using human-authored templates covering 300 relations.
  • MMSearch: MMSearch contains 300 manually collected examples across 14 subdomains, covering current news and rare verified knowledge.
  • SimpleVQA: SimpleVQA combines factual examples from existing VQA datasets with expert-annotated internet-sourced questions and contains 2,025 examples.
  • SimpleVQA: The English subset used for SimpleVQA evaluation contains 1,013 question–answer pairs.
  • LiveVQA: LiveVQA contains 3,602 VQA pairs from six news platforms and 14 categories, including basic visual and multi-hop questions.
  • Table 8 reports performance comparisons on general VQA benchmarks.

F.1 RL Training Setting

RL training uses Qwen2.5-VL-7B with GRPO in veRL on a search-balanced FVQA-train set. The configuration combines search-required and search-free examples with bounded tool interaction and explicit reward regularization.

  • The FVQA-train dataset contains approximately 1,600 search-free and 3,400 search-required samples.
  • Training uses Qwen2.5-VL-7B as the backbone and implements GRPO through the veRL framework.
  • Each training prompt generates 8 rollouts, with each rollout allowing up to 3 tool calls.
  • The search penalty and format-score weights are both set to 0.1, while β is fixed at 0.001 and the clip ratio ϵ at 0.2.
  • Training runs on 4 nodes with 8 Nvidia H100 GPUs each, using total and mini-batch sizes of 512 and 128.
  • The learning rate is 2e-6, and the step-50 checkpoint is used for downstream evaluation after convergence.

F.2 SFT Training Setting

The section describes the SFT comparison setup, inference procedure, LLM-based evaluation, and benchmark assessment of general VQA capabilities.

  • SFT Training Setting: The SFT comparison fine-tunes Qwen2.5-VL-7B on 8,000 GPT-4o-generated samples containing up to three dialogue rounds.GPT-4o responses provide supervision, while user questions and later tool-returned content are masked during loss computation.
  • SFT Training Setting: Inference uses veRL with vLLM, top_p 1.0, temperature 0, and one generated response per test sample.
  • SFT Training Setting: Evaluation uses GPT-4o-20241120 as an LLM judge whose binary Yes/No decisions determine final accuracy.The judge runs with top_p 0.1 and temperature 0.
  • SFT Training Setting: MMSearch-R1-7B achieves comparable performance to Qwen2.5-VL-7B across AI2D, ChartQA, LLaVA-Wilder, MathVista, MME, and OCRBench.It slightly outperforms the base model on LLaVA-Wilder and MathVista.
  • SFT Training Setting: Table 9 reports Acc (%) as LLM-as-Judge accuracy and SR (%) as the percentage of maximum allowed search steps used.The reward variants distinguish exact-match scoring (∗EM) from GPT-4o-based correctness judgments (∗4o).

G.2 Ablation Study on Reward Modeling

The reward-modeling ablation compares exact-match and GPT-4o-based rewards, finding that semantic judging improves accuracy while changing search behavior.

  • Reward Modeling: 59.5% average accuracy is achieved with GPT-4o reward, 3.8 points above exact-match reward.The ∗4o version improves across all datasets relative to the ∗EM version.
  • Reward Modeling: GPT-4o reward reduces false negatives from exact matching and improves robustness by judging semantic correctness.
  • Reward Modeling: GPT-4o reward may introduce bias, higher training cost, and difficulty with questions beyond its knowledge scope.
  • Reward Modeling: The experiment uses a subset of FVQA-train with relatively few search-free examples, producing a higher average search rate at convergence than Table 1.

H Limitations

MMSearch-R1’s limitations concern unstable multimodal tool interactions, inflexible exact-match rewards, and risks from retrieved web information.

  • Tool Interaction: External multimodal search tools remain constrained by stability and quality issues, including full-image submission and variable text-search components.The text pipeline includes SerpAPI, Jina Reader, and a summarization model.
  • Tool Interaction: Approximately 0.2% of image-search and 1% of text-search interactions fail end-to-end by returning no valid results.Partial failures, such as fewer than five expected results, make the rates higher.
  • Reward Design: Exact-string-match rewards suit short factual questions but may penalize semantically correct answers with different phrasing.This limits generalization to more complex or open-ended QA tasks.
  • Scaling Boundary: More robust tool interactions and more expressive rewards are needed to scale MMSearch-R1 toward adaptive and reliable multimodal reasoning agents.
  • Broader Risks: Autonomous retrieval and summarization may expose outdated, biased, or misleading information and raise verifiability, copyright, and misinformation concerns.The paper recommends filtering, source attribution, and provenance-tracing mechanisms.

J Examples of FVQA Dataset

The FVQA examples illustrate knowledge-intensive, fact-oriented visual question answering across diverse visual and textual knowledge types, alongside representative construction examples and model case studies.

  • FVQA Dataset: FVQA examples use knowledge-intensive, fact-oriented QA pairs spanning a broad range of visual and textual knowledge types.
  • FVQA-auto-vc: FVQA-auto-vc examples illustrate the dataset construction process for automatically generated visual-question-answering data.
  • Model Case Studies: Figures 9 and 10 present case studies of complex real-world information-seeking VQA using search invocation, query generation, and retrieved-information synthesis.Some retrieved search results are omitted for formatting and readability.
  • Example Content: The examples include an arts/place taxonomy and questions about a canceled lunar rover project and the Battle of Flodden.
Loading 2506.20670v1…