Source-linked AI summary
MM-DeepResearch: A Simple and Effective Multimodal Agentic Search Baseline
Huanjin Yao, Qixiang Yin, Min Yang, Ziwang Zhao, Yibo Wang, Haotian Luo, Jingyi Zhang, Jiaxing Huang
TL;DR
MM-DeepResearch targets scarce multimodal search data, ineffective search trajectories, and costly online training. It combines Hyper-Search, DR-TTS, and an offline search engine, achieving superior benchmark performance and outperforming prior agents trained with costly online APIs.
Problem
Multimodal deep research is limited by scarce search-intensive QA data, inadequate multi-turn search trajectories, and online search API costs reaching thousands of dollars per training run.
Method
The paper generates multimodal QA with Hyper-Search, synthesizes trajectories with specialized tool experts and DR-TTS, and enables reinforcement learning through an offline multi-tool search engine.
Results
MM-DeepResearch achieves superior performance across multiple benchmarks and outperforms previous agents trained with costly online search APIs.
Takeaways & Limitations
The combined designs provide a baseline that supports multimodal deep research training with search-intensive data, synthesized trajectories, and offline search.
Takeaways & Limitations
The implementation evaluates both models without native agentic tool use and models with native agentic tool-use capabilities as separate base-model settings.
Abstract
from arXiv · showhide
We aim to develop a multimodal research agent capable of explicit reasoning and planning, multi-tool invocation, and cross-modal information synthesis, enabling it to conduct deep research tasks. However, we observe three main challenges in developing such agents: (1) scarcity of search-intensive multimodal QA data, (2) lack of effective search trajectories, and (3) prohibitive cost of training with online search APIs. To tackle them, we first propose Hyper-Search, a hypergraph-based QA generation method that models and connects visual and textual nodes within and across modalities, enabling to generate search-intensive multimodal QA pairs that require invoking various search tools to solve. Second, we introduce DR-TTS, which first decomposes search-involved tasks into several categories according to search tool types, and respectively optimize specialized search tool experts for each tool. It then recomposes tool experts to jointly explore search trajectories via tree search, producing trajectories that successfully solve complex tasks using various search tools. Third, we build an offline search engine supporting multiple search tools, enabling agentic reinforcement learning without using costly online search APIs. With the three designs, we develop MM-DeepResearch, a powerful multimodal deep research agent, and extensive results shows its superiority across benchmarks. Code is available at https://github.com/HJYao00/MM-DeepResearch
1. Introduction
MM-DeepResearch addresses limited multimodal search data, ineffective multi-turn search trajectories, and costly online training through Hyper-Search, DR-TTS, and an offline search engine. The resulting agent achieves superior performance across multiple benchmarks while using online search APIs only for evaluation.
- Existing models are constrained by fixed, bounded knowledge, limiting their ability to handle information-intensive and open-world tasks.
- Multimodal deep research agents must support explicit reasoning, planning, multi-tool invocation, and cross-modal information synthesis.
- Public search-intensive multimodal QA data is scarce, effective multi-turn search trajectories are lacking, and online search APIs can cost thousands of dollars per training run.
- Hyper-Search connects visual and textual nodes within and across modalities to generate QA pairs requiring multiple search tools, while DR-TTS trains specialized tool experts and recomposes them for trajectory search.
- The offline search engine supports information-based and knowledge-based search, speeds tool responses, reduces interactive training time, and avoids costly online API use.
- MM-DeepResearch achieves superior performance across multiple benchmarks and outperforms previous agents trained with costly online search APIs.
2. Related Work
Related work progresses from general multimodal reasoning and static retrieval workflows toward deep research agents that iteratively seek and synthesize external evidence, including multimodal tool use.
- Multimodal language models combine large-scale pretraining, instruction tuning, long-chain reasoning, and increasingly autonomous tool invocation.
- Workflow-based search agents use predefined RAG-based or prompt-based pipelines to retrieve external knowledge before reasoning.
- Deep research agents iteratively seek information and synthesize evidence through external search tools, with multimodal systems extending this paradigm to image and text search.
3. MM-DeepResearch Data
The data pipeline combines Hyper-Search for multimodal QA construction and filtering with DR-TTS for specialized-expert trajectory synthesis, supported by multiple search tools.
- Hyper-Search: Hyper-Search constructs multimodal QA data through hypergraph construction, QA generation, and data filtering.
- Hypergraph construction: The hypergraph represents image and webpage-content nodes, with captions and summaries providing cues for multimodal QA generation.
- Hypergraph construction: Image expansion uses reverse search for relevant webpages and visual search for similar images, while text expansion retrieves relevant webpages and images from page content.
- Hypergraph construction: Hyperedges connect parent nodes with newly generated multimodal nodes, explicitly modeling cross-modal retrieval relationships.
- Multimodal QA generation: QA generation combines cross-modal and cross-source evidence within single hyperedges or across multiple hyperedges to increase search depth and difficulty.
- Search QA filtering: MLLM filtering removes low-quality, duplicated, visually irrelevant, search-free, or unverifiable instances, producing 3K QA pairs called Hyper-Search-3K.
- DR-TTS: DR-TTS decomposes tasks by required search tools, trains specialized experts, and recomposes them for joint tree search to discover diverse valid trajectories.
- Search tools: The search framework includes information-based tools for grounded factual retrieval and knowledge-based tools for specialized knowledge acquisition.
4. MM-DeepResearch Agents
MM-DeepResearch uses trajectory-based supervised fine-tuning followed by multi-turn reinforcement learning with an offline multimodal search engine. The training design teaches tool invocation and information synthesis while avoiding the cost of online search during RL, then uses online tools at test time with response summarization.
- Training Recipe: The model is trained in two stages: supervised fine-tuning on search tool trajectories, followed by agentic multi-turn reinforcement learning with an offline search engine.The supervised stage provides a cold start before reinforcement learning.
- Multi-Turn SFT: Search trajectories train reasoning steps, tool calls, and final answers while masking tool responses from the SFT loss.Tool responses remain observations that condition subsequent predictions, and the final answer is supervised only at the final step.
- Offline Search Engine: The offline search engine supports visual and textual retrieval without the prohibitive cost of repeatedly querying online search APIs during reinforcement learning.Its corpus combines generated textual queries, fetched webpages, Wikipedia data, and pre-collected image-related information.
- Multi-Turn Reinforcement Learning: Multi-turn GRPO samples trajectories containing reasoning, tool invocations, tool responses, and final answers, then rewards both valid interaction format and answer accuracy.Format-compliant trajectories receive reward 1, while invalid or unparsable tool calls receive 0; accuracy is assessed by an LLM against the golden answer.
- Optimization: The final reward combines accuracy and format rewards, and GRPO optimizes the model with a clipped objective and KL penalty.The coefficient α controls the balance between the two reward components.
- Inference: At test time, online search responses are verified and summarized by an auxiliary LLM to reduce redundancy and prevent context overflow during reasoning.This processing addresses the longer webpage content typically returned by online search.
5. Experiment
The experiments evaluate MM-DeepResearch across six search-intensive benchmarks and compare models with and without native agentic tool-use capabilities. Results and ablations show improvements from the proposed data, trajectories, and offline search tools.
- Evaluation setup: MM-DeepResearch is evaluated on six information-intensive benchmarks requiring search tools, using models with and without native agentic tool-use capabilities.The study compares against non-agentic and agentic search MLLMs and uses online retrieval during evaluation.
- Main results: 23% and 7.1% average improvements over Visual-ARFT and MMSearch-R1-7B show that MM-DeepResearch-7B learns agentic search capabilities from Qwen2.5-VL-7B.It also exceeds WebWatcher by 7.7% on SimpleVQA and 12.3% on MM-Search.
- Main results: 17% average improvement over Qwen3-VL-8B and 14.9% over Qwen3-VL-32B show gains beyond agentic foundational MLLMs.MM-DeepResearch-8B also gains 3.4 points over SenseNova-MARS-8B on average and improves 4.2% on SimpleVQA.
- Ablation study: 66.9 on MMSearch is reached with T2T, T2I, and I2T information-based search, while adding knowledge-based T2T raises the score to 67.8.The information-based tools increase performance from 11.7, indicating complementary effects between information-based and knowledge-based search.
6. Conclusion
MM-DeepResearch integrates reasoning, planning, multi-turn tool invocation, and long-context cross-modal integration through three training components. Experiments report effectiveness across models with and without native agentic capabilities.
- Conclusion: MM-DeepResearch integrates reasoning and planning, multi-turn search tool invocation, and long-context cross-modal information integration for deep research tasks.Its components are designed for search-intensive QA generation, search-trajectory synthesis, and scalable reinforcement-learning training.
- Conclusion: Hyper-Search, DR-TTS, and an offline search engine support training a search agent from scratch while enhancing search and reasoning capabilities.The conclusion presents these three components as the core development designs.
- Conclusion: Experiments demonstrate effectiveness across models both with and without native agentic capabilities.The paper positions MM-DeepResearch as a simple baseline for multimodal deep research agents.
A. Multimodal QA Dataset
The multimodal QA dataset uses Hyper-Search to organize visual and textual retrieval material across seven domains. Its offline search engine controls retrieved context to support efficient multimodal reasoning.
- Dataset construction: Visual sources from seven categories—arts, sports, education, history, movies, places, and technology—serve as starting image nodes for Hyper-Search.Figures 4–10 provide dataset examples across these categories.
- Offline search engine: The offline corpus combines retrieved webpage content with large-scale English Wikipedia from 2018 to improve coverage and diversity.The corpus is built using a FlashRAG-based retrieval framework.
- Retrieval settings: During training, the engine retrieves the top-3 textual passages and retains images above 0.7 embedding similarity, keeping one image per tool query.During evaluation, retrieval expands to the top-5 textual results and up to three images per query.
C. Evaluation Benchmark
MM-DeepResearch is evaluated on six multimodal benchmarks spanning factual visual question answering, browsing, current visual knowledge, commonsense knowledge, information seeking, and long-horizon multimodal search.
- Benchmark suite: Six information-intensive benchmarks are used to evaluate MM-DeepResearch.The benchmarks require invoking search tools to solve their tasks.
- SimpleVQA: SimpleVQA evaluates factuality on short natural-language questions grounded in images using 300 sampled examples.Its evaluation combines manual verification with an LLM-as-judge scoring system.
- MMSearch: MMSearch tests visual-textual integration, external retrieval, provenance verification, and long tool-augmented reasoning chains on 171 image-based examples.The benchmark emphasizes localized visual cues and cross-modal evidence gathering.
- LiveVQA: LiveVQA contains over 107,000 samples focused on up-to-date visual knowledge from recent news, video, and academic platforms.It evaluates visual information beyond models’ training cutoffs.
- FVQA: FVQA requires external commonsense knowledge and provides a supporting fact as a structured knowledge triplet for each image-question-answer example.The benchmark tests retrieval and integration of factual knowledge beyond visual perception.
- InfoSeek: InfoSeek evaluates information-seeking questions that cannot be answered from common sense or image content alone, using 2,000 evaluation instances.The instances are sampled by MMSearch-R1.
D.1. Discussion on Different Embedding Models for the Offline Search Engine
Offline text retrieval is relatively robust to embedding-model choice. Because e5-base-v2 offers comparable retrieval quality with greater efficiency, the paper adopts it for large-scale indexing and retrieval.
- Embedding-model performance differences for offline text retrieval are marginal.
- jina-embeddings-v4 achieves slightly better performance but has higher cost and slower inference.
- e5-base-v2 is adopted because it provides comparable retrieval quality with greater efficiency for large-scale offline indexing and retrieval.
D.2. Comparing Online and Offline Search Performance at Evaluation
Online search performs better during evaluation because it accesses newer and larger-scale web information. Offline search remains competitive on SimpleVQA and MMSearch and provides useful retrieval signals for learning search behaviors.
- Online search consistently achieves better evaluation performance than offline search.
- Offline search attains competitive results on SimpleVQA and MMSearch despite not fully matching online search.
- Offline search is sufficiently effective to provide meaningful retrieval signals and stimulate learning of search behaviors.
D.3. Discussion on the Impact of Top-k Text Retrieval during Training
Moderate top-k retrieval performs best during training, while retrieving too many passages produces diminishing returns and can degrade performance by adding noise and reasoning difficulty.
- Moderate values of k lead to better training performance than very small or large retrieval settings.
- Retrieving five passages achieves the best performance on MMSearch, outperforming k = 3 and k = 10.
- Excessive retrieval can dilute relevant evidence and increase reasoning difficulty, leading to performance degradation.
D.4. Discussion of Online and Offline Search Costs.
Offline search substantially reduces the cost and latency of reinforcement-learning training compared with online search. It therefore offers a cost-efficient and scalable alternative for multi-rollout training.
- Online search costs around 640 dollars per hundred steps during reinforcement-learning training.
- Online search averages about 60 seconds per sample, whereas offline search takes about 1 second per sample.
- Offline search operates with no additional search-related cost and is a cost-efficient, scalable alternative for GRPO training.