Source-linked AI summary
Vision-DeepResearch: Incentivizing DeepResearch Capability in Multimodal Large Language Models
Wenxuan Huang, Yu Zeng, Qiuchen Wang, Zhen Fang, Shaosheng Cao, Zheng Chu, Qingyu Yin, Shuang Chen, Zhenfei Yin, Lin Chen, Zehui Chen, Xu Tang, Yao Hu, Shaohui Lin, Philip Torr, Feng Zhao, Wanli Ouyang
TL;DR
Existing multimodal deep-research systems use overly simple visual retrieval and limited search trajectories, leaving them poorly suited to noisy, knowledge-intensive tasks. Vision-DeepResearch addresses this with multi-turn, multi-entity, multi-scale visual and textual search, trained through synthesized trajectories, cold-start supervision, and reinforcement learning. It significantly outperforms prior multimodal deep-research MLLMs and strong closed-source agent workflows across six factual benchmarks.
Problem
Existing systems rely on simplistic visual queries and short reasoning trajectories, limiting robust evidence aggregation in noisy, complex multimodal research tasks.
Method
Vision-DeepResearch combines multi-turn, multi-entity, and multi-scale visual and textual search with synthesized trajectories, cold-start supervision, and reinforcement learning.
Results
Vision-DeepResearch significantly outperforms prior multimodal deep-research MLLMs and workflows built on GPT-5, Gemini-2.5-Pro, and Claude-4-Sonnet across six factual benchmarks.
Takeaways & Limitations
The approach scales multimodal deep-research trajectories to dozens of reasoning steps and hundreds of search-engine interactions.
Takeaways & Limitations
RL training was not exhaustively scaled because of API cost and wall-clock constraints, and larger-scale optimization may provide further benefits.
Abstract
from arXiv · showhide
Multimodal large language models (MLLMs) have achieved remarkable success across a broad range of vision tasks. However, constrained by the capacity of their internal world knowledge, prior work has proposed augmenting MLLMs by ``reasoning-then-tool-call'' for visual and textual search engines to obtain substantial gains on tasks requiring extensive factual information. However, these approaches typically define multimodal search in a naive setting, assuming that a single full-level or entity-level image query and few text query suffices to retrieve the key evidence needed to answer the question, which is unrealistic in real-world scenarios with substantial visual noise. Moreover, they are often limited in the reasoning depth and search breadth, making it difficult to solve complex questions that require aggregating evidence from diverse visual and textual sources. Building on this, we propose Vision-DeepResearch, which proposes one new multimodal deep-research paradigm, i.e., performs multi-turn, multi-entity and multi-scale visual and textual search to robustly hit real-world search engines under heavy noise. Our Vision-DeepResearch supports dozens of reasoning steps and hundreds of engine interactions, while internalizing deep-research capabilities into the MLLM via cold-start supervision and RL training, resulting in a strong end-to-end multimodal deep-research MLLM. It substantially outperforming existing multimodal deep-research MLLMs, and workflows built on strong closed-source foundation model such as GPT-5, Gemini-2.5-pro and Claude-4-Sonnet. The code will be released in https://github.com/Osilly/Vision-DeepResearch.
1. Introduction
Existing multimodal deep-research methods struggle with noisy visual retrieval and limited reasoning depth, while Vision-DeepResearch combines multi-scale multimodal search with automated trajectory synthesis and training.
- MLLMs remain challenged by complex fact-intensive VQA because their internal world knowledge is limited.
- Existing methods oversimplify multimodal search by relying on full-image retrieval and few text queries, making them vulnerable to visual noise and unstable hit rates.
- Prior systems also restrict reasoning depth and search breadth, hindering multi-hop reasoning and evidence aggregation across diverse sources.
- Vision-DeepResearch integrates multi-turn, multi-entity, and multi-scale visual and textual search into an MLLM through cold-start supervision and reinforcement learning.
- Its data pipeline combines an MLLM with a text-based DeepResearch foundation LLM to generate long-horizon, multi-tool trajectories and verified factual VQA data.
- Extensive experiments show significant gains over prior multimodal deep-research MLLMs and workflows built on GPT-5, Gemini-2.5-Pro, and Claude-4-Sonnet across six factual benchmarks.
2. Method
Vision-DeepResearch addresses noisy multimodal retrieval and limited search depth by combining iterative visual search with text-based deep-research reasoning. Its data pipeline constructs long-horizon trajectories that bridge visual and textual tool use.
- Motivation: Vision-DeepResearch models retrieval as trial-and-error exploration, refining queries across multiple visual regions, scales, and textual attempts based on intermediate results.This design targets higher hit rates and more reliable evidence accumulation for ambiguous inputs.
- Motivation: Existing multimodal deep-research systems rely on coarse, often one-shot visual and textual retrieval that struggles with noisy and unstable search environments.Images may contain clutter and multiple entities, while small query changes can produce substantially different textual results.
- High-quality multimodal trajectory generation: The trajectory pipeline combines visual search with text-based deep-research reasoning by bridging visual contexts through detailed image descriptions.The bridged context preserves visual reasoning, actions, and observations before a text-based foundation model continues the trajectory with web search, website summarization, and Python tools.
- High-quality multimodal trajectory generation: Visual search generates localized entity-level crops at multiple scales, submits them as tool calls, and accumulates observations from search, webpage visits, and webpage summaries.An auxiliary MLLM summarizes long webpage content and verifies correspondence between the crop and the matched result.
- High-quality multimodal trajectory generation: A judge model evaluates whether accumulated visual evidence is sufficient for downstream reasoning, continuing search when the binary hit signal is zero and terminating when it is one.The judge is designed to make a relatively lenient sufficiency decision before the text-based reasoning stage.
- Verified factual VQA generation: Verified factual VQA generation increases reasoning complexity through answer obfuscation and entity obfuscation, alternating both to avoid rigid chains and visual-free shortcuts.Answer obfuscation chains relations around the answer, whereas entity obfuscation replaces entities through multi-hop webpage links.
2.3. Training
Vision-DeepResearch is trained with supervised fine-tuning and reinforcement learning on multimodal deep-research trajectories, teaching multi-turn, multi-entity, and multi-scale evidence integration.
- The model is trained using generated multimodal deep-research trajectories and verified VQA with supervised fine-tuning followed by reinforcement learning.
- SFT uses trajectories containing questions, images, and multi-turn multimodal and text-only research steps.
- SFT teaches multi-turn, multi-entity, and multi-scale reasoning while integrating visual and textual evidence and developing long-horizon planning.
- The training section includes ablation studies of the rollout pipeline and of training data and methods.
High-throughput Asynchronous Rollout Architecture
The training system uses asynchronous, high-throughput rollouts and RL with online search interactions, while engineering safeguards address long-tail latency, anomalous trajectories, and numerical instability.
- A multi-threaded asynchronous rollout pipeline uses queued scheduling and a tool pool for concurrent multi-tool calls.
- RL uses GRPO with a Leave-One-Out trick on 15K high-quality VQA instances and a real online environment containing visual search, text search, and website visiting.
- The reward is pure accuracy: the model receives 1.0 for a correct generated answer and 0.0 otherwise.
- Trajectory interruption detects repetitive generation and terminates trajectories that exceed the repetition threshold.
- Anomalous, over-budget, or error-dominated trajectories are masked from gradient updates to avoid overly punitive negative signals.
- The authors use BF16 because long rollouts of up to 64K context caused numerical overflow and instability under FP16.
- The engineering techniques are presented as crucial for stable large-scale agentic-reasoning RL training rather than as algorithmic innovations.
3. Experiments
Experiments compare Vision-DeepResearch with existing systems and isolate the effects of retrieval pipelines, training data, and RL. Results favor multi-scale visual search combined with text search and online RL.
- Experiments compare methods across multimodal retrieval and reasoning benchmarks, then study pipeline components, data choices, and RL training.
- Main Results: Tool-free answering is weak at 24.1% Avg., while a ReAct-style Gemini-2.5 Pro workflow reaches 50.7% Avg.; naive RAG provides limited gains.
- Main Results: Vision-DeepResearch-8B improves over Qwen3-VL-8B-Instruct (Agentic) by +10.4% Avg., while the 30B-A3B model reaches 56.9 Avg. (+16.0).
- Pipeline Ablation: Removing retrieval gives 12.0% Avg., whole-image search gives 16.0% Avg., and adding text search raises whole-image retrieval to 29.3% Avg.
- Pipeline Ablation: The full CIS+TS pipeline achieves VDR 37.8%, MMS+ 28.5%, BC-VL 53.7%, and 40.0% Avg., outperforming the ablated retrieval variants.
- Data Ablation: SFT with verified, text-only, and fuzzy multi-hop trajectories improves results, while RL produces the best scores: VDR 37.8%, MMS+ 28.5%, and BC-VL 53.7%.
- RL Training: RL shortens trajectories while increasing rewards and improves performance by an average of 3.1% on three challenging benchmarks.
- RL Training: RL training was not exhaustively scaled because of API cost and wall-clock constraints.
4. Conclusion
Vision-DeepResearch scales multimodal deep-research trajectories to dozens of reasoning steps and hundreds of search-engine interactions, yielding substantially stronger performance.
- Vision-DeepResearch scales multimodal deep-research trajectories to dozens of reasoning steps and hundreds of search-engine interactions.
A.1. Text-only DeepResarch LLMs
Text-only deep-research LLMs model information retrieval as iterative reasoning, tool use, and evidence integration, but their textual access limits multimodal understanding.
- Deep-research LLMs formulate complex information retrieval as an iterative reasoning–tool call–re-reasoning loop.Agents generate queries, browse web pages, and integrate evidence into long reasoning chains.
- This paradigm has improved open-domain question answering and knowledge-intensive reasoning tasks.
- Text-only agents lack visual perception, entity localization, and cross-modal consistency verification because they rely almost exclusively on textual retrieval.This limits their support for complex web-page understanding, GUI comprehension, and other multimodal applications.
A.2. Multimodal DeepResearch MLLMs
Multimodal deep-research MLLMs extend tool-based reasoning into visual environments, but existing approaches remain coarse and poorly optimized for long-horizon visual–text interaction.
- WebWatcher uses reverse image search to convert text-based question answering into visual question answering and supports multi-step image reasoning.It also constructs supervised fine-tuning data for image retrieval and reasoning.
- MMSearch-R1 uses GRPO to encourage models to invoke both image and text search tools.
- Prior systems often rely on one-shot full-level or entity-level image retrieval, making fine-grained visual search unreliable in realistic web settings.Textual retrieval may likewise require multiple attempts to reach the desired target.
- Training commonly targets short contexts or fewer-than-five-round retrieval, leaving models prone to premature termination or shallow search on complex tasks.The stated gap is systematic data and objective design for visual–text search over tens of rounds.
B. Experiment Setups
The experiments train Qwen3-VL models with supervised and reinforcement learning data from an automated multimodal research pipeline, then evaluate them across six benchmarks against diverse model and agent baselines.
- Training Data and Models: Training uses 30K high-quality visual deep-research trajectories for SFT, with SFT and RL applied to Qwen3-VL-30B-A3B-Instruct and only SFT to Qwen3-VL-8B-Instruct.Autoregressive supervision is applied at each trajectory step.
- Benchmarks: Evaluation covers six benchmarks: VDR-Bench, FVQA, MMSearch-Plus, MMSearch, LiveVQA, and BrowseComp-VL.
- Benchmarks: VDR-Bench uses its test-mini split, BC-VL its full split, MMSearch all VQA instances, and MMSearch-Plus its single-image subset.FVQA and LiveVQA are evaluated on randomly selected samples.
- Training Data and Models: The data pipeline generates long-horizon, multi-tool trajectories and produces factual VQA instances through verification and obfuscation for trajectory synthesis and RL training.
- Baselines: Baselines include proprietary and open-source multimodal models under direct-answer and ReAct-style agentic reasoning settings.All models receive a unified multimodal toolset and consistent judge prompt.
C. Case Study
The section refers to Fig. 4, but the supplied passage does not state what the figure shows.
- The passage provides only a cross-reference to Fig. 4.