PaperlayerView exact revision on arXiv

Source-linked AI summary

WebWatcher: Breaking New Frontier of Vision-Language Deep Research Agent

Xinyu Geng, Peng Xia, Zhen Zhang, Xinyu Wang, Qiuchen Wang, Ruixue Ding, Chenxi Wang, Jialong Wu, Yida Zhao, Kuan Li, Yong Jiang, Pengjun Xie, Fei Huang, Jingren Zhou

arXiv:2508.05748v3cs.IR

TL;DR

Multimodal deep research is limited by text-centric research and systems that must integrate visual perception, textual reasoning, knowledge, and external tools. WebWatcher combines vision-language reasoning, multi-tool interaction, synthetic multimodal trajectories, reinforcement learning, and the BrowseComp-VL benchmark. It consistently outperforms open-source multimodal agents and proprietary systems on four reasoning benchmarks while remaining competitive on SimpleVQA.

  • Problem

    Existing deep-research work is primarily text-centric, while multimodal agents need stronger integrated perception, reasoning, knowledge, and tool-use capabilities.

  • Method

    WebWatcher combines vision-language reasoning, multiple external tools, synthetic multimodal trajectories for training, reinforcement learning, and BrowseComp-VL.

  • Results

    WebWatcher consistently outperforms existing open-source multimodal agents and proprietary systems on four reasoning benchmarks and achieves competitive performance on SimpleVQA.

  • Takeaways & Limitations

    BrowseComp-VL provides a challenging evaluation setting for multimodal agents requiring perception, cross-modal inference, information seeking, and planning.

  • Takeaways & Limitations

    Excessive edge or texture inference can amplify downstream tool errors.

Abstract

from arXiv · show

Web agents such as Deep Research have demonstrated superhuman cognitive abilities, capable of solving highly challenging information-seeking problems. However, most research remains primarily text-centric, overlooking visual information in the real world. This makes multimodal Deep Research highly challenging, as such agents require much stronger reasoning abilities in perception, logic, knowledge, and the use of more sophisticated tools compared to text-based agents. To address this limitation, we introduce WebWatcher, a multi-modal Agent for Deep Research equipped with enhanced visual-language reasoning capabilities. It leverages high-quality synthetic multimodal trajectories for efficient cold start training, utilizes various tools for deep reasoning, and further enhances generalization through reinforcement learning. To better evaluate the capabilities of multimodal agents, we propose BrowseComp-VL, a benchmark with BrowseComp-style that requires complex information retrieval involving both visual and textual information. Experimental results show that WebWatcher significantly outperforms proprietary baseline, RAG workflow and open-source agents in four challenging VQA benchmarks, which paves the way for solving complex multimodal information-seeking tasks.

1 Introduction

Multimodal deep research remains underdeveloped because existing systems often separate visual perception from deep textual reasoning and rely on rigid, scenario-specific pipelines. WebWatcher addresses this gap with integrated multimodal reasoning, multiple external tools, specialized training data, and the BrowseComp-VL benchmark.

  • Existing deep-research advances are primarily text-centric, leaving visual information in real-world information-seeking tasks underaddressed.
  • Current multimodal agents often use rigid, template-driven pipelines and visual tools that struggle to integrate visual reasoning with deep textual understanding.
  • WebWatcher combines vision-language reasoning with multiple external tools to support multimodal deep research.The paper identifies effective tool use alongside cross-modal reasoning as necessary for this setting.
  • The training-data pipeline targets in-depth, multi-step reasoning and strategic planning rather than single-step visual perception.It is designed to combine high-quality visual content with complex reasoning.
  • BrowseComp-VL extends BrowseComp-style information retrieval into the visual domain, requiring cross-modal inference, external search, image retrieval, and webpage browsing.Its queries are long and entity-obfuscated, so perception alone is insufficient.
  • WebWatcher consistently outperforms existing open-source multimodal agents and proprietary systems on four reasoning benchmarks while remaining competitive on SimpleVQA.

2 Data Preparation

BrowseComp-VL is built to test multimodal agents on difficult, web-based questions requiring visual grounding, multi-hop reasoning, and information synthesis. Its construction combines challenging textual QA generation, visual grounding, entity masking, and multi-stage quality filtering.

  • Dataset overview: BrowseComp-VL spans 5 major domains and 17 fine-grained subfields for multimodal reasoning in real-world web environments.
  • Difficulty levels: 199 Level 1 VQA pairs require multi-hop retrieval over explicit entities, while 200 Level 2 pairs use fuzzed entities and attributes.Level 2 questions require planning, comparison, and synthesis rather than direct lookup.
  • Textual QA generation: The generation pipeline first creates diverse, knowledge-intensive textual QA pairs from recursively browsed authoritative sources.Root URLs include arXiv, GitHub, and Wikipedia; GPT-4o synthesizes question-answer pairs from aggregated content.
  • QA-to-VQA conversion: Retained entities are grounded with authentic web images, transformed through entity masking, and filtered for query-image alignment and visual answerability.Each textual QA pair produces K multimodal examples, with K = 2 in the implementation; validation also uses image captions.

3 Trajectory Generation and Post-Training

WebWatcher is trained from filtered ReAct-style multimodal tool-use trajectories, using supervised fine-tuning for cold start and GRPO reinforcement learning for further decision-making refinement. The training process emphasizes correct, consistent, multi-step tool use and stable optimization.

  • Tool-augmented training: WebWatcher integrates Web Image Search, Web Text Search, webpage visits, Code Interpreter, and internal OCR for multimodal research.
  • Trajectory generation: Each ReAct-style trajectory consists of repeated think-act-observe cycles that select tools, receive observations, and eventually finish with an answer.The action space includes discrete tool-use actions and a Finish action that terminates the episode.
  • Trajectory selection: Trajectory filtering retains ground-truth-matching answers, checks step-by-step consistency, and removes examples with fewer than three tool calls.Consistency checks discard hallucinated content, contradictions, and unjustified tool calls.
  • Supervised fine-tuning: Supervised fine-tuning predicts each next tool-use action from the image, question, and previous actions and observations, providing a structured cold start.
  • Reinforcement learning: GRPO ranks groups of complete trajectories by relative return, normalizes rewards without a separate value function, and uses clipped updates with a KL penalty.Rewards combine binary tool-call format compliance with semantic answer accuracy; rollouts use groups of N = 16 trajectories.

4 Experiments

Experiments evaluate WebWatcher across challenging multimodal benchmarks, compare it with direct-inference, RAG, and reasoning baselines, and analyze tool use, reinforcement-learning cold starts, and repeated sampling.

  • Experimental Setup: The experiments use Qwen2.5-VL-7B and Qwen2.5-VL-32B, evaluate five benchmarks, and judge answer correctness with an LLM-as-Judges approach.Pass@k is generated through repeated sampling with temperature 0.6 and top-p 0.95.
  • Main Results: WebWatcher-32B achieves 18.2% average on HLE, surpassing the GPT-4o-based OmniSearch baseline by a substantial margin.It also shows particularly significant gains in Biology and Physics.
  • Tool-Use Analysis: BrowseComp-VL concentrates 62% of interactions on Web Text Search, whereas visual-content benchmarks use Web Image Search for around one-third or more of calls.HLE distributes calls more evenly across Web Text Search, Web Image Search, and Code Interpreter, while Code Interpreter is used when calculation is needed.
  • Cold Start for RL Training: Cold-start reinforcement learning compares an instruction-only initialization with extra SFT on trajectories demonstrating tool use and step-by-step visual reasoning.The analysis tests whether trajectory-based SFT initialization supports robust multi-hop interaction with the environment.
  • Pass@k Analysis: 41.9% pass@32 on HLE rises from 13.6% pass@1, with 20.3% at three roll-outs and 35.7% at sixteen.The curve increases monotonically, while marginal gains taper after approximately sixteen samples.

5 Related Work

Prior work established strong text-only deep-research agents, while multimodal benchmarks and agents remained limited in integrated reasoning, planning, and flexible tool use.

  • Deep Research: Text-only deep-research agents autonomously search, read, reason, and synthesize information from the open web.Proprietary systems exhibit near-expert performance, but undisclosed architectures and data pipelines limit rigorous ablation and reproducibility.
  • Multimodal VQA Benchmark: Existing VQA benchmarks mainly assess single-step perception or shallow retrieval rather than integrated multimodal reasoning and planning.Examples include OK-VQA and A-OKVQA, which emphasize static knowledge grounding and heuristic answer prediction.
  • Multimodal VQA Benchmark: Recent benchmarks broaden evaluation, but remain constrained by formats, domains, scale, manual curation, or limited planning structure.MMTU-Bench, MicroVQA, Open3DVQA, Dyn-VQA, MMMU, and MMMU-Pro address selected aspects without comprehensively combining multi-step reasoning, cross-modal integration, scale, automation, and quality control.

6 Conclusion

WebWatcher unifies multimodal deep research through vision-language reasoning, multi-tool interaction, trajectory-based supervision, and test-time scaling. The paper also introduces BrowseComp-VL to evaluate complex multimodal information-seeking and strategic planning.

  • Framework: WebWatcher combines complex vision-language reasoning with multi-tool interaction in a unified multimodal DeepResearch framework.The framework is designed for agents capable of autonomy, flexibility, and deep reasoning in real-world problems.
  • Benchmark: BrowseComp-VL evaluates in-depth multimodal reasoning and strategic planning through a challenging visual question-answering dataset.The paper describes it as part of a broader effort to address the underdeveloped multimodal DeepResearch landscape.
  • Training and Evaluation: WebWatcher uses automated action-observation trajectory generation for cold-start supervision and GRPO, alongside test-time scaling.These components are presented as mechanisms for equipping agents with robust tool-use capabilities and enhancing evaluation.

B Categories of BrowseComp-VL

BrowseComp-VL organizes questions across broad subject categories and uses prompts and evaluation procedures to assess answer correctness, trajectory quality, and tool-call rationality.

  • Categories: BrowseComp-VL spans five categories: sciences, engineering and computer science, social sciences and humanities, arts and sports, and other topics.The taxonomy is adapted from HLE and BrowseComp.
  • Answer Evaluation: The evaluation prompt compares a question, standard answer, and predicted answer, returning A, B, or C for correctness assessment.Rules address numerical near matches, requested-answer scope, and details inferable from the question.
  • Trajectory Evaluation: Tool-call rationality evaluation checks information non-redundancy, goal alignment, and logical reasoning and accuracy.The procedure judges whether calls are necessary, directly useful, efficient, precise, non-redundant, and specification-compliant.

D Experimental Details

The experimental framework defines tools through structured prompts and executes them in a ReAct trajectory with bounded tool use. Available operations cover code execution, text and image search, webpage visits, and image OCR.

  • Tool Framework: Each tool has a structured prompt specifying its callable format and semantic capability for model-directed invocation.The design supports reasoning about tool usage inside the tool-call block.
  • Available Tools: The tool suite includes Python code execution, Google text search, Google image search, webpage visits, and internal OCR.Inputs include code, search queries, image URLs, webpage URLs, and OCR image URLs, respectively.
  • Trajectory Format: Qwen-Agent implements the ReAct framework, with no more than 15 tool calls per complete trajectory.A trajectory alternates thinking, tool calls, tool responses, and a final answer.

D.3 Training Details

Training uses Llama-Factory for supervised fine-tuning and Verl for reinforcement learning, with separately specified optimization settings for each stage.

  • Training frameworks: Llama-Factory is used for SFT, while Verl is used for RL training.
  • SFT settings: SFT uses batch size 32, learning rate 5e-6 with a minimum of 1e-10, warmup plus cosine decay, and weight decay 0.1.
  • RL settings: RL uses eight rollouts per group, temperature 1.0, top_p = 1.0, batch size 128, mini batch size 32, and learning rate 1e-6.

D.4 Benchmarks

The evaluation covers BrowseComp-VL, HLE, LiveVQA, SimpleVQA, and MMSearch, spanning multimodal reasoning, current visual knowledge, factual VQA, and rare or recent information.

  • BrowseComp-VL: BrowseComp-VL uses 100 Level 1 and 200 Level 2 instances, manually verified by PhD-level AI experts.The benchmark requires strong planning and proficient use of external tools.
  • HLE: HLE evaluates 330 multimodal questions sampled from a 2,500-question expert-written benchmark spanning diverse academic fields.Its questions require evidence synthesis from obscure or fragmented sources and abstract reasoning.
  • LiveVQA: LiveVQA evaluates a 300-example subset of 3,602 multi-hop VQA instances covering recent global news across six sources and fourteen topics.
  • SimpleVQA: SimpleVQA evaluates 300 randomly sampled examples from its 1,013 English question-answer pairs.The broader benchmark contains 2,025 factual VQA examples in English and Chinese.
  • MMSearch: MMSearch evaluation uses its visual subset of 171 image-paired examples from 300 manually curated cases across 14 subdomains.The examples cover recent news and rare knowledge.

D.5 Evaluation Prompt

The evaluation prompt judges whether an extracted final answer matches the precise correct answer, while recording reasoning and confidence.

  • Scoring target: The evaluator judges whether the response correctly answers the question using the precise and unambiguous correct_answer.
  • Answer extraction: The evaluator extracts the exact final answer string, using None when no clear final answer is present.
  • Judgment rationale: The reasoning field briefly explains differences or equivalence between extracted_final_answer and correct_answer without restating background.
  • Correctness: The correct field is yes when the extracted answer matches the correct answer within small numerical margins, otherwise no.
  • Confidence: Confidence is recorded on a 0%–100% scale, defaulting to 100% when no confidence is provided.

D.6 Case Study

The case study applies web search, webpage inspection, OCR-oriented reasoning, and code verification to solve a 4x4 KenKen puzzle. The reported top row is 4, 3, 1, 2, although the intermediate verification exposes inconsistencies in the displayed solution.

  • Case setup: The case asks for four numbers on the top horizontal line of a completed KenKen puzzle and requires identifying its constraints first.
  • Information retrieval: Image search returns KenKen lesson pages but does not directly retrieve the puzzle shown in the image.
  • Solving strategy: The solving approach emphasizes arithmetic constraints, placement rules, and cages with limited valid combinations, including the 2+ cage.
  • Constraint solving: The attempted grid is represented as a 4x4 array and checked against seven cage constraints involving sums and products.
  • Verification: The verification output initially reports the top-left 8+ constraint as false while the other six listed checks are true.
  • Answer: The reported top row extracted from the solved grid is 4, 3, 1, 2.
Loading 2508.05748v3…