PaperlayerView exact revision on arXiv

Source-linked AI summary

WebWatcher: Breaking New Frontier of Vision-Language Deep Research Agent

Xinyu Geng, Peng Xia, Zhen Zhang, Xinyu Wang, Qiuchen Wang, Ruixue Ding, Chenxi Wang, Jialong Wu, Yida Zhao, Kuan Li, Yong Jiang, Pengjun Xie, Fei Huang, Jingren Zhou

arXiv:2508.05748v3cs.IR

TL;DR

Deep-research agents have largely remained text-centric, although multimodal information seeking requires reasoning across visual and textual information. WebWatcher combines vision-language reasoning and tool use with supervised and reinforcement-learning training, and outperforms open-source multimodal agents and proprietary systems on four reasoning benchmarks while remaining competitive on SimpleVQA.

  • Problem

    Most deep-research advances remain text-centric, despite multimodal agents needing reasoning over visual as well as textual information.

    Source: §1 · p. 2

  • Method

    WebWatcher is a vision-language research agent that uses external tools, with supervised fine-tuning on generated ReAct trajectories followed by reinforcement learning to optimize tool use and decision-making.

    Source: §1 · p. 2, §3 · p. 6

  • Results

    WebWatcher outperforms existing open-source multimodal agents and proprietary systems on four reasoning benchmarks and is competitive on SimpleVQA, scoring 58.7% on LiveVQA.

    Source: §1 · p. 3, §4.2 · p. 10

  • Takeaways & Limitations

    WebWatcher establishes a strong foundation for future multimodal DeepResearch agents.

    Source: §6 · p. 13

Abstract

from arXiv · show

Web agents such as Deep Research have demonstrated superhuman cognitive abilities, capable of solving highly challenging information-seeking problems. However, most research remains primarily text-centric, overlooking visual information in the real world. This makes multimodal Deep Research highly challenging, as such agents require much stronger reasoning abilities in perception, logic, knowledge, and the use of more sophisticated tools compared to text-based agents. To address this limitation, we introduce WebWatcher, a multi-modal Agent for Deep Research equipped with enhanced visual-language reasoning capabilities. It leverages high-quality synthetic multimodal trajectories for efficient cold start training, utilizes various tools for deep reasoning, and further enhances generalization through reinforcement learning. To better evaluate the capabilities of multimodal agents, we propose BrowseComp-VL, a benchmark with BrowseComp-style that requires complex information retrieval involving both visual and textual information. Experimental results show that WebWatcher significantly outperforms proprietary baseline, RAG workflow and open-source agents in four challenging VQA benchmarks, which paves the way for solving complex multimodal information-seeking tasks.

1 Introduction

Multimodal deep research remains underexplored despite the need to combine visual and textual reasoning, and WebWatcher is introduced as a tool-using vision-language agent for such tasks.

1 Introduction

  • Current multimodal agents are limited by rigid pipelines: visual-only tools struggle with cross-modal reasoning, while search-only agents cannot handle complex interactions or computation.The GAIA example requires both searching and browsing linked pages; vision-only analysis and search-based agents fail to solve it.Source: §1 · p. 2, Figure 2 · p. 3
  • WebWatcher is a vision-language web agent designed for deep research, combining textual and visual reasoning with multiple external tools.Its tools include web image search, web text search, webpage visits, a code interpreter, and internal OCR.Source: §1 · pp. 2–3
  • Its training-data pipeline turns high-difficulty, multi-hop web questions into multimodal VQA items, using entity masking and multi-stage filtering.The process begins with random walks over diverse web sources; its QA-to-VQA pipeline is designed to work with existing QA datasets.Source: §1 · p. 2
  • WebWatcher is finetuned on synthesized tool-use trajectories and further optimized with GRPO reinforcement learning.An automated prompting pipeline constructs trajectories from action-observation sequences, with LLM prompting and rule-based filtering for quality.Source: §1 · p. 3
  • BrowseComp-VL is a BrowseComp-style visual benchmark with long, entity-obfuscated queries requiring perception, cross-modal inference, information gathering, and planning.Solving its tasks requires external tools such as web search, image retrieval, and webpage browsing; perception alone is insufficient.Source: §1 · p. 3
  • WebWatcher outperforms open-source multimodal research agents and proprietary systems on four reasoning benchmarks, while remaining competitive on SimpleVQA.The benchmarks named are HLE, LiveVQA, BrowseComp-VL, and MMSearch.Source: §1 · p. 3

2 Data Preparation

BrowseComp-VL is built by creating knowledge-intensive, multi-hop textual questions and grounding them in authentic images, preserving textual reasoning complexity in multimodal examples. Its preparation combines difficulty-tiered QA generation, image-based question transformation, and filtering for quality.

2.1 Data Overview

  • BrowseComp-VL spans five domains and 17 subfields, with 199 explicit-entity Level 1 pairs and 200 fuzzed-entity Level 2 pairs.Level 1 requires multi-hop retrieval across sources; Level 2 obscures entities or attributes and requires planning, comparison, and synthesis.Source: §2.1 · p. 4
  • Rather than prompting directly from images, the pipeline creates challenging multi-hop textual QA pairs first, then grounds them in relevant visual content.This is intended to avoid shallow, single-hop image questions and retain textual reasoning complexity alongside visual richness.Source: §2.1 · p. 4

2.2.1 Generate QA

  • Level 1 QA uses authoritative web sources and recursive hyperlink traversal, while Level 2 questions fuzz key references to require contextual reasoning and synthesis.For Level 1, GPT-4o synthesizes QA from aggregated content; the graph-expansion procedure uses depth 3 and branching factor 3 to sample multi-hop reasoning paths.Source: §2.2.1 · pp. 4–5

2.2.2 QA-to-VQA Conversion

  • For VQA conversion, target entities are filtered for visual grounding, authentic web images are retrieved, and GPT-4o replaces explicit entity mentions with image references.Each retrieved image is paired with the transformed question and answer to form a multimodal instance.Source: §2.2.2 · pp. 5–6

2.3 Quality Control

  • Quality control removes failed entity masking, scores image-question relevance and visual reasoning, and filters image queries that cannot be answered from the image and captions.Captions are available during validation to reduce false negatives caused by missing world knowledge.Source: §2.3 · p. 6

3 Trajectory Generation and Post-Training

WebWatcher uses automatically generated, quality-filtered tool-use trajectories for supervised fine-tuning (SFT) cold start, then applies reinforcement learning (RL) to further optimize tool use and decision-making.

3.1.1 Multimodal Tools

  • The multimodal toolset combines image and text search, webpage visits, code interpretation, and OCR to support visual understanding, information retrieval, navigation, and computation.Web Image Search returns relevant images, captions, and webpage URLs; Web Text Search retrieves titles and URLs, while Visit summarizes pages for a specified goal.Source: §3.1.1 · p. 6

3.1.2 Automated Trajectory Annotation

  • GPT-4o generates ReAct-style trajectories for BrowseComp-VL questions as iterative thought, tool-action, and observation cycles, ending with a final answer.Actions can retrieve external information, navigate webpages, or perform computations; the Finish action returns the answer and terminates the episode.Source: §3.1.2 · pp. 6–7

3.1.3 Trajectory Filtering and Quality Assurance

  • Three-stage filtering retains trajectories with ground-truth-matching answers, GPT-4o-verified consistent steps, and at least three tool calls.The consistency check discards hallucinated content, contradictions, and unjustified tool calls to avoid rewarding lucky guesses rather than meaningful tool use.Source: §3.1.3 · p. 7

3.2 Supervised Fine-Tuning as Cold Start

  • SFT trains WebWatcher on filtered trajectories to predict each next tool-use action from the input image, question, and prior actions and observations.As a cold start, SFT teaches meaningful tool use and structured, multi-step reasoning.Source: §3.2 · p. 7

3.3 Reinforcement Learning

  • GRPO refines the SFT policy using group-relative trajectory returns, normalized advantages, and a clipped objective with a KL penalty to stabilize updates and encourage exploration.Rewards combine a binary tool-call format score and an LLM-graded semantic accuracy score, with w = 0.2 balancing them; terminal rewards are ranked across rollout groups of N = 16 trajectories.Source: §3.3 · pp. 7–8

4 Experiments

WebWatcher-32B achieves strong results across the reported VQA benchmarks, including a new state-of-the-art HLE average score. Analyses show benchmark-adaptive tool use, benefits from SFT cold starts, and rising HLE Pass@k with more attempts.

4.1 Setup

  • 8,000 high-quality tool-use trajectories formed the SFT set, with 2,000 VQA samples reserved for GRPO.Data sources were BrowseComp-VL, 5,000 long-tail VQA examples, and hard VQA, in a final 5:3:2 ratio; SFT and RL used Qwen2.5-VL-7B and Qwen2.5-VL-32B across five benchmarks.Source: §4.1 · p. 8

4.2 Main Results

  • 18.2% average on HLE makes WebWatcher-32B a new state-of-the-art, above Gemini-2.5-Pro at 15.8% and o4-mini at 16.0%.The passage also reports particular gains in Biology and Physics and says WebWatcher-32B surpasses the GPT-4o-based OmniSearch baseline by a substantial margin.Source: Table 1 · p. 9, §4.2 · pp. 9–10
  • 58.7% on LiveVQA and 55.3% on MMSearch, alongside 52.6% on SimpleVQA, show WebWatcher-32B’s strong results across four challenging benchmarks.BrowseComp-VL requires multi-page browsing and fine-grained visual grounding, and most baselines score below 20%.Source: Table 2 · p. 10, §4.2 · p. 10

4.3 Analysis

  • 62% of BrowseComp-VL tool interactions use Web Text Search, while visual benchmarks shift Web Image Search to around one-third or more of calls.Across benchmarks, Code Interpreter is reserved for genuine calculation, and the tool mix tracks task demands rather than defaulting to one strategy.Source: Figure 5 · p. 11, §4.3 · p. 11
  • 13.6% Pass@1 on HLE rises to 20.3% at k=3, 35.7% at k=16, and 41.9% at k=32.The curve keeps increasing with slower gains beyond k=8; the authors suggest 8–16 roll-outs to balance accuracy and compute, attributing gains to informative, decorrelated trajectories.Source: §4.3 · pp. 11–12
  • Cold-start SFT lifts initial scores to 0.12 on HLE, 0.30 on BrowseComp-VL, and 0.45 on LiveVQA, whereas instruction-only initialization stays near zero for many steps.Afterward HLE and BrowseComp-VL oscillate without clear upward drift, while LiveVQA maintains a 0.06–0.18 margin over the instruction-start baseline; larger-model CoT injection destabilized training.Source: §4.3 · p. 11, Figure 6 · p. 12

5 Related Work

The related work traces a shift toward deep-research agents while identifying their text-bound focus and the lack of VQA benchmarks combining multi-step reasoning, cross-modal integration, scale, and automation.

Deep Research

Multimodal VQA Benchmark

  • Existing VQA benchmarks often lack integrated multimodal reasoning and planning; newer datasets remain limited by format, scale, scope, or planning structure (Chen et al., 2024; Ying et al., 2024; Burgess et al., 2025; Zhang et al., 2025; Chen et al., 2025).MMMU and MMMU-Pro surface MLLM performance limits on domain-specific and multi-image tasks, but no existing benchmark covers all four requirements: multi-step reasoning, cross-modal integration, large scale, and automated quality control (Yue et al., 2024a;b).Source: §5 · pp. 12–13
  • The authors introduce a large-scale automated VQA benchmark for planning-oriented, multi-hop, context-rich multimodal reasoning and scalable evaluation of goal-directed agent behavior.The dataset is presented as supporting flexible agent behavior evaluation.Source: §5 · p. 13

6 Conclusion · A Symbol Definition

WebWatcher combines vision-language reasoning and multi-tool interaction with a new multimodal benchmark, scalable data pipelines, trajectory-based training, and test-time scaling. The appendix also defines mathematical symbols used in the paper.

6 Conclusion

  • WebWatcher unifies complex vision-language reasoning with multi-tool interaction as a framework for multimodal DeepResearch.Source: §6 · p. 13
  • BrowseComp-VL provides a challenging dataset for in-depth multimodal reasoning and strategic planning.The authors also introduce a scalable pipeline that transforms complex textual QA examples into VQA.Source: §6 · p. 13
  • An automated pipeline generates action-observation trajectories to provide high-quality supervision for cold start training and GRPO.Source: §6 · p. 13
  • A test-time scaling strategy further enhances evaluation.Source: §6 · p. 13
  • WebWatcher establishes a foundation for autonomous, flexible multimodal DeepResearch agents intended to solve real-world problems through deep reasoning.Source: §6 · p. 13

A Symbol Definition

  • The appendix includes a table defining mathematical symbols and their meanings.Source: Table 3 · p. 14

B Categories of BrowseComp-VL · C Prompts

BrowseComp-VL organizes its benchmark by subject area, while its prompts support answer verification for reject-sampled trajectories and assess the reasoning and tool use in model interactions.

B Categories of BrowseComp-VL

C Prompts

  • For reject-sampling trajectories, a prompt asks a model to judge predicted answers against the question and standard answer, returning only A, B, or C.The prompt is used to obtain trajectories with correct responses through reject sampling.Source: C Prompts · pp. 15–16
  • Near-matching numerical answers count as correct, wrong numbers as incorrect, and vague ranges as not attempted..Source: C Prompts · p. 15
  • Predictions need only provide the portion of a detailed standard answer asked for, and missing details inferable from the question count as correct.Source: C Prompts · p. 16
  • A second prompt guides GPT-4o to assess trajectory logic and task fit, judging tool calls for nonredundancy, goal alignment, and concise, evidence-based reasoning.The rubric also checks that calls are necessary, efficient, precise, and non-redundant, and that reasoning avoids guessing or fabrication.Source: C Prompts · p. 16
  • The tool-call rubric compares the user’s question with the generated tool-call and thought snippet, accepting only when all criteria are met and otherwise flagging shortcomings.Source: C Prompts · p. 16

D Experimental Details · D.1 Tool Definition · D.2 ReAct Trajectories

Appendix D defines the ReAct tools through structured prompts specifying callable formats and semantic capabilities, then describes the Qwen-Agent implementation and trajectory format with a 15-call limit.

D.1 Tool Definition

  • The ReAct framework uses structured prompts to define each tool’s callable format and semantic capability, enabling appropriate invocation within <tool_call> blocks.Source: §D.1 · p. 17
  • The Python tool executes code for calculation, data analysis, or content extraction and requires a code string.Source: §D.1 · p. 17
  • Google text search retrieves the top 10 text excerpts using one or more required search queries.Source: §D.1 · p. 17
  • Google image search retrieves the top 10 images and descriptions from an image URL and should be used only once.Source: §D.1 · p. 17
  • The webpage tool visits a required URL and returns a summary guided by the required goal describing the information sought.Source: §D.1 · p. 17
  • The OCR tool extracts image text with an internal OCR engine, supporting charts, screenshots, and scanned documents via a required image URL.Source: §D.1 · p. 17

D.2 ReAct Trajectories

  • The ReAct framework is implemented through Qwen-Agent and limits trajectories to no more than 15 tool calls.Source: §D.2 · p. 17
  • A trajectory cycles through <think>, <tool_call>, and <tool_response> steps before ending with an <answer>.Source: §D.2 · p. 18

D.3 Training Details

Training uses Llama-Factory for supervised fine-tuning and Verl (Sheng et al., 2025) for reinforcement learning, with separately specified optimization settings for each stage.

D.3 Training Details

  • SFT uses Llama-Factory with batch size 32, learning rate 5e-6 (minimum 1e-10), warmup plus cosine decay, and weight decay 0.1.Source: §D.3 · p. 18
  • RL uses Verl (Sheng et al., 2025) with 8 rollouts per group, temperature 1.0, top_p = 1.0, batch size 128, mini batch size 32, and learning rate 1e-6.Source: §D.3 · p. 18

D.4 Benchmarks

The evaluation spans five challenging benchmarks, using subsets to assess multimodal information retrieval and visual-textual reasoning.

D.4 Benchmarks

  • BrowseComp-VL uses 300 manually verified instances—100 from Level 1 and 200 from Level 2—and demands planning and external-tool proficiency.PhD-level AI experts manually verified all examples for accuracy and consistency.Source: §D.4 · p. 18
  • HLE (Phan et al., 2025) comprises 2,500 expert-written cross-disciplinary questions; evaluation uses 330 multimodal questions requiring evidence synthesis and abstract reasoning.The benchmark includes science, engineering, and humanities questions that go beyond simple retrieval, often involving obscure or fragmented sources.Source: §D.4 · p. 18
  • LiveVQA (Fu et al., 2025) contains 3,602 multi-hop questions about recent global news across six sources and fourteen topics; evaluation uses 300 examples.It assesses answering questions grounded in up-to-date visual knowledge.Source: §D.4 · p. 18
  • SimpleVQA (Cheng et al., 2025) has 2,025 English and Chinese factual VQA examples; evaluation samples 300 from its 1,013 English question-answer pairs.The benchmark combines curated image-question pairs from recent VQA datasets with expert-annotated web images.Source: §D.4 · p. 18
  • MMSearch (Jiang et al., 2024) contains 300 manually curated examples across 14 subdomains, and evaluation uses its 171 image-paired examples.The benchmark covers recent news and rare knowledge.Source: §D.4 · p. 18

D.5 Evaluation Prompt

The section specifies HLE’s response-accuracy scoring prompt: extract a final answer and judge whether it matches the supplied correct answer.

D.5 Evaluation Prompt

  • The official HLE prompt judges whether an extracted final answer matches the precise correct answer, allowing small numerical margins and requiring only a brief explanation of differences or equivalence.If no clear final answer is present, use None.Source: §D.5 · pp. 18–19
  • The evaluation response includes a confidence score from 0%–100%, defaulting to 100% when none is provided.The specified output also includes the extracted answer, brief comparison, and correct yes/no judgment.Source: §D.5 · p. 19

D.6 Case Study

The case study asks an agent to read a KenKen puzzle from an image and report its top row. The agent returns 4312, but its stated verification is internally inconsistent and does not establish a valid solution.

D.6 Case Study

  • After searching for KenKen materials, the agent treats the image as a 4×4 puzzle using 1–4 without row or column repeats and identifies cages 8+, 8*, 6*, 2+, 4*, 8+, and 4+.It plans to use OCR to extract the grid and cage layout.Source: §D.6 · pp. 20–21
  • The verification output is false for the first cage, and the later explanation contradicts the displayed grid by changing its top-left values from 4 and 3 to 4 and 4.The displayed grid’s top row is 4, 3, 1, 2, so the corrected 4 + 4 check does not validate that grid.Source: §D.6 · pp. 21–22
  • The agent’s final answer gives the puzzle’s top row as 4, 3, 1, 2 (4312).It also says all checks pass, despite reporting that the top-left cage check failed.Source: §D.6 · p. 22

Related papers

Loading 2508.05748v3…