PaperlayerView exact revision on arXiv

Source-linked AI summary

WebAggregator: Enhancing Compositional Reasoning Capabilities of Deep Research Agent Foundation Models

Rui Wang, Ce Zhang, Jun-Yu Ma, Jianshu Zhang, Hongru Wang, Yi Chen, Boyang Xue, Tianqing Fang, Zhisong Zhang, Hongming Zhang, Haitao Mi, Dong Yu, Kam-Fai Wong

arXiv:2510.14438v2cs.CL

TL;DR

Existing resources emphasize shallow information seeking, while deep research requires integrating distributed materials into coherent insights. WebAggregator gathers interconnected web information and synthesizes reasoning-intensive questions for training. Its models improve performance across benchmarks, and WebAggregatorQA shows that reference retrieval alone does not ensure success.

  • Problem

    Existing resources often emphasize shallow hops or entity seeking, while deep research requires integrating distributed materials into coherent insights.

    Source: §1 · p. 1

  • Method

    WebAggregator explores interconnected web resources and synthesizes verifiable, reasoning-intensive questions through a multi-hop composition taxonomy and automated quality control.

    Source: §1 · p. 2, §2 · p. 3

  • Results

    WebAggregator-8B significantly surpasses GPT-4.1, while the 32B variant outperforms all strong baselines, with gains reported across diverse benchmarks.

    Source: §1 · p. 2, §6 · p. 9

  • Takeaways & Limitations

    Accurate retrieval alone does not guarantee task success because current models still struggle to integrate distributed information.

    Source: §6 · p. 9

    The pipeline involves execution time, though the authors report it remains highly cost-effective compared with expert human annotation.

    Source: Limitations · p. 10

Abstract

from arXiv · show

The hallmark of Deep Research agents lies in compositional reasoning, the capacity to aggregate distributed, heterogeneous information into coherent logical insights. However, current agentic systems are often retrieval-heavy but reasoning-light, where success is predominantly determined by simple entity-seeking rather than the multi-step aggregation of scattered evidence. To address this, we propose a data synthesis pipeline WebAggregator, designed to shift the agentic paradigm from retrieval-centric to compositional aggregation. Our approach first employs Proactive Explorer to collect interconnected knowledge, then Compositional Logic Proposer to weave knowledge into complex questions using over 12 composition guidelines derived from a rigorous deconstruction of the Deep Research problem setting. By leveraging 10K verifiable QA pairs grounded on 50K websites, we curate a high-quality SFT dataset via rejection sampling. Fine-tuning on this corpus fundamentally transforms agent behavior, fostering deliberate composition reasoning and reduced tool redundancy. The resulting WebAggregator-32B surpasses GPT-4.1 and matches Claude-3.7-Sonnet on GAIA, WebWalkerQA, and XBench. To address the lack of benchmarks that emphasize both reasoning and retrieval, we introduce the WebAggregatorQA testbed, which reveals that even with perfect retrieval, top-tier models still underperformed. These results demonstrate that compositional reasoning, not retrieval, is the true performance ceiling for next-generation research agents.

1 Introduction

Existing resources often emphasize shallow information seeking rather than integrating distributed materials into coherent insights. WebAggregator constructs reasoning-intensive questions from interconnected web information, and its training data improves agent performance while WebAggregatorQA reveals that accurate retrieval alone is insufficient.

1 Introduction

  • Existing multi-hop datasets offer shallow logical hops, while web-agent benchmarks prioritize entity-seeking puzzles on static pages (Yang et al., 2018; Shi et al., 2025; Wu et al., 2025b).The paper identifies limited environmental complexity and tasks weighted toward search effort over reasoning depth; 30.79% of WebWalkerQA and 43.2% of TaskCraft tasks require only simple entity localization.Source: §1 · p. 1
  • WebAggregator explores interconnected resources from anchor URLs, then synthesizes multi-hop aggregation tasks using a four-category taxonomy refined into 12 subtypes.The taxonomy covers Element, Set, Scientific Analysis, and Temporal Reasoning.Source: §1 · p. 2, Figure 3 · p. 3
  • WebAggregatorQA contains approximately 10K questions designed to require compositional reasoning even when necessary references are provided.Rejection sampling and quality control are used to train the WebAggregator model family on tasks intended to resist direct extraction and shallow pattern matching.Source: §1 · p. 2
  • WebAggregator-8B significantly surpasses GPT-4.1, while the 32B variant outperforms all strong baselines on GAIA, WebWalkerQA, and XBench.Fine-tuning also encourages deeper reasoning and reduces redundant tool use.Source: §1 · p. 2
  • 28.3% is Claude-3.7-Sonnet’s score on WebAggregatorQA, rising only to 42.1% with full reference context.The analysis attributes the remaining difficulty to gaps such as ineffective self-refinement and faulty information utilization.Source: §1 · p. 2
  • The authors argue that future progress must move beyond retrieval-centric paradigms toward rigorous compositional reasoning for complex tasks.WebAggregatorQA is presented as a testbed for that challenge.Source: §1 · p. 3

2 WebAggregator

WebAggregator frames training-task synthesis as an agentic process that gathers interconnected evidence from real web resources and turns it into reasoning-intensive questions. Its stages combine exploration, guided question composition, quality control, trajectory sampling, and test-set construction.

2 WebAggregator

  • WebAggregator turns anchor-URL exploration into reasoning-intensive QA through Proactive Explorer and Compositional Logic Proposer, followed by automated quality control.The pipeline frames synthesis as an agentic task and aims to produce scalable, challenging QA pairs grounded in real web resources.Source: §2 · p. 3

2.1 Anchor URL Collection

  • From 5,000 topic-diverse queries, Google Search yielded over 160,000 anchor links spanning 11+ domains.The queries came from QA and scientific datasets.Source: §2.1 · p. 3

2.2.1 Proactive Explorer

  • Proactive Explorer navigates from an anchor URL across text, files, images, and dynamic elements, while a minimum number of page visits controls task difficulty and knowledge coverage.Across 5,296 trajectories, 48.36% used four distinct tools, 28.55% used five, and 13.41% used six or more; every trajectory used Search, Visit, and ScreenShot.Source: §2.2.1 · pp. 3–4

2.2.2 Compositional Logic Proposer

  • Compositional Logic Proposer uses four categories—Element, Set, Scientific Analysis, and Temporal Reasoning—refined into 12 flexible subtypes for context-specific questions.The guides are not rigid templates; Element calculations can, for example, be instantiated as addition, subtraction, or ratio calculation.Source: §2.2.2 · p. 4

2.2.3 Quality Control

  • Quality control combines proposer self-refinement with a second agent solving tasks and checking reference URLs, filtering out 11.72% of initial samples.The process also balances domains and compositional types by normalizing anchor-domain distribution and adjusting prompts for underrepresented operations.Source: §2.2.3 · p. 4

2.2.4 Trajectory Sampling

  • GPT-4.1 with SmolAgents generated training trajectories, and rejection sampling retained 6,184 that had correct predictions and valid formats while keeping real-web observation anomalies.Anomalous observations such as page failures were retained to improve generalization to similar real-web situations.Source: §2.2.4 · p. 4

2.2.5 Statistics of WebAggregatorQA

  • WebAggregatorQA contains 9,883 tasks, including 200 reserved for testing, across 54,064 unique URLs and 12 domains; most construction trajectories took around 15 steps.The authors describe the step count as avoiding questions made from only a few reasoning steps.Source: §2.2.5 · pp. 4–5

2.3 WebAggregatorQA testset for Compositional Reasoning Evaluation

  • The manually reviewed WebAggregatorQA test set contains 159 text and multimodal samples, categorized as Level 1 (24), Level 2 (99), and Level 3 (36).At least two annotators aligned on 155 samples; four difficult samples were retained after their references and questions were verified.Source: §2.3 · p. 5

3 Experiments

Experiments fine-tune Qwen2.5 and Qwen3 models on WebAggregatorQA and evaluate them alongside foundation-model and agent baselines. The tuned 32B models surpass most strong baselines, while results on WebAggregatorQA expose substantial difficulty for current agents and a remaining gap for smaller models.

3.1 Experimental Setups

  • The evaluation compares varied Qwen2.5 and Qwen3 models fine-tuned on WebAggregatorQA with foundation-model and agent baselines across GAIA-text, XBench, WebWalkerQA, and WebAggregatorQA.The authors report pass@1 and pass@3 scores evaluated by GPT-4.1.Source: §3.1 · p. 6

3.2 Main Results

  • After WebAggregatorQA tuning, Qwen models improve across benchmarks, and the 32B variants surpass most strong baselines, including GPT-4.1 and WebShaper; WebAggregator-32B reaches 69.9% pass@3 on GAIA-text.In zero-shot settings, closed-source models surpass Qwen models.Source: §3.2 · p. 6
  • GPT-4.1-powered SmolAgents score 43.7% accuracy on GAIA-text but 25.8% on WebAggregatorQA, where the gap between Claude and GPT-4.1 is also smaller.The authors interpret this pattern as neither model solving the harder WebAggregatorQA questions effectively.Source: §3.2 · p. 6
  • Fine-tuning Qwen3-8B on 500 and 1,200 trajectories yields 36.9% and 38.83% accuracy on GAIA-text, respectively; the 8B-versus-32B gap persists on complex WebAggregatorQA tasks.WebAggregator-8B reaches parity with GPT-4.1 on GAIA-text and outperforms previous baselines on WebWalkerQA and XBench.Source: §3.2 · p. 6
  • Figure 5 summarizes task distributions by required tool and aggregation-operation counts, alongside the proportions of information sources and aggregation operations needed.Source: Figure 5 · p. 7

4 Analysis

WebAggregatorQA requires multi-source evidence and varied, often multiple composition operations, yet strong base models solve fewer than 7% of its questions. Analyses show that retrieval alone does not remove the reasoning bottleneck, while tuning shifts behavior toward reasoning and failures can cascade.

4.1 Requirements for Solving WebAggregatorQA

  • Fewer than 7% of WebAggregatorQA questions are solved by Claude-3.7 and GPT-4.1, while tasks require Search and Web Text and frequently Files.Tasks also require varied composition operations, often multiple operations.Source: §4.1 · p. 7

4.2 Challenges from Compositional Reasoning

  • Even after all reference URLs are visited, accuracy gains remain modest; WebAggregator-32B closes the retrieval–accuracy gap, but composition logic remains a ceiling.Table 3 reports counts and accuracy for trajectories visiting all reference URLs and obtaining gold knowledge.Source: §4.2 · p. 7, Table 3 · p. 8
  • WebAggregatorQA tasks require more total steps but have lower tool-call density, indicating greater reliance on reasoning steps than acquiring external information.Tool-call density is defined as tool calls per step; Figure 6 compares steps and tool-use density across test sets.Source: §4.2 · p. 7, Figure 6 · p. 7

4.3 Behavior Evolution of WebAggregator

  • After tuning, Type B reasoning ratios rise and tool-call density falls; total token use grows 12–17%, driven exclusively by Type B reasoning.The analysis classifies internal thoughts as procedural/planning (Type A) or compositional reasoning (Type B); computation-intensive scientific analysis and set operations remain key challenges.Source: §4.3 · p. 8, Figure 7 · p. 8
  • Tuning also shifts behavior from concentrated, simple tool use toward a more diverse operational distribution and multi-step aggregation of scattered evidence.Before tuning, 40% of trajectories relied on only two tools.Source: §4.3 · p. 8

4.4 Screening of Failure Modes

  • Failure modes include missed sources, operational errors, faulty composition, accumulated inaccuracies, and ineffective refinement; initial composition errors can propagate and consume context.The cascade can increase the probability of hallucinations or timeouts.Source: §4.4 · pp. 8–9

5 Related Work

Prior datasets and benchmarks often emphasize static content, entity tracing, or information seeking over complex composition. Related work also explores training and test-time optimization for research agents.

Resources for Deep Research Agents

  • Existing multi-hop and synthesized agent datasets are limited by static content or insufficient goal-oriented depth, while some logical-flow methods emphasize entity tracing over knowledge composition.The cited approaches include tasks built from offline knowledge graphs and entity expansion.Source: §5 · p. 9

Benchmarking Deep Research Agents

  • Most deep-research benchmarks prioritize information seeking and reference filtering, with few assessing compositional reasoning; some are confined to Wikipedia or long sequences of simple actions.The passage identifies GAIA as a relevant benchmark that is becoming saturated by recent agents.Source: §5 · p. 9

Training and Optimization for Research Agents

  • Research-agent optimization includes rejection-sampling SFT, reinforcement learning with synthesized queries or execution-error supervision, and test-time self-evolving.Synthesized queries can provide process-level supervision for credit assignment; execution errors can supply corrective supervision for multi-turn tool robustness.Source: §5 · p. 9

6 Conclusion

WebAggregator bridges information seeking and compositional reasoning through WebAggregatorQA and its models. The authors report performance gains across benchmarks and identify integration of distributed information—not retrieval alone—as a bottleneck.

6 Conclusion

  • WebAggregator introduces a framework, WebAggregatorQA, and models to bridge information seeking and compositional reasoning in deep research agents.Source: §6 · p. 9
  • The authors report that verifiable, reasoning-intensive training data significantly boosts agent performance across diverse benchmarks.Source: §6 · p. 9
  • Accurate retrieval does not guarantee task success because current models struggle to integrate distributed information.Source: §6 · p. 9

Limitations

WebAggregator's construction requires environmental interaction and execution time, although it is described as cost-effective relative to expert annotation. The appendix passages also describe its agent tools and quality-control procedures.

Limitations

  • Data construction requires environmental interaction and execution time, though it remains cost-effective compared with human annotation requiring hours of expert labor per task.Source: Limitations · pp. 9–10

A Agent Structure

  • The code-based ReAct agent combines search, page parsing, dynamic interaction, file processing, and image captioning, within a fixed 30-step task budget.The implementation uses SmolAgents and DOM parsing for finer web interactions.Source: A Agent Structure · p. 13

B.1 More Explanations of Quality Control

  • Construction quality control uses iterative question refinement and source-alignment checks, filtering out about 11.72% of data; domain and operation sampling promote diversity.The process also uses a website keyword blacklist to exclude pages matching it or containing identified datasets from retrieval and evaluation.Source: §B.1 · p. 13

B.2.1 Data Collection and Verification

The test set was built from revised, topic-diverse seed tasks and checked for clarity, reliable evidence, and a unique answer. Its construction also aimed to increase reasoning complexity and information-source diversity, with agent-assisted verification and independent human cross-validation.

B.2.1 Data Collection and Verification

  • Annotators revised 200 topic-diverse tasks selected from WebAggregatorQA to seed the test set and improve coverage across domains.The section notes that a single annotator may struggle to generate diverse samples across multiple fields.Source: §B.2.1 · p. 13
  • Revised questions were required to be clear and unambiguous, with exactly one correct answer supported by validated, reliable, stable, and directly relevant URLs.Pages with strict CAPTCHA were replaced with more stable ones.Source: §B.2.1 · pp. 13, 15
  • The running example and multimodal test sample illustrate interaction with varied content, including file processing and independently locating an image for clues.Figure 10 notes that questions built from file knowledge also test file-processing abilities.Source: Figure 10 · p. 14, Figure 11 · p. 16
  • Task revisions sought more reasoning steps and varied information forms, with answers not directly available on a webpage.The accompanying examples include statistical analysis, and Table 4 presents aggregation operations as high-level guidance rather than low-level constraints.Source: §B.2.1 · p. 15, Table 4 · p. 15
  • GPT-4.1-assisted solving helped identify ambiguities, followed by human review and a second verification cycle before independent cross-validation.Figure 9 describes a more diverse and balanced distribution of tools after fine-tuning; trajectories using only two tools consistently use Search and Visit.Source: Figure 9 · p. 14, §B.2.1 · p. 15
  • 155 of 159 references received independent annotation agreement; four tasks were omitted because of excessive complexity.The construction involved three human annotators, and each sample required an average of three hours of annotation work.Source: §B.2.1 · pp. 13, 15, 16

B.3 Data Construction Prompt

The data-construction prompt guides agents to explore web sources and build challenging multi-hop questions requiring evidence from multiple pages. It specifies question-design constraints and offers examples of compositional operations, including statistical analysis and computation.

B.3 Data Construction Prompt

  • The information-gathering instructions call for exploring a starting URL, browsing at least the specified number of websites, and recording navigation and findings after each action.The prompt cautions against relying only on search queries or Wikipedia.Source: §B.3 · p. 17
  • Questions must require reasoning across multiple sources and must not be answerable by a simple search or a single page.They should be natural, concise, self-contained, and based on details from at least five to eight webpages.Source: §B.3 · p. 17
  • Questions should provide precise clues without naming sources, and answers should be concise, verifiable, stable over time, and derived rather than directly retrieved.The prompt rejects arbitrary combinations and requires the relevant numbers or elements to be discoverable from web content.Source: §B.3 · pp. 17, 19
  • The prompt requires at least one composition operation, such as statistical analysis of webpage data using mean, variance, or standard deviation.Examples include finding a median or comparing growth consistency and average annual percentage increases.Source: §B.3 · p. 18
  • The prompt includes sample questions involving correlation, trend forecasts, and comparison of temperature increases against the global average.It also gives examples involving historical motorcycle sales, exponential smoothing, and sports statistics.Source: §B.3 · pp. 18–19
  • Operation examples span element-wise calculations, list and set operations, membership checks, inverse questions, forecasting, and computation-intensive tasks.The examples include sums, averages, intersections, exclusions, trend prediction, and batch analysis using code.Source: §B.3 · p. 18

B.4 Prompt of Data Quality Checking Agent

The quality-checking prompts assess both evidence and answer quality, including source reliability, fidelity, uniqueness, and question clarity. They also specify checks for retrieval and compositional necessity and provide structured output fields.

B.4 Prompt of Data Quality Checking Agent

  • The question checks cover valid URLs, relevant evidence, self-containment, retrieval necessity, clarity, and temporal stability.The prompt defines temporal stability as an answer remaining consistent over time.Source: §B.4 · p. 20, §B.5 · p. 22
  • Composition necessity requires at least three different aggregation operations so the answer cannot be obtained by direct retrieval.The prompt treats this as a question-passing criterion.Source: §B.4 · p. 20
  • The answer-quality assessment checks information fidelity, ground-truth validity, and uniqueness or unambiguity.The prompts ask whether conflicting sources or precision differences could support multiple answers.Source: §B.4 · p. 20
  • Ground-truth answers should reflect authoritative sources, and numerical or statistical answers receive particular emphasis on verifiability.The prompt contrasts a validated answer with an example that contradicts scientific consensus.Source: §B.4 · p. 20
  • The checker receives a question, answer, and evidence URLs, then returns separate pass indicators for evidence, question, and answer alongside a domain label.The format specifies a one-word domain chosen from a listed set.Source: §B.4 · pp. 20–21

B.5 Prompt of Intergrated Data Quality Checking Tool

  • The integrated checking tool repeats checks for self-containment, retrieval necessity, compositional necessity, clarity, and temporal stability.Its prompt asks the checker to provide advice for improvements as well as verify each standard.Source: §B.5 · p. 22

C Data Contamination Analysis

The contamination analysis compares similarity between GAIA and a paraphrased version of GAIA against overlap between GAIA and WebAggregatorQA. The reported contrast is significant similarity for the paraphrase comparison but negligible overlap with WebAggregatorQA.

C Data Contamination Analysis

  • The analysis compared original and paraphrased GAIA questions, then sampled 100,000 WebAggregatorQA-training/GAIA pairs for cross-dataset similarity.It used Qwen3-Embedding-8B semantic similarity, ROUGE-L, and Jaccard similarity.Source: C Data Contamination Analysis · p. 23
  • GAIA had significant similarity with its paraphrased counterpart, while its overlap with WebAggregatorQA was negligible.The paper describes this as strong evidence that WebAggregatorQA is free from GAIA contamination.Source: C Data Contamination Analysis · p. 23

D Close Screening Agent Trajectories

D.1 Example 1

The three examples expose distinct compositional failures: faulty information extraction, inadequate iterative checking, and error accumulation across a multi-step data task.

Example 1

  • The set-filter-counting trajectory returned 2014 and 3, while the stated true answer was 2014&2018 and 2.The error analysis labels this a faulty composition.Source: Example 1 · p. 27

Example 2

  • The example is labeled “Underperformed Iterative Refinement”: although v60 was the next major patch on August 17, 2024, the agent proceeds without checking whether v70 is reasonable.Source: Example 2 · p. 28

D.3 Example 3

  • Pre-reducing the country list changed the final intersection and caused the World Bank indicator task to return an empty list.The task combines rankings across five indicators for 2014–2023.Source: §D.3 · pp. 29, 32

Related papers

Loading 2510.14438v2…