PaperlayerView exact revision on arXiv

Source-linked AI summary

WebShaper: Agentically Data Synthesizing via Information-Seeking Formalization

Zhengwei Tao, Jialong Wu, Wenbiao Yin, Junkai Zhang, Baixuan Li, Haiyang Shen, Kuan Li, Liwen Zhang, Xinyu Wang, Yong Jiang, Pengjun Xie, Fei Huang, Jingren Zhou

arXiv:2507.15061v1cs.CLcs.AI

TL;DR

High-quality training data for information-seeking agents is sparse and difficult to construct. WebShaper formalizes information-seeking tasks with set theory and Knowledge Projections, and achieves the best performance among open-source information-seeking agents on GAIA and WebWalkerQA.

  • Problem

    High-quality information-seeking training data is sparse and difficult to construct, while existing synthesis methods derive questions from collected information structures.

    Source: §1 · p. 2

  • Method

    WebShaper formalizes information-seeking tasks with set theory and Knowledge Projections to control task complexity and reasoning paths.

    Source: §1 · p. 3, §2 · p. 4

  • Results

    WebShaper achieves the best performance among open-source information-seeking agents on GAIA and WebWalkerQA.

    Source: §1 · p. 3

  • Takeaways & Limitations

    The resulting data contains varied information-seeking forms that can stimulate different information-seeking capabilities.

    Source: §4.3.7 · p. 14

    The expansion strategy can include redundant sentences about constants that do not increase the task’s reasoning chain.

    Source: §3.2.2 · p. 8

Abstract

from arXiv · show

The advent of Large Language Model (LLM)-powered agents has revolutionized artificial intelligence by enabling solutions to complex, open-ended tasks through web-based information-seeking (IS) capabilities. The scarcity of high-quality training data has limited the development of IS agents. Existing approaches typically adopt an information-driven paradigm that first collects web data and then generates questions based on the retrieval. However, this may lead to inconsistency between information structure and reasoning structure, question and answer. To mitigate, we propose a formalization-driven IS data synthesis framework WebShaper to construct a dataset. WebShaper systematically formalizes IS tasks through set theory. Central to the formalization is the concept of Knowledge Projections (KP), which enables precise control over reasoning structure by KP operation compositions. During synthesis, we begin by creating seed tasks, then use a multi-step expansion process. At each step, an agentic Expander expands the current formal question more complex with retrieval and validation tools based on our formalization. We train our model on the synthesized dataset. Experiment results demonstrate that WebShaper achieves state-of-the-art performance among open-sourced IS agents on GAIA and WebWalkerQA benchmarks.

1 Introduction

WebShaper formalizes information-seeking tasks before retrieval to guide synthesis, aiming to improve task coverage, controllability, and question-answer consistency. The paper reports that agents trained on its dataset outperform open-source baselines on GAIA and WebWalkerQA.

Motivation

  • Existing information-driven synthesis can produce inconsistent reasoning or incorrect answers and redundant, homogeneous information structures that limit knowledge coverage.These methods retrieve and organize information before using an LLM to generate questions from it.Source: §1 · p. 2, Figure 2 · p. 2

Formalization-driven synthesis

  • WebShaper formalizes IS tasks before information collection, guiding synthesis with requirements that specify reasoning structures and complexity.Its set-theoretic formalization treats tasks as compositions of Knowledge Projections, which are sets of entities.Source: §1 · pp. 2–3

Framework advantages

  • The framework aims to broaden task coverage, control task complexity, and improve consistency between information and reasoning structures and between questions and answers.Formalization parameters specify reasoning structure and complexity, while interpretable, verifiable representations support consistency.Source: §1 · p. 3

Expander

  • An agentic Expander starts from seed tasks and iteratively collects knowledge, constructs and validates formalized problems, and generates questions.Layer-wise expansion increases complexity while minimizing redundancy and preventing reasoning shortcuts.Source: §1 · p. 3

Experiments

  • WebShaper achieves the best performance among open-source IS agents on GAIA and WebWalkerQA.The paper also reports that WebShaper-trained models consistently outperform baselines across multiple benchmarks.Source: §1 · p. 3

2 Information-Seeking Formalization

The section formalizes information-seeking as finding a target entity set from prompted facts, using Knowledge Projections (KPs) and compositions of union and intersection to represent conditions.

2 Information-Seeking Formalization

  • An information-seeking task searches for knowledge and facts prompted by given facts to locate the answer entity set T.In the example, solving the task involves identifying the team, then finding players by season and birth decade.Source: §2 · p. 4
  • A Knowledge Projection R(V) returns entities related to members of V under relation R, making it the basic unit of an information-seeking task.Entities form a universal set E, and R is a subspace of entity pairs; for bornIn, R({90s}) denotes people born in the 1990s.Source: §2 · p. 4
  • R-Union represents broader or uncertain conditions by combining projected sets, such as players who played in any year from 2000 through 2010.The section motivates this operation with cases where the target's exact year is unknown, but a year range is known.Source: §2 · p. 4
  • Intersection represents targets that satisfy multiple conditions simultaneously, such as players who played in 2000 and were born in the 1990s.The example combines projections for playAt and bornIn.Source: §2 · pp. 4–5
  • The formalization defines a target set T, allows it to be recursively derived by replacing entity sets with other target sets, and frames the task as finding the entities T contains.The paper says the earlier question example can then be formalized using these definitions.Source: §2 · p. 5

3 Data Synthesis

WebShaper synthesizes information-seeking data by formalizing tasks as compositions of knowledge projections, then expanding seed questions through retrieval and validation. The pipeline also constructs task-solving trajectories and trains an agent with supervised fine-tuning followed by reinforcement learning.

3 Data Synthesis

  • The pipeline starts with seed tasks and uses formalization-guided, multi-step expansion in which an agent retrieves knowledge and constructs and validates new formal questions.Expansion proceeds layer by layer to reduce redundancy and reasoning shortcuts.Source: §3 · p. 5

3.1 Seed Question Construction

  • 18k seed questions were built from LLM-generated, article-grounded question-answer pairs and filtered by requiring at least one correct answer across five rollouts.The team generated pairs from content collected through random walks over a hyperlink-preserving offline Wikipedia database; filtering used WebDancer (Wu et al., 2025a) and QwQ (Team, 2025).Source: §3.1 · pp. 5–6

3.2 Agentic Expansion

  • Agentic Expansion uses a KP representation, layer-wise expansion, and an autonomous Expander to retrieve information and validate generated questions.The approach addresses the complexity of recursive formalizations and intermediate knowledge filtering and selection.Source: §3.2 · p. 6

3.2.1 KP Representation

  • KP Representation encodes projections as triplets, expresses intersections as triplet lists, merges R-Union sets, and flattens recursive projections through shared variables.A triplet [X, r, S] identifies a relation and its input and output sets; variables represent unspecified sets, while constants specify elements.Source: §3.2.1 · pp. 6–7

3.2.2 Layer-wise Expansion Strategy

  • Layer-wise expansion replaces each leaf constant with a variable connected to newly constructed facts, preserving the answer while avoiding redundancy and reasoning shortcuts.The number of expansion layers controls task coverage and difficulty.Source: §3.2.2 · pp. 7–8

3.2.3 Expander Agent

  • The Expander follows ReAct (Yao et al., 2023), using search and summarization to retrieve knowledge and validate sub-questions for formal consistency and answerability.Validation rejects a sub-question if QwQ answers it with the same value as the input constant.Source: §3.2.3 · pp. 8–9

3.3 Trajectory Construction

  • Task-completing trajectories use a QwQ-based ReAct agent with Search and Visit, are filtered for correctness and quality, and yield 5,000 trajectories.Five rollouts are performed per question; filters remove tool-call errors, hallucinated observations, and severe repetition.Source: §3.3 · p. 9

3.4 Agent Training

  • Agent training uses supervised fine-tuning followed by GRPO reinforcement learning, with the supervised loss masked on observations.GRPO is applied to sampled rollouts for question-answer pairs (Shao et al., 2024).Source: §3.4 · p. 9

4 Experiments

Experiments evaluate WebShaper on GAIA and WebWalkerQA using LLM-as-Judges Pass@1, and examine its dataset, training effects, synthesis design, and reasoning trajectories.

4.2 Main Results

  • 52.2 on WebWalkerQA is WebShaper’s highest score; on GAIA, WebShaper on Qwen-2.5-72B exceeds second-best WebSailor by 4.7, and WebShaper leads open-source methods on both benchmarks.WebShaper performs best across backbone settings and is the only open-source method reported to score above 60 points, close to the OpenAI DR system.Source: Table 1 · p. 10, §4.2 · p. 11

4.3.1 Data Statistics

  • WebShaper’s dataset covers diverse topics with balanced representation across major domains such as Sports, Politics, and Entertainment.The authors attribute breadth to seed-task topic diversity and further expansion, and say the balance supports domain-agnostic, multi-domain training.Source: §4.3.1 · p. 11, Figure 5 · p. 11

4.3.2 Data Comparison

  • WebShaper achieves the highest average GAIA scores on Qwen-2.5-32B, Qwen-2.5-72B, and QwQ-32B, outperforming baseline datasets including WebWalkerQA and MHQA.The comparison samples 5,000 examples per dataset and fine-tunes the three backbones; gains also hold for similar-sized models.Source: §4.3.2 · p. 11, Table 2 · p. 12

4.3.3 RL Stimulation

  • RL raises GAIA Pass@1 by +7.8 points for Qwen2.5-32B and +13.5 points for Qwen2.5-72B, and also improves WebWalkerQA performance.The authors attribute the gains to formalized task breadth and complexity prompting iterative information queries during RL.Source: §4.3.3 · p. 12, Figure 6 · p. 12

4.3.4 Formalization

  • Formalization-based synthesis outperforms natural-language expansion across all tested backbones and supports more task forms with fewer error-propagated inconsistencies in question-answer pairs.The compared datasets were used to fine-tune Qwen2.5-32B, Qwen2.5-72B, and QwQ under otherwise unchanged training settings.Source: §4.3.4 · pp. 12–13, Figure 7 · p. 13

4.3.5 Layer-wise Expansion Strategy

  • Layer-wise expansion outperforms sequential expansion across all tested base models, mitigating redundancy and reasoning shortcuts through controllable structure.Both variants use the same Expander and task formalization.Source: §4.3.5 · p. 13, Figure 7 · p. 13

4.3.6 Tool Call Analysis

  • WebShaper has a pronounced long tail: tasks requiring over 3 search operations occur at 3–4× the rate in E2HQA and MHQA, with non-zero proportions through 30 total tool calls.Source: §4.3.6 · p. 13

4.3.7 Case Study

  • The case study shows synthesized tasks avoid redundancy and reasoning shortcuts, requiring information seeking across variables; R-Union summarizes distributed web content and increases difficulty.The paper says formalization enables varied information-seeking forms that stimulate different model capabilities.Source: §4.3.7 · p. 14, Figure 9 · p. 14

5 Related Work

Related work includes information-driven synthesis for information-seeking agents and formalization-based synthesis in mathematics and knowledge-base question answering. WebShaper applies formalization to information-seeking tasks through set theory.

5.1 Information-Seeking Data Synthesis

  • Prior information-seeking synthesis includes WebWalkerQA’s linear navigation chains (Wu et al., 2025b), CRAWLQA’s information aggregation (Wu et al., 2025a), and SailorFog-QA’s entity-coreference fuzzy reasoning (Li et al., 2025a).These methods focus on retrieving and connecting knowledge, while WebShaper emphasizes structural representation and principled modeling of the QA process.Source: §5.1 · p. 14
  • Information-seeking agents rely on limited or simplistic datasets, while GAIA (Mialon et al., 2023), BrowseComp (Wei et al., 2025), and BrowseComp-zh (Zhou et al., 2025) provide only test sets.The test-only benchmarks therefore restrict their applicability for training agents.Source: §5.1 · p. 14

5.2 Formalization-based Data Synthesis

  • Mathematical data synthesis formalizes problems and proofs, including subgoal decomposition, validation, and iterative expansion of prover capability.DeepSeek-MathProver converts competition problems into formal statements and verifies LLM-generated proofs in Lean 4 (Xin et al., 2024); DeepSeek-MathProverV2 decomposes proofs into subgoals (Ren et al., 2025). Leang et al. (2025) use formal and natural-language conversions and verification to synthesize theorem-prover-judge data, while Goedel-Prover iteratively trains provers to prove previously unproved statements (Lin et al., 2025).Source: §5.2 · p. 15
  • For knowledge-base question answering, LACT formalizes queries with first-order logic and binary-tree decomposition, then trains using an easy-to-hard curriculum (Xia et al., 2025).WebShaper instead establishes information-seeking formalization via set theory.Source: §5.2 · p. 15

6 Conclusion

WebShaper formalizes information-seeking tasks using set theory and Knowledge Projections, then expands them through retrieval and validation. The framework achieves state-of-the-art performance on GAIA and WebWalkerQA while enabling deliberate control over task design.

6 Conclusion

  • WebShaper achieves state-of-the-art performance on GAIA and WebWalkerQA and enables deliberate engineering of cognitive challenges for information-seeking agents.The conclusion frames this as a shift from reactive information organization to proactive task specification.Source: §6 · p. 15
  • Set theory formalization and compositions of Knowledge Projections enable precise engineering of information-seeking tasks’ reasoning structures and complexity.The framework is presented as addressing structural inconsistencies, controllability, diversity, and coverage limitations of information-driven approaches.Source: §6 · p. 15
  • The agentic Expander uses layer-wise expansion with autonomous knowledge retrieval and rigorous validation to minimize redundancy and prevent reasoning shortcuts.Source: §6 · p. 15

A Agent Details

WebComposer supports information seeking through search and targeted page visits, with search retrieving candidate documents and visit extracting information relevant to a specified goal.

Search

  • Search queries Google in parallel and returns the top 10 results per query, each with a title, snippet, and URL.Search is one of WebComposer’s two information-seeking tools.Source: A Agent Details · p. 19

Visit

  • Visit retrieves a page’s full content using Jina, then Qwen-2.5-72B summarizes information relevant to its designated visit goal.Each page is paired with a specific visit goal.Source: A Agent Details · p. 19

B Training Details · B.1 SFT · B.2 RL

Training details specify distinct SFT and RL configurations, including optimizer settings, schedules, rollout parameters, and batch sizes.

B.1 SFT

  • SFT uses a batch size of 32 and a learning rate of 5e-6.Source: §B.1 · p. 19
  • SFT applies warmup with cosine decay and a weight decay of 0.1.Source: §B.1 · p. 19

B.2 RL

C Case Study

A representative KIMI-Researcher case contains redundant SSCS-related constants and a reasoning shortcut to the target, despite having a single-hop reasoning path.

C Case Study

  • The KIMI-Researcher example includes redundant SSCS-related constants and a shortcut directly to the target, while its underlying reasoning is single-hop.SSCS stands for “Strange Stories from a Chinese Studio.”Source: C Case Study · p. 19, Figure 10 · p. 19

D Broader Impact

WebShaper combines explicit task formalization with agent-driven synthesis to construct training data for intelligent agents. Its broader implications span controllable agent training data and AI research beyond the constraints of naturally occurring human-generated data.

Implications in Agent Training Data Synthesis

  • Formal task structures enable precise control over training-data properties by decoupling task definitions from data generation.Source: D Broader Impact · p. 20
  • Tasks can be parameterized by difficulty, modality, or compositional structure to expose agents to controlled challenges and stress-test edge cases.Source: D Broader Impact · p. 20
  • Formal task models specify synthesized data, reducing noise and improving consistency for reliability- and safety-critical applications such as autonomous systems or medical AI.Source: D Broader Impact · p. 20
  • Agent-driven synthesis translates formal task representations into training instances without laborious manual annotation or heuristic pipelines, reducing computational overhead while preserving task fidelity.Source: D Broader Impact · p. 20

Implications for AI Research and Development

  • By formalizing tasks independently of human behavioral priors, the framework can generate training data beyond natural-data constraints for domains requiring superhuman reasoning.Examples include advanced scientific modeling and combinatorial optimization.Source: D Broader Impact · p. 20
  • Formal task representations abstract away domain-specific noise, supporting learning of invariant principles across diverse contexts.Source: D Broader Impact · p. 20

Related papers

Loading 2507.15061v1…