Source-linked AI summary
WebShaper: Agentically Data Synthesizing via Information-Seeking Formalization
Zhengwei Tao, Jialong Wu, Wenbiao Yin, Junkai Zhang, Baixuan Li, Haiyang Shen, Kuan Li, Liwen Zhang, Xinyu Wang, Yong Jiang, Pengjun Xie, Fei Huang, Jingren Zhou
TL;DR
High-quality information-seeking training data is scarce and difficult to construct, while information-driven synthesis can misalign information and reasoning structures. WebShaper formalizes IS tasks with set theory and Knowledge Projections, then uses seed tasks and agentic expansion to synthesize training data; experiments report the best performance among open-source IS agents on GAIA and WebWalkerQA.
Problem
High-quality information-seeking training data is sparse and difficult to construct, limiting the development of IS agents, while existing synthesis methods may create structural inconsistencies.
Method
WebShaper formalizes IS tasks through set theory and Knowledge Projections, then uses seed tasks and multi-step agentic expansion with retrieval and validation to synthesize data.
Results
WebShaper achieves the best performance among open-source IS agents on the GAIA and WebWalkerQA benchmarks.
Takeaways & Limitations
Formalization-driven synthesis provides systematic control over information-seeking task coverage, reasoning structure, and complexity.
Takeaways & Limitations
The layer-wise expansion strategy addresses redundancy because some connected constants do not increase the task-solving reasoning chain.
Abstract
from arXiv · showhide
The advent of Large Language Model (LLM)-powered agents has revolutionized artificial intelligence by enabling solutions to complex, open-ended tasks through web-based information-seeking (IS) capabilities. The scarcity of high-quality training data has limited the development of IS agents. Existing approaches typically adopt an information-driven paradigm that first collects web data and then generates questions based on the retrieval. However, this may lead to inconsistency between information structure and reasoning structure, question and answer. To mitigate, we propose a formalization-driven IS data synthesis framework WebShaper to construct a dataset. WebShaper systematically formalizes IS tasks through set theory. Central to the formalization is the concept of Knowledge Projections (KP), which enables precise control over reasoning structure by KP operation compositions. During synthesis, we begin by creating seed tasks, then use a multi-step expansion process. At each step, an agentic Expander expands the current formal question more complex with retrieval and validation tools based on our formalization. We train our model on the synthesized dataset. Experiment results demonstrate that WebShaper achieves state-of-the-art performance among open-sourced IS agents on GAIA and WebWalkerQA benchmarks.
1 Introduction
WebShaper addresses the scarcity and difficulty of constructing high-quality information-seeking training data by formalizing tasks before collecting information and synthesizing QA data. Its formalization-driven synthesis controls task structure and complexity, while experiments report strong benchmark performance.
- Motivation: High-quality information-seeking training data is sparse and difficult to construct, limiting the development of effective IS agents.Existing agentic systems depend on high-quality question-answer trajectories for supervised fine-tuning and reinforcement learning.
- Existing Approaches: Prior methods retrieve and organize information before generating questions, mapping information structures into reasoning structures.These approaches include linear information chains, web-link graphs, and entity-coreference networks.
- WebShaper: WebShaper formalizes IS tasks first, then guides information collection and QA synthesis through the formalization.The framework is grounded in set theory and derives tasks from compositions of Knowledge Projections.
- Formalization Benefits: Knowledge Projection compositions enable systematic task generation with controllable collection complexity and reasoning paths.The framework is intended to improve task coverage, controllability, and consistency between information-reasoning structures and question-answer pairs.
- Evaluation: WebShaper achieves the best performance among open-source IS agents on GAIA and WebWalkerQA.The experiments evaluate models trained with the synthesized WebShaper dataset and compare them with existing training datasets and baselines.
- Data Synthesis: The agentic Expander iteratively generates and validates questions in alignment with the formalization.Synthesis begins with foundational seed tasks and proceeds through multi-step expansion.
2 Information-Seeking Formalization
The paper formalizes information-seeking tasks as target entity sets recursively composed from Knowledge Projections and set operations. This representation captures uncertainty, simultaneous conditions, and multi-step question-answer reasoning.
- Task Definition: An information-seeking task q(T) searches for facts prompted by given facts and returns a target entity set T.The paper illustrates the task with a question asking for players satisfying team, season, and birth-period conditions.
- Worked Example: The example requires identifying Berliner FC Dynamo before retrieving its players from the 2004 and 2005 seasons and filtering by birth decade.The resulting answer set includes Robert Rudwaleit and Danny Kukulies.
- Knowledge Projection: A Knowledge Projection maps an entity subset through a relation to the entities connected to its members.For example, applying bornIn to the set {90s} yields people born in the 1990s.
- R-Union: R-Union combines relation results across multiple entity sets when the target is specified by a broader or uncertain condition.A player condition spanning 2000–2010 can be represented as the union of results for each year.
- Intersection: Intersection identifies entities satisfying several conditions simultaneously, including conditions expressed through different relations.The example intersects players associated with playing in 2000 and being born in the 1990s.
- Recursive Composition: IS task targets are recursively derived by replacing entity sets with other target sets, enabling compositions of Knowledge Projections and operations.The formalization defines T as a target set and treats solving as finding the entities contained in the questioned target.
3 Data Synthesis
WebShaper synthesizes information-seeking data by formalizing tasks with Knowledge Projections and expanding filtered seed questions through agentic, layer-wise retrieval and validation. This formalization enables controlled reasoning structures while addressing redundancy and reasoning shortcuts in prior expansion paradigms.
- Seed Question Construction: 18k seed questions were constructed after retaining questions with at least one correct answer among five WebDancer rollouts.The seed questions were acquired from random walks over an offline Wikipedia database and then filtered for answerability.
- Agentic Expansion: Agentic Expansion uses a KP-aware Expander that autonomously retrieves and validates information during multi-step question expansion.The process expands qn(T) into qn+1(T), while layer-wise expansion is used to reduce redundancy and reasoning shortcuts.
- KP Representation: Knowledge Projections represent information-seeking tasks as recursively composited operations over constants and variables.A KP uses triplets [X, r, S], while variables and constants support intersections, R-Union, and recursive compositions.
- KP Representation: The KP representation handles R-Union by merging sets and handles recursion by flattening nested projections into intersected triplets.The representation relies on the distributive law for R-Union and variable-mediated triplets for recursive projections.
- Layer-wise Expansion Strategy: Random expansion can add redundant constant-to-constant facts, while sequential expansion can create shortcuts that connect constants directly to the target.The formalization exposes these structural problems and supports controlled expansion paradigms beyond random or sequential structures.
- Layer-wise Expansion Strategy: Layer-wise expansion traverses leaf constants, replaces each with new fact patterns, and merges the resulting sub-question into the current question.Each expansion preserves the answer while the number of expanding layers controls task coverage and difficulty.
4 Experiments
WebShaper is evaluated against existing datasets and open-source information-seeking agents on GAIA and WebWalkerQA, with analyses of coverage, training-data effects, reinforcement learning, and task complexity. Across these evaluations, it achieves strong benchmark performance while producing broad, structurally complex training data.
- Main Results: WebShaper achieves the best performances among open-sourced methods on both GAIA and WebWalkerQA.It reports a 4.7 score advantage over WebSailor on GAIA with Qwen-2.5-72B and the highest WebWalkerQA score of 52.2.
- Main Results: WebShaper performs best across backbone settings, indicating that its synthesized data generalizes across different models.The method is described as the only open-source approach scoring above 60 points and as approaching OpenAI DR performance.
- Data Statistics: WebShaper provides broad, balanced domain coverage across topics including Sports, Politics, and Entertainment.Its seed-task construction and agentic expansion contribute to diversity while avoiding over-reliance on a single domain.
- Data Comparison: Models trained on WebShaper achieve the highest average GAIA scores across Qwen-2.5-32B, Qwen-2.5-72B, and QwQ-32B backbones.The comparison uses 5,000 samples from each dataset and reports improvements over datasets such as WebWalkerQA and MHQA.
- RL Stimulation: +7.8 points for the 32B model and +13.5 points for the 72B model on GAIA after reinforcement learning.The experiments report improvements after RL training for both Qwen2.5-32B and Qwen2.5-72B, with additional gains on WebWalkerQA.
- Data Statistics: WebShaper contains more complex search, navigation, and tool-use trajectories than comparison datasets.Its data includes tasks with more than three searches, trajectories extending beyond three visits, and non-zero proportions up to 30 tool calls.
- Case Study: Case studies show that formalized structures reduce redundancy and reasoning shortcuts while requiring information-seeking across variables.R-Union also summarizes distributed web contents, increasing difficulty in resolving variables K, N, and M.
5 Related Work
Related work situates WebShaper among information-seeking agents, benchmarks, and formalization-based data synthesis methods. Prior systems rely on limited or simple datasets, while theorem-proving research demonstrates formalization-driven synthesis in another domain.
- Information-Seeking Agents: Recent information-seeking agents integrate web interaction into LLM reasoning but often depend on limited or overly simplistic datasets.The cited literature includes both agent methods and benchmarks such as GAIA.
- Formalization-Based Synthesis: Formalization-based synthesis has been used in theorem proving to transform mathematical problems into formal statements and verified proofs.DeepSeek-MathProver uses Lean 4 verification, while its successor decomposes proofs into subgoals for training-data synthesis.
6 Conclusion
WebShaper formalizes information-seeking tasks with set theory and Knowledge Projections, then expands them through retrieval and validation to engineer complex reasoning structures. It reports state-of-the-art performance on GAIA and WebWalkerQA while providing controllable task design.
- Conclusion: WebShaper uses set-theoretic task formalization and Knowledge Projection compositions to engineer controllable reasoning structures and complexity.Its agentic Expander performs layer-wise expansion with knowledge retrieval and validation to reduce redundancy and reasoning shortcuts.
- Conclusion: WebShaper achieves state-of-the-art performance on GAIA and WebWalkerQA benchmarks among open-source information-seeking agents.The conclusion presents this result alongside the framework’s controllable task design.
A Agent Details
WebComposer supports information seeking through search and visit tools. Search retrieves candidate documents, while visit extracts information from specified pages according to a goal.
- Tools: Search retrieves relevant documents for natural-language queries and returns the top-10 results for each query.Each result includes a title, snippet, and URL, and multiple queries can run in parallel.
- Tools: Visit extracts goal-relevant information from a specified webpage after retrieving its full content.A summarization model performs the extraction using the designated visit goal.
B.1 SFT
The paper specifies separate training configurations for supervised fine-tuning and reinforcement learning, including batch sizes, learning rates, and scheduling or rollout parameters.
- SFT: SFT uses a batch size of 32 and a learning rate of 5e-6.Training uses warmup plus cosine decay and weight decay of 0.1.
- SFT: SFT applies warmup plus cosine decay scheduling with weight decay of 0.1.
- RL: RL uses groups of 8 rollouts, temperature 1.0, topp = 1.0, batch size 128, and mini batch size 32.
- RL: RL training uses a learning rate of 1e-6.
C Case Study
The case study compares an example involving Strange Stories from a Chinese Studio and highlights how apparent complexity can conceal a simple reasoning structure.
- Question in Natural Language: The case question concerns counting sentences spoken by the scholar Yu Jing in “The Green-Clothed Girl.”
- Case comparison: “SSCS” stands for “Strange Stories from a Chinese Studio.”
- Case comparison: KIMI-Researcher’s example contains redundant SSCS-related constants and a shortcut connecting directly to the target variable.
- Case comparison: Despite its apparent complexity, the underlying reasoning structure is a single-hop path.
D Broader Impact
WebShaper frames agent training-data synthesis around explicit task formalization and agent-driven generation. The framework emphasizes controllable complexity, quality assurance, scalability, and broader applicability beyond human-generated data patterns.
- Broader Impact: The framework combines task formalization with agent-driven synthesis to construct training data for intelligent agents.
- Implications in Agent Training Data Synthesis: Explicit formalization enables systematic adjustment of task difficulty, modality, and compositional structure.This provides controlled gradients of challenge rather than relying on ad-hoc natural data.
- Implications in Agent Training Data Synthesis: Formal task models serve as specifications that reduce synthesis noise and improve consistency.The paper identifies this as important for reliability- and safety-sensitive applications.
- Implications in Agent Training Data Synthesis: Agent-driven synthesis supports scalable generation without laborious manual annotation or heuristic-based pipelines.The claimed benefit is reduced computational overhead while preserving fidelity to the intended task design.
- Implications for AI Research and Development: The framework is presented as enabling training data to exceed constraints associated with natural human-generated data.The paper connects this possibility to domains requiring superhuman reasoning, including scientific modeling and combinatorial optimization.
- Implications for AI Research and Development: Formal task representations can support learning invariant principles across diverse domains and tasks.