PaperlayerView exact revision on arXiv

Source-linked AI summary

WebDancer: Towards Autonomous Information Seeking Agency

Jialong Wu, Baixuan Li, Runnan Fang, Wenbiao Yin, Liwen Zhang, Zhengwei Tao, Dingchu Zhang, Zekun Xi, Gang Fu, Yong Jiang, Pengjun Xie, Fei Huang, Jingren Zhou

arXiv:2505.22648v3cs.CL

TL;DR

WebDancer addresses the challenge of training autonomous agents for deep, multi-step information seeking in real-world web environments. It builds a ReAct agent through scalable QA synthesis, trajectory sampling, supervised cold start, and reinforcement learning, and reports strong performance on GAIA and WebWalkerQA. The analyses provide systematic insights and pathways for developing more capable agentic models, while also identifying transfer and stabilization limitations.

  • Problem

    Existing information-seeking approaches have limited evidence for robust behavior across out-of-distribution web environments, complex interactions, and long-term objectives.

  • Method

    WebDancer uses scalable web QA synthesis and a four-stage pipeline of trajectory sampling, supervised fine-tuning, and reinforcement learning within ReAct.

  • Results

    Experiments on GAIA and WebWalkerQA show strong performance for WebDancer and support the effectiveness of its training pipeline.

  • Takeaways & Limitations

    The work provides actionable, systematic pathways for advancing agentic models that tackle complex real-world information-seeking tasks.

  • Takeaways & Limitations

    Reasoning patterns from strong reasoner models do not transfer easily to instruction-tuned models, while Long-CoT can increase invalid repetition and context-length failures in smaller models.

Abstract

from arXiv · show

Addressing intricate real-world problems necessitates in-depth information seeking and multi-step reasoning. Recent progress in agentic systems, exemplified by Deep Research, underscores the potential for autonomous multi-step research. In this work, we present a cohesive paradigm for building end-to-end agentic information seeking agents from a data-centric and training-stage perspective. Our approach consists of four key stages: (1) browsing data construction, (2) trajectories sampling, (3) supervised fine-tuning for effective cold start, and (4) reinforcement learning for enhanced generalisation. We instantiate this framework in a web agent based on the ReAct, WebDancer. Empirical evaluations on the challenging information seeking benchmarks, GAIA and WebWalkerQA, demonstrate the strong performance of WebDancer, achieving considerable results and highlighting the efficacy of our training paradigm. Further analysis of agent training provides valuable insights and actionable, systematic pathways for developing more capable agentic models. The codes and demo will be released in https://github.com/Alibaba-NLP/WebAgent.

1 Introduction

WebDancer frames autonomous web information seeking as an end-to-end problem spanning scalable data construction, trajectory sampling, supervised cold start, and reinforcement learning. The resulting pipeline is evaluated on GAIA and WebWalkerQA, with analyses of data efficiency and agent learning.

  • Research gap: Prior information-seeking agents rely either on prompting or SFT/RL, but challenges remain in robust behavior across changing web environments, complex interactions, and long-term objectives.The paper positions these limitations as motivation for a systematic training-stage approach.
  • Approach: The agent follows ReAct by interleaving reasoning and tool actions, using observations from the external environment to guide subsequent decisions until task completion.This tool-augmented loop is optimized end to end for autonomous multi-turn information seeking.
  • Data motivation: The data strategy addresses the scarcity of large, challenging training corpora because existing web QA benchmarks are small and often shallow.The paper cites 466 GAIA examples, 680 WebWalkerQA examples, and 1,266 BrowseComp examples as representative sizes.
  • Approach: WebDancer uses a four-stage pipeline: challenging QA construction, high-quality trajectory sampling, agentic-task fine-tuning, and reinforcement learning for generalization.The stages are presented as a data-centric and training-stage guideline for building long-term information-seeking web agents.
  • Evaluation: WebDancer is evaluated on the GAIA and WebWalkerQA information-seeking benchmarks, alongside analyses of data efficiency, agentic evaluation, and agent learning.The introduction reports these experiments as evidence for the effectiveness of the pipeline.

2 Deep Information Seeking Dataset Synthesis

The dataset synthesis strategy creates web questions that are diverse, specific, and increasingly multi-step. It combines recursive crawling with iterative entity-to-question refinement, then prepares and filters ReAct trajectories for agent training.

  • Dataset goals: The dataset targets diverse, complex questions requiring more interaction steps than shallow queries typically solvable in 2–3 steps.Complexity is controlled through the number of required interaction steps and refinement operations.
  • CRAWLQA: CRAWLQA recursively follows hyperlinks from official and knowledgeable root websites, then uses GPT-4o to synthesize QA pairs from the collected content.The construction emulates human browsing across sources such as arXiv, GitHub, and wikis.
  • E2HQA: E2HQA searches for information about an entity, restructures the retrieved content into a replacement query, and iteratively transforms the original question into a harder multi-step problem.The transformation preserves the original answer while adding a solvable subproblem at each refinement.
  • Agent trajectories: The ReAct agent alternates Thought, Action, and Observation steps, using search and visit tools before terminating with an answer action.Search returns top-10 titles and snippets, while visit returns summarized evidence and observations.
  • Trajectory filtering: Trajectory filtering applies validity control, correctness verification, and quality assessment based on non-redundancy, goal alignment, and logical reasoning and accuracy.Invalid formats and trajectories with more than two actions are discarded before quality filtering.

3 Multi-Step Multi-Tool Agent Learning

WebDancer first uses supervised fine-tuning to teach models to alternate reasoning and actions, then applies on-policy DAPO reinforcement learning to improve multi-tool agency from outcome-based rewards. The pipeline focuses optimization on model-generated decisions and dynamically filters uninformative prompts.

  • Supervised fine-tuning: SFT on decision-making trajectories provides a cold start that couples multiple reasoning and action steps while preserving the model’s original reasoning capabilities.The SFT data are obtained from complete agentic trajectories.
  • Supervised fine-tuning: The SFT loss excludes tokens corresponding to external tool feedback, so training focuses on the agent’s autonomous decision steps.This token-selection rule aligns the supervised objective with the agent-generated portion optimized during RL.
  • Reinforcement learning: DAPO reinforcement learning samples candidate executions for each question-answer pair and updates the policy to strengthen multi-turn, multi-tool behavior using outcome-based rewards.The RL stage builds on the SFT initialization and optimizes the policy’s interleaving of thought and action.
  • Reinforcement learning: RL optimization applies only to model-generated execution tokens, while model outputs and tool responses together form the input context for the old policy.The clipping range controls the importance-sampling ratio used in the policy update.
  • Reinforcement learning: Dynamic sampling over-samples and filters prompts with accuracy equal to 1 or 0, helping exclude unreliable synthetic instances from policy learning.The mechanism concentrates training on remaining QA pairs that provide usable learning signals.

4 Experiments

Experiments evaluate WebDancer and related frameworks on GAIA and WebWalkerQA, then test WebDancer on more challenging BrowseComp datasets. WebDancer shows strong performance, including substantial gains over vanilla ReAct and a best-case result surpassing GPT-4o.

  • Evaluation Setup: The experiments evaluate web agents on GAIA and WebWalkerQA using LLM-as-Judges and Pass@1.Qwen-7B and Qwen-32B use Short-CoT training data, while QwQ-32B uses Long-CoT data.
  • Main Results: Frameworks without agentic capabilities perform poorly on both GAIA and WebWalkerQA.The comparison treats active information seeking and agentic decision-making as central capabilities for these tasks.
  • Main Results: Open-source agentic frameworks built on native strong reasoning models consistently outperform their non-agentic counterparts.The reported comparison highlights QwQ-32B-based agentic approaches among open-source frameworks.
  • Main Results: WebDancer achieves substantial gains over vanilla ReAct across different model scales.The gains are reported within the extensible ReAct framework, indicating improvement over the corresponding baseline.
  • Main Results: WebDancer surpasses GPT-4o in the best-case scenario.This result is reported alongside strong performance across the evaluated benchmarks.
  • More Challenging Benchmarks: WebDancer demonstrates consistently strong performance on BrowseComp and BrowseComp-zh under PASS@1/PASS@3 evaluation.These datasets are intended to reflect more complex information-seeking scenarios.

5 Analysis

The analysis examines WebDancer’s data efficiency, evaluation metrics, training dynamics, knowledge transfer, and robustness in changing web environments. Results emphasize the roles of filtering, cold-start SFT, RL, and environment variability.

  • Training dynamics: For non-reasoning models, RL substantially improves Pass@3 and Cons@3, whereas LRM gains are marginal except for consistency.
  • Benchmark results: 64.1% Pass@3 on GAIA and 62.0% on WebWalkerQA are achieved by the best-performing model.
  • Data efficiency: Robust filtering improves low-data performance, while long-CoT can produce hallucinations and invalid repetition, especially in smaller models.
  • Training dynamics: 5% Pass@3 on GAIA under reinforcement learning alone shows that SFT cold start is essential for multi-step, multi-tool instruction following.
  • Knowledge transfer: Reasoning patterns transfer poorly from strong reasoners to small instruction models, with Long-CoT also increasing invalid rates through repetition and context overrun.
  • Data efficiency: Mixing short-CoT and long-CoT does not produce significant performance improvements.

6 Related Works

Related work spans prompting-based information seeking, tool-augmented agents, and training-based agent learning. Existing evidence reports limited generalization for pure SFT agents in adaptive operational contexts.

  • Information Seeking Agents and Benchmarks: Information-seeking agents use tool-augmented reasoning, outcome-based RL, or online RL in real web environments.
  • Agents Learning: Agent learning has progressed from in-context learning toward SFT and RL, while pure SFT agents often generalize poorly to adaptive operational contexts.

7 Conclusion

The paper proposes a systematic framework for building end-to-end, multi-step information-seeking web agents from scratch. WebDancer achieves strong performance on GAIA and WebWalkerQA, while the analysis identifies actionable directions for agent training.

  • WebDancer combines scalable QA data synthesis with a two-stage pipeline of SFT and on-policy RL.
  • WebDancer achieves strong performance on GAIA and WebWalkerQA and motivates systematic pathways for developing more sophisticated agentic models.

A Limitations

The framework has scope, tooling, data-utilization, computational, reasoning-pattern, and broader deployment limitations. The authors identify these boundaries as directions for future development.

  • Tool Number and Type: The current framework integrates only two basic information-seeking tools, limiting retrieval capabilities that require browser modeling or external APIs.The authors propose adding modular browser tools and a Python sandbox.
  • Task Generalization and Benchmarks: Experiments cover only two short-answer information-seeking tasks, leaving document-level research and long-form generation outside the demonstrated scope.The authors note that open-domain long-form writing creates reward-modeling challenges.
  • Data Utilization: Only a small subset of the collected data, such as 5,000 pairs, can currently be used in RL because of computational and stability constraints.The authors call for more efficient data-utilization strategies.
  • High Rollout Cost: RL rollouts incur substantial computational and time overhead because each includes multiple tool invocations and LLM completions.This limits scalability and slows iterative development and experimentation.
  • Hybrid Thinking: The models are trained on either short- or long-CoT data, rather than a hybrid dataset that dynamically controls reasoning length.A hybrid reasoning agent is proposed as future work.
  • Thinking Pattern: Tool invocation can produce hallucinated calls or redundant actions after an answer has already been confirmed.The paper gives nonexistent calculation tools and over-action as examples.

D Training Dataset

The training datasets combine collected QA samples with generated trajectories, quality filtering, and difficulty-oriented open datasets. The data construction targets reliable, multi-step agent training.

  • Dataset Construction: The corpus includes 40K E2HQA and 60K CRAWLQA samples used to generate trajectories with QwQ or GPT-4o.A multi-stage filtering process is applied to ensure trajectory quality.
  • Dataset Statistics: Training statistics report average tokenized thought length for the datasets.Table 4 separately reports Long-CoT and Short-CoT SFT data statistics.
  • Filtering Criterion: Trajectory filtering constrains each 10-gram to at most four occurrences to reduce repetitive patterns.The stated purpose is to prevent models from internalizing detrimental patterns.
  • Open-only Datasets: Open-only data draws from MuSiQue, Bamboogle, PopQA, 2Wiki, and HotpotQA, with RAG-based filtering used to remove easy questions.These datasets are selected to support question difficulty.

E.1 Benchmarks

The benchmark setup evaluates WebDancer against non-agentic and agentic frameworks on GAIA and WebWalkerQA, using ReAct-style tool interaction and specified training infrastructure.

  • Benchmarks: The experiments use 103 text-only GAIA validation questions and 680 WebWalkerQA test questions.GAIA targets complex information retrieval, while WebWalkerQA targets deep web retrieval.
  • Compared Frameworks: The comparison includes No Agency baselines, OpenAI Deep Research, and open-source frameworks such as WebThinker and Search-o1.The baselines cover direct RAG, closed-source agency, and open-source web agents.
  • Agent Format: WebDancer uses multi-turn ChatML in which tool responses are user messages and model thoughts and actions are assistant messages.This format represents the interaction sequence used for training and inference.
  • Training and Inference: RL uses 16 rollouts with temperature 0.6 and top-p 0.95, while experiments run on 32 nodes containing eight NVIDIA H20 GPUs each.Additional repetition-penalty settings differ between LRMs and LLMs.
  • Prompting: The setup uses the ReAct prompt configuration for LLMs and LRMs, with prompts specifying allowed tools and final-answer behavior.Figures 6 and 7 present the corresponding prompt designs.

F Case Study

The case study illustrates WebDancer’s multi-step information seeking through decomposition, hypothesis testing, gap handling, and iterative reflection. The agent combines USGS evidence with a separate location search.

  • Step-by-step Decomposition: WebDancer decomposes the complex question into sequential steps using “First ... Then ... Finally.”The case study presents decomposition as an explicit thinking pattern.
  • Hypothesis Testing: The agent hypothesizes that Nemo is Amphiprion ocellaris and verifies the species against USGS nonnative-occurrence data.The hypothesis links the movie reference to a specific fish species before retrieval.
  • Handling Information Gaps: After the USGS page omitted ZIP codes, WebDancer identified the missing information and initiated a separate search for Fred Howard Park’s address.This shows gap handling through an additional tool action.
  • Evidence Verification: The USGS record identifies a 2018 Amphiprion ocellaris occurrence at Fred Howard Park in Pinellas County, Florida, but does not provide a ZIP code.The case study distinguishes the source record from the separately retrieved address information.
  • Iterative Reflection: A second search returns Fred Howard Park at 1700 Sunset Drive, Tarpon Springs, Florida 34689.The case study uses this address to supply the requested ZIP code.
  • Case Result: The case study concludes that 34689 is the requested ZIP code because the provided occurrence data cites only the Florida location.The result depends on combining the USGS occurrence record with the address search.
Loading 2505.22648v3…