PaperlayerView exact revision on arXiv

Source-linked AI summary

WebExplorer: Explore and Evolve for Training Long-Horizon Web Agents

Junteng Liu, Yunji Li, Chi Zhang, Jingyang Li, Aili Chen, Ke Ji, Weiyu Cheng, Zijia Wu, Chengyu Du, Qidi Xu, Jiayuan Song, Zhengmao Zhu, Wenhu Chen, Pengyu Zhao, Junxian He

arXiv:2509.06501v3cs.CL

TL;DR

Existing open-source web agents struggle with difficult information-seeking tasks, motivating better challenging training data. WebExplorer generates such data through model-based exploration and long-to-short query evolution, then trains WebExplorer-8B with supervised fine-tuning and reinforcement learning. The resulting 8B model achieves state-of-the-art performance at its scale across information-seeking benchmarks and generalizes to HLE.

  • Problem

    Existing open-source web agents have limited performance on challenging search benchmarks, while challenging information-seeking training data is scarce.

  • Method

    WebExplorer uses model-based exploration and iterative long-to-short query evolution to synthesize challenging query-answer pairs, then applies supervised fine-tuning followed by GRPO reinforcement learning.

  • Results

    WebExplorer-8B achieves state-of-the-art performance at its scale across multiple information-seeking benchmarks and reports 17.3% on HLE.

  • Takeaways & Limitations

    An 8B agent trained on WebExplorer-QA can perform long-horizon web search, averaging over 16 tool calls after reinforcement learning.

Abstract

from arXiv · show

The paradigm of Large Language Models (LLMs) has increasingly shifted toward agentic applications, where web browsing capabilities are fundamental for retrieving information from diverse online sources. However, existing open-source web agents either demonstrate limited information-seeking abilities on complex tasks or lack transparent implementations. In this work, we identify that the key challenge lies in the scarcity of challenging data for information seeking. To address this limitation, we introduce WebExplorer: a systematic data generation approach using model-based exploration and iterative, long-to-short query evolution. This method creates challenging query-answer pairs that require multi-step reasoning and complex web navigation. By leveraging our curated high-quality dataset, we successfully develop advanced web agent WebExplorer-8B through supervised fine-tuning followed by reinforcement learning. Our model supports 128K context length and up to 100 tool calling turns, enabling long-horizon problem solving. Across diverse information-seeking benchmarks, WebExplorer-8B achieves the state-of-the-art performance at its scale. Notably, as an 8B-sized model, WebExplorer-8B is able to effectively search over an average of 16 turns after RL training, achieving higher accuracy than WebSailor-72B on BrowseComp-en/zh and attaining the best performance among models up to 100B parameters on WebWalkerQA and FRAMES. Beyond these information-seeking tasks, our model also achieves strong generalization on the HLE benchmark even though it is only trained on knowledge-intensive QA data. These results highlight our approach as a practical path toward long-horizon web agents.

1. Introduction

WebExplorer addresses difficult web-agent training by generating challenging information-seeking data through model-based exploration and long-to-short query evolution. Training on this data yields an 8B agent with strong performance across information-seeking benchmarks and generalization to HLE.

  • Existing open-source web agents have limited performance on challenging search benchmarks, while stronger commercial systems lack transparent training details.
  • Challenging query-answer pairs are scarce, although difficult information-seeking data is essential for developing capable agents.More than half of BrowseComp-en queries reportedly remain unsolved by human annotators.
  • Model-based exploration uses iterative search and browsing from a seed entity to construct an information space and synthesize initial query-answer pairs.The approach avoids explicit rule-based graph construction and its expansion and node-selection overhead.
  • Iterative evolution removes explicit clues and introduces obfuscation, producing queries that require longer solution processes and more exploratory search attempts.
  • WebExplorer-8B achieves state-of-the-art performance at its scale across multiple information-seeking benchmarks, including 15.7% on BrowseComp-en and 32.0% on BrowseComp-zh.It also reports 62.7% on WebWalkerQA, 75.7% on FRAMES, and 17.3% on HLE.

2. WebExplorer: Synthesizing Challenging QA Pairs

WebExplorer-QA is synthesized by autonomously exploring information spaces from seed entities, then making initially solvable queries harder through strategic obfuscation. The resulting evolved dataset increases reasoning demands while retaining reasonable solvability.

  • Manual curation makes challenging information-seeking benchmarks costly and small, limiting their use for large-scale training.BrowseComp-en contains queries that more than half of human annotators could not solve after two hours.
  • Graph-based methods explicitly construct knowledge graphs, whereas evolution-based methods typically inject information to lengthen queries.
  • 2.2. Model-Based Exploration: WebExplorer uses LLMs to navigate the web autonomously from a seed entity, internally simulating graph construction without explicit graph maintenance.The model iteratively searches and browses before synthesizing query-answer pairs from the explored information space.
  • 2.2. Model-Based Exploration: The exploration process discovers interconnected facts through search and browse actions before producing a query-answer pair requiring reasoning across multiple connections.The Brazil National Team example focuses on the 1976 FA Cup and related players and referees.
  • 2.3. Iterative Query Evolution: Initial QA pairs remain relatively easy for proprietary models because they contain explicit information and clear search entry points.Claude-4-Sonnet achieves 86.6% accuracy on the initial QA pairs.
  • 2.4. The WebExplorer-QA Dataset: Evolution lowers Claude-4-Sonnet accuracy from 86.6% to 67.1% while raising average solution turns from 7.9 to 9.9.The evolved dataset has the highest average turn count among compared datasets.
  • 2.4. The WebExplorer-QA Dataset: BrowseComp-en remains harder than Evolved QA, but its below-20% accuracy on most open-source models makes fully matching its difficulty unnecessary for current training.
  • 2.4. The WebExplorer-QA Dataset: WebExplorer-QA increases complexity while maintaining reasonable solvability, supporting its use as high-quality training data for web-navigation agents.

3. Cold Start and Reinforcement learning

WebExplorer trains its agent in two stages: supervised fine-tuning establishes tool-use and long-horizon search capabilities, followed by GRPO reinforcement learning. Progressive context and turn expansion supports increasingly complex trajectories and benchmark performance.

  • Supervised fine-tuning provides cold-start initialization for search, browse invocation, and foundational long-horizon search capabilities.
  • Reinforcement learning with GRPO further improves reasoning and decision-making while allowing direct exploration of synthesized QA pairs.
  • The reward design combines response-format correctness with answer accuracy judged by DeepSeek-V3.
  • WebExplorer-8B reports accuracy results across information-seeking benchmarks using the evaluation setup described for Table 2.The table caption specifies Avg@4 and LLM-as-Judge evaluation for WebExplorer-8B scores.
  • Training expands progressively from 64K tokens and 50 tool turns to 128K tokens and 100 turns.Intermediate training uses 96K tokens and a 75-turn limit.

4. Experiments

The experiments evaluate WebExplorer-8B across diverse information-seeking benchmarks and analyze its training dynamics. The model achieves state-of-the-art performance at its scale, generalizes beyond information-seeking tasks, and increasingly uses long tool-call trajectories during RL.

  • Experimental Setups: WebExplorer-8B is evaluated on BrowseComp-en/zh, GAIA, WebWalkerQA, FRAMES, XBench-DeepSearch, and HLE.HLE tests generalization on difficult academic questions beyond the model’s knowledge-intensive QA training data.
  • Experimental Setups: The scaffold uses search and browse tools, while training combines supervised fine-tuning with GRPO-based reinforcement learning.RL training supports 128K context length and up to 100 tool-calling turns.
  • Experimental Results: 15.7% on BrowseComp-en and 32.0% on BrowseComp-zh are achieved after reinforcement learning.The RL model surpasses the previous best sub-10B model across all benchmarks, establishing state-of-the-art performance at its parameter scale.
  • Experimental Results: 8B parameters suffice for best open-source sub-100B performance on BrowseComp-en, BrowseComp-zh, FRAMES, and WebWalkerQA.The model achieves 75.7% on FRAMES and 62.7% on WebWalkerQA, exceeding WebShaper-72B by 10 absolute points on WebWalkerQA.
  • Experimental Results: 17.3% on HLE surpasses previous 32B models, including ASearcher-Web-QwQ and WebThinker-32B.The training methodology is BrowseComp-inspired but uses seed entities from diverse Wikipedia domains.
  • RL Training Dynamics and Performance Analysis: Over 16 average tool calls and over 40K trajectory tokens emerge during RL training as BrowseComp-en rises from 7.9% to 15.7%.Average tool calls increase from approximately 11 to over 16, while performance on both BrowseComp-en and BrowseComp-zh improves consistently.

5. Conclusion

WebExplorer synthesizes challenging query-answer pairs through model-based exploration and iterative query evolution, then trains WebExplorer-8B with supervised fine-tuning and reinforcement learning. The resulting model achieves state-of-the-art performance across information-seeking benchmarks, generalizes to HLE, and averages over 16 tool-calling turns after RL.

  • 5. Conclusion: WebExplorer combines model-based exploration and iterative query evolution to synthesize QA pairs requiring over 10 tool-calling turns.The resulting WebExplorer-8B model is trained with supervised fine-tuning followed by GRPO-based reinforcement learning.
  • 5. Conclusion: WebExplorer-8B achieves state-of-the-art performance across BrowseComp-en/zh, GAIA, WebWalkerQA, FRAMES, and XBench-DeepSearch.It achieves the best performance at its scale while significantly outperforming larger models such as WebSailor-72B.
  • 5. Conclusion: Over 16 average tool-calling turns are reached during RL training, supporting the approach for long-horizon web agents.The conclusion presents this scaling behavior as validation of the training approach.

A. Trajectory Example

The trajectory example follows an agent solving a highly constrained gaming-related question through iterative search and browsing. It identifies the relevant game, company, award, campaign, dates, and final funding amount, while the displayed trajectory includes an unresolved age-milestone check.

  • Trajectory Structure: The example illustrates multi-step reasoning through alternating thought, search, browse, and tool-response stages.The trajectory is presented as an example from the WebExplorer supervised fine-tuning data.
  • Query Decomposition: The trajectory decomposes a complex query about a gaming company, an award-winning game, a crowdfunding campaign, and an age milestone.The agent explicitly breaks the task into identifying the company, award, campaign, and related biographical information.
  • Entity Identification: Spirit of the Century is identified as a pulp adventure game by Evil Hat Productions that won a Silver ENnie for Best Rules in 2007.Search results connect the game with Evil Hat Productions and the 2007 award.
  • Campaign Verification: The browse step confirms the Fate Core campaign crossed from 2012 into 2013 but finds no mention of Fred Hicks’ birthday or age milestone.This leaves the age-milestone condition unsupported within the trajectory’s browsing result.

B. Details of WebExplorer-QA Synthesis Framework

The WebExplorer-QA synthesis framework uses two stages guided by prompting strategies for autonomous information gathering and query evolution. Compared with more complex prior synthesis processes, it is presented as a simpler prompting-based framework intended to support reproducibility.

  • Framework Overview: The framework uses a simpler prompting-based synthesis process than prior approaches involving complex construction procedures.The authors provide methodological details to facilitate reproducibility.
  • Framework Overview: WebExplorer-QA operates through two stages: autonomous information gathering and query evolution.Both stages are guided by carefully designed prompting strategies leveraging large language models.

B.1. Model-Based Exploration

Model-based exploration starts from a seed entity, uses web search and browsing to gather information, and generates challenging QA pairs with subtle clues and unique, verifiable answers.

  • Model-Based Exploration: The process begins with a seed entity and exemplar BrowseComp-en QA pairs that guide the model’s question-generation behavior.The model searches and browses from the seed, then synthesizes a query-answer pair from the collected knowledge.
  • Model-Based Exploration: Initial QA pairs use subtle, obscured clues while keeping answers unique and verifiable across the constructed information space.The resulting questions require reasoning across multiple websites.
  • Model-Based Exploration: The model is instructed to collect internet information, select a truth, and create a question whose answer must be discovered through search.The prompt requires several subtle or blurred clues while preserving a unique truth.
  • Model-Based Exploration: The exploration prompt formats each generated pair with question and truth fields inside the required answer structure.The prompt specifies one exact truth corresponding to the challenging question.

B.2. Iterative Query Evolution

Iterative query evolution transforms initial QA pairs into harder variants by reducing information and obfuscating details while preserving a unique answer, using model-driven search and browsing.

  • Iterative Query Evolution: The second stage takes the full first-stage trajectory and transforms each initial QA pair into a more challenging variant.Its mechanisms include removing redundant descriptions and obfuscating details such as dates, locations, and proper names.
  • Iterative Query Evolution: The prompting approach avoids complex graph-construction heuristics and predefined evolution rules, supporting quick reproducibility of challenging QA-pair generation.The stated benefit is a simple framework that leverages language understanding and web exploration.
  • Iterative Query Evolution: Evolution removes redundant clues, makes selected descriptions vaguer, and replaces specific entities with alternative terms while keeping the truth unique.These operations are explicitly applied to the original question-answer pair.
  • Iterative Query Evolution: The question can be iteratively intensified for up to five rounds, with evolved questions recorded before producing the final question-truth pair.Search and browsing may be used during the evolution process.

C. Tool Schemas

The system uses two tools for web-based information retrieval: search for discovering relevant content and browse for analyzing content in detail.

  • Tool Schemas: Web-based retrieval relies on a search tool for finding relevant content and a browse tool for detailed content analysis.The paper introduces these as the system’s two primary retrieval tools.

C.1. Search Tool

The search tool queries Google to discover relevant URLs and snippets, accepting arrays of query strings and returning the top ten results for each query.

  • Search Tool: The search tool uses a Google Search API to discover relevant URLs and content snippets from input queries.Its returned information supports finding candidate web sources.
  • Search Tool: Its schema defines the search operation and the query parameter as an array.The queries are sent to Google and produce brief result records.
  • Search Tool: The tool accepts an array of query strings and returns titles, URLs, and content snippets for each query.The results are configured to include the top 10 entries per query.

C.2. Browse Tool

The browse tool analyzes content from a specified URL to answer a targeted query. It combines Jina for content extraction with Gemini Flash for language-model analysis.

  • The browse tool takes a URL and a specific query as inputs.The retrieved URL content is sent to a language model as the basis for answering the query.
  • Jina performs content extraction from the browsed URL.
  • Gemini Flash analyzes the extracted content and generates a targeted answer.
Loading 2509.06501v3…