PaperlayerView exact revision on arXiv

Source-linked AI summary

Beyond Ten Turns: Unlocking Long-Horizon Agentic Search with Large-Scale Asynchronous RL

Jiaxuan Gao, Wei Fu, Minyang Xie, Shusheng Xu, Chuyi He, Zhiyu Mei, Banghua Zhu, Yi Wu

arXiv:2508.07976v4cs.CLcs.AI

TL;DR

Open-source search agents lack reliable expert-level Search Intelligence for ambiguous, knowledge-intensive questions, and existing approaches face limits in search horizon, scalability, and QA data quality. ASearcher addresses this with fully asynchronous agentic RL and autonomous synthesis of challenging QA data, yielding substantial benchmark gains and long-horizon search behavior. The resulting ASearcher-Web-QwQ outperforms existing open-source agents across the reported evaluation settings.

  • Problem

    Open-source search agents struggle with expert-level reasoning over ambiguous queries, while existing approaches have limited search turns and insufficient challenging QA data.

  • Method

    ASearcher combines fully asynchronous agentic RL with an LLM-based data-synthesis agent for large-scale, high-quality QA construction.

  • Results

    ASearcher outperforms state-of-the-art open-source agents across model sizes and evaluation settings, including 78.0% and 34.3% improvements on xBench-DeepSearch and GAIA, respectively.

  • Takeaways & Limitations

    ASearcher provides an open-source pipeline that unlocks long-horizon search behavior through scalable training and high-quality data.

  • Takeaways & Limitations

    The case study reports that existing short-horizon RL agents fail to learn complex search strategies, while prompt-based agents may miss key information and fail to verify conclusions.

Abstract

from arXiv · show

Recent advancements in LLM-based agents have demonstrated remarkable capabilities in handling complex, knowledge-intensive tasks by integrating external tools. Among diverse choices of tools, search tools play a pivotal role in accessing vast external knowledge. However, open-source agents still fall short of achieving expert-level Search Intelligence, the ability to resolve ambiguous queries, generate precise searches, analyze results, and conduct thorough exploration. Existing approaches fall short in scalability, efficiency, and data quality. For example, small turn limits in existing online RL methods, e.g. <=10, restrict complex strategy learning. This paper introduces ASearcher, an open-source project for large-scale RL training of search agents. Our key contributions include: (1) Scalable fully asynchronous RL training that enables long-horizon search while maintaining high training efficiency. (2) A prompt-based LLM agent that autonomously synthesizes high-quality and challenging QAs, creating a large-scale QA dataset. Through RL training, our prompt-based QwQ-32B agent achieves substantial improvements, with 78.0% and 34.3% Avg@4 gains on xBench and GAIA, respectively. Notably, our agent exhibits extreme long-horizon search, with tool calls exceeding 100 turns and output tokens exceeding 400k during training time. With a simple agent design and no external LLMs, ASearcher-Web-QwQ achieves Avg@4 scores of 51.1 on xBench and 58.7 on GAIA, surpassing existing open-source 32B agents. Finally, we also show that ASearcher-Web-QwQ could achieve performance of commercial systems using external summary tool in a zero-shot transfer manner and test-time search. We open-source our models, training data, and codes in https://github.com/inclusionAI/ASearcher.

1 Introduction

Open-source search agents struggle with expert-level Search Intelligence because complex queries require resolving uncertainty, navigating conflicting information, and conducting deep multi-turn exploration. ASearcher addresses these limitations with asynchronous RL, scalable QA synthesis, and agents that improve substantially across challenging benchmarks.

  • Motivation: Complex search requires resolving ambiguous queries, generating precise searches, analyzing results, and exploring conflicting external knowledge.The introduction illustrates this challenge with conflicting medal counts for China in the 2012 London Olympics.
  • Limitations: Existing online RL methods limit search trajectories, while available QA datasets are often outdated, oversimplified, or too small for complex strategy learning.The paper identifies insufficient search turns and inadequate large-scale, high-quality QA pairs as two critical obstacles.
  • Approach: ASearcher provides fully asynchronous agentic RL that decouples trajectory execution from model updates, enabling relaxed limits such as 128 turns per trajectory.The system prevents long trajectories from blocking training and supports deeper search paths without sacrificing training efficiency.
  • Approach: ASearcher includes an LLM-based synthesis agent that generates challenging, uncertain, and grounded QA pairs requiring multi-turn tool use.From 14k seed QAs, the pipeline produces 134k samples, including 25.6k requiring external tools.
  • Results: ASearcher-Web-QwQ achieves Avg@4 scores of 51.1 on xBench-DeepSearch and 58.7 on GAIA, surpassing open-source agents.The agent also records 74.7 Pass@4 on GAIA and 75.0 on xBench-DeepSearch.
  • Contribution: ASearcher is an open-source pipeline for training LRM-based and LLM-based search agents through scalable RL and high-quality data.The project is presented as a foundation for broader innovation in complex real-world agent tasks.

2 Limitations of Existing Open-source Approaches

The case study shows that existing open-source approaches struggle with complex search: short-horizon RL agents hallucinate or fail to decompose queries, while prompt-based agents miss and fail to verify key information. ASearcher-Web-QwQ instead demonstrates uncertainty-aware reasoning, precise extraction, cross-document inference, and grounded verification.

  • Case Study: The GAIA case requires resolving multiple unknown entities, extracting exact information from noisy webpages, and cross-referencing documents to distinguish misleading answers.The intended answer depends on connecting a 2021 article with papers associated with two people and verifying the resulting conclusion.
  • Existing Online RL Approaches: Search-R1-32B fails to decompose a complex query, produces unsupported conclusions, and leaves unknown information unresolved.Its training turn limit is small, such as 4, resulting in a short tool-use horizon.
  • Prompt-based LLM Agents: Search-o1 (QwQ) can locate relevant species, articles, and papers through many tool calls but misses key information and cannot verify conclusions.The case study attributes this failure to insufficient expert-level reasoning over retrieved contents and history contexts.
  • ASearcher-Web-QwQ: ASearcher-Web-QwQ decomposes the question into precise queries and usually examines one website at a time while summarizing related information.Its search path lists candidate answers, uses cross-document connections, and spends additional turns verifying earlier conclusions.
  • ASearcher-Web-QwQ: ASearcher exhibits uncertainty-aware reasoning, precise key-information extraction, cross-document inference, and grounded verification.These behaviors correspond to examining possibilities, isolating relevant facts, connecting documents, and checking conclusions against related materials.

3 ASearcher

ASearcher is an open-source project for unlocking search intelligence through large-scale RL training. It trains agents to solve complex questions by resolving uncertainties and performing multi-turn tool calls.

  • 3 ASearcher: ASearcher is an open-source project designed to unlock search intelligence through large-scale reinforcement learning.The subsequent framework includes agent design, training data, data synthesis, and asynchronous RL training.
  • 3 ASearcher: ASearcher trains search agents to solve complex questions by exhaustively resolving uncertainties and performing multi-turn tool calls.The paper presents these capabilities as the basis for its search-agent approach.

3.1 Agent Design

ASearcher uses a simple two-tool agent whose search and browsing actions support retrieval, webpage summarization, and answer generation. Reasoning, tool calling, and summarization are optimized jointly through end-to-end RL, with different history and prompting choices for base LLMs and LRMs.

  • Tools: The agent combines a search engine that returns snippets and URLs with a web browser that retrieves webpage contents.Strategically combining both tools is intended to support complex problem solving and key-information extraction.
  • Webpage Summarization: The agent summarizes excessively long webpages into compact summaries, and RL training optimizes this summarization process.Summarization is therefore part of the trained search-agent behavior rather than only a post-processing step.
  • Model Instantiations: ASearcher supports base LLMs such as Qwen2.5-7B/14B and advanced LRMs such as QwQ-32B with different history-management and prompting choices.The framework investigates both model types as search-agent instantiations.
  • Base LLMs: Base LLMs use append-only chronological histories containing responses, search results, and webpage summaries.The full history is provided as input for producing reasoning texts and actions.
  • LRMs: LRMs receive separate prompts for tool selection, summarization, and answering, while histories retain summarized thoughts and tool calls instead of thinking processes.This compacts the input so long-response LRMs retain sufficient token budget for generation.
  • End-to-End Reinforcement Learning: All LLM-generated responses, including thinking, tool calling, and summarization, are trained end-to-end with reinforcement learning.The optimization covers the agent’s main generated components rather than only its final answer.

3.2 Training Data

ASearcher combines filtered open-source QA data with synthesized questions designed to be challenging, reliable, and search-dependent. Its synthesis agent increases complexity through fact injection and uncertainty-enhancing fuzzing, then verifies quality and answer uniqueness.

  • Data sources: The training data combines filtered open-source QA samples with synthesized question-answer pairs designed to teach generalizable search strategies.The open-source component is filtered for difficulty and quality, while synthesis targets challenging search behavior.
  • Open-source data: From 304k open-source QA pairs, 16k challenging samples are retained after filtering for solvability, difficulty, and tool-use requirements.Questions are removed when no response is correct, accuracy is at least 50%, or the answer requires at most one search turn.
  • Data synthesis agent: The synthesis agent iteratively modifies seed questions by injecting related external facts and fuzzing details to increase complexity and uncertainty.Supporting facts are tracked and updated throughout synthesis.
  • Quality verification: Each synthetic question undergoes checks for clarity, factual grounding, difficulty without tools, and answer uniqueness.Difficulty is measured by generating multiple answers without external tools, while uniqueness checks prevent fuzzing from creating multiple valid answers.
  • Search dependence: Questions answerable from the language model’s intrinsic knowledge without search are filtered out because they provide little value for search-agent training.This final filtering emphasizes questions requiring extensive search and reasoning.
  • Dataset scale: Starting from 14,107 seed questions, curation produces 25,624 final entries averaging 4.27 injections and 2.10 fuzzes per selected question.The initial synthesis averages 6.3 injections and 3.2 fuzzes per seed, with up to three retained variations per seed.

3.3 Asynchronous Agentic RL Training

The paper argues that complex search tasks require long trajectories, while their highly variable execution times make batch RL inefficient. Fully asynchronous RL addresses this by separating trajectory execution from model updates and enabling long-horizon training.

  • Long-horizon requirements: Accuracy improves when ASearcher-Web-QwQ is forced to use more minimum turns across GAIA, xBench-DeepSearch, and Frames.The result supports using longer trajectories for complex problem-solving.
  • Runtime variance: The longest training trajectories can use dozens more tool calls and up to two orders of magnitude more tokens than shorter trajectories.This creates highly unpredictable per-trajectory runtime during RL training.
  • Batch-generation bottleneck: Batch generation is bottlenecked by the slowest trajectory, causing GPU idle time and under-utilization despite overlap between rollout and training.A one-step-off system still waits for the current batch to complete before progressing.
  • Asynchronous design: The training paradigm applies asynchronization both to parallel trajectory rollouts and to the separation of rollouts from model updates.These are described as two distinct aspects of the approach.
  • Asynchronous design: Asynchronous rollouts let trajectories independently issue tool and inference requests without waiting for other trajectories to finish.Concurrent server handling improves training efficiency by allowing independent trajectory execution.
  • Asynchronous design: Fully asynchronous RL decouples trajectory execution from model updates, so long trajectories do not block generation and training starts when sufficient trajectories form a batch.Trajectories can span multiple model versions, reducing idle time and improving utilization.

3.4 Training Details

ASearcher formulates search-agent interaction as an MDP and trains the policy with GRPO using grouped trajectory rewards. Dynamic filtering removes queries without informative reward variation, while reward design depends on the agent type.

  • MDP formulation: The MDP state contains history, search results, and retrieved webpages, while actions are generated tokens that may encode tool calls.Tool calls can be extracted from actions through tags such as <search> search query </search>.
  • MDP formulation: At each timestep, the policy maps the current state to an action, with transitions determined by applying any tool calls.The objective is to maximize expected return over the interaction trajectory.
  • GRPO training: GRPO generates groups of trajectories for each question and optimizes the policy using advantages computed from relative rewards within each group.The training objective uses the trajectories sampled from the previous policy.
  • Training efficiency: Dynamic filtering removes queries whose responses all receive identical rewards, including already-solved questions and incorrectly labeled examples.Such queries produce zero advantages and therefore lack meaningful training signals.
  • Reward design: Rewards are sparse and computed at trajectory completion, combining format reward with F1 for base LLMs and using LLM-as-Judge for LRM-based agents.Format rewards are omitted for LRM agents such as QwQ because they inherently preserve proper output formatting.

4 Experiments

ASearcher is evaluated across local-RAG and web-search settings, standard and challenging QA benchmarks, and multiple model scales. It consistently outperforms comparable open-source agents while showing transferable search strategies and increasingly long-horizon behavior.

  • Evaluation Setup: ASearcher evaluates local-RAG and web-based search agents on single-hop, multi-hop, and challenging QA benchmarks using F1 and LLM-as-Judge metrics.The challenging benchmarks report Avg@4 and Pass@4 scores.
  • Training Dynamics: The two-stage curriculum retains questions requiring at least 5 tool calls in its second stage, and keyword analysis shows increased reflection and external-information references.The second stage increases occurrences of keywords such as “doc” and “mention” after step 200.
  • Main Results: ASearcher consistently outperforms existing open-source agents across model sizes, evaluation settings, and both F1 and LasJ metrics.The reported settings include local RAG, web search on standard benchmarks, and web search on challenging benchmarks.
  • Local Knowledge Base with RAG: 60.0 F1 and 65.6 LasJ are achieved by ASearcher-Local-14B, surpassing the larger Search-R1-32B retrieval-based baseline.At 7B, ASearcher reaches average F1 58.0 and LasJ 61.0.
  • Web-based Search and Browsing on Standard QA Benchmarks: 61.5 average F1 is achieved by ASearcher-Web-14B, surpassing SimpleDeepSearcher, while ASearcher-Local-14B generalizes strongly to web search.The local-RAG-trained model is evaluated zero-shot in the web setting.
  • Web-based Search and Browsing on Challenging Benchmarks: 58.7 GAIA Avg@4 and 51.1 xBench-DeepSearch Avg@4 are achieved by ASearcher-Web-QwQ on challenging benchmarks, alongside higher Pass@4 performance than open-source agents.These tasks require real-web interaction and information beyond models’ internal knowledge.
  • Effect of RL Training: +15.0, +22.4, and +14.6 improvements are reported on GAIA, xBench-DeepSearch, and Frames after RL training.Pass@4 also improves, including a 24.0 improvement on xBench-DeepSearch.
  • Training Dynamics: During training, search queries increase to 6 for local agents, while ASearcher-Web-QwQ reaches around 40 tool calls and over 150k tokens in reported dynamics.The broader training setup uses turn limits of 32 for 7B/14B models and 128 for ASearcher-Web-QwQ.

5 Related Works

Related work develops search agents through prompting, supervised trajectories, offline reinforcement learning, and synthetic QA construction. Synthetic data methods aim to scale training beyond human annotation by generating realistic and structurally challenging search tasks.

  • Search Agents: Prompt-based search agents leverage external tools but remain limited by the underlying LLM and cannot be reliably improved through environment feedback.Other approaches construct supervised trajectories or use offline reinforcement learning.
  • Synthetic Data for Search Agents: Synthetic data provides a scalable alternative to relying solely on large-scale human annotation for search-agent training data.Recent methods generate realistic QA trajectories through web interaction and LRM-based curation.
  • Synthetic Data for Search Agents: WebSailor constructs structurally challenging tasks through sampling and fuzzing, while WebShaper uses set-theoretic techniques for search-task construction.These methods represent distinct strategies for synthesizing challenging search data.

6 Conclusion

The paper presents ASearcher as an open-source project combining fully asynchronous agentic RL with large-scale high-quality QA synthesis. Across model sizes and evaluation settings, it reports stronger performance than state-of-the-art open-source agents and aims to support broader agent training.

  • Conclusion: ASearcher combines a fully asynchronous agentic RL training system with a data-synthesis agent for constructing large-scale high-quality QA pairs.The project is released as an open-source system for large-scale RL training.
  • Conclusion: Across model sizes and evaluation settings, ASearcher outperforms state-of-the-art open-source agents.The evaluated instantiations include Qwen2.5-7B, Qwen2.5-14B, and QwQ-32B.
  • Conclusion: The authors present ASearcher’s asynchronous training and data-synthesis insights as potentially useful for training advanced agents in broader applications.

A Full Case Study

The case study contrasts short-horizon or capability-limited search with ASearcher-Web-QwQ’s decomposition, focused browsing, cross-document inference, and verification. The resulting behavior addresses uncertainty and noisy information in a complex multi-document question.

  • Case Study Setup: The GAIA case study requires resolving two conditions and four unknown variables through staged identification and cross-referencing across documents.The answer depends on finding the relevant species, article, papers, and associated evidence.
  • Case Study Challenges: Complex search requires handling uncertainty, extracting exact information from noisy webpages, and confirming conclusions across related documents.The agent must list candidate entities and compare information across webpages and papers.
  • Existing Online RL Approaches: Search-R1-32B makes redundant queries, hallucinates unsupported conclusions, fails to resolve unknown variables, and exhibits only a short tool-use horizon under a turn limit of 4.The case study characterizes this behavior as elementary search strategy learning.
  • Prompt-based LLM Agents: Search-o1 (QwQ) performs extensive tool calls and identifies several entities, but misses key information and fails to verify conclusions despite finding relevant evidence.Its errors are attributed in the case study to insufficient expert-level reasoning over retrieved contents and history.
  • ASearcher-Web-QwQ: ASearcher-Web-QwQ decomposes the query into focused searches, visits relevant websites, summarizes information, connects documents, and confirms its answer with additional tool calls.In the example, it infers “Mice” through cross-document connections and then verifies the conclusion.
  • ASearcher-Web-QwQ: ASearcher-Web-QwQ exhibits uncertainty-aware reasoning, key information extraction, cross-document inference, and rigorous confirmation.
Loading 2508.07976v4…