PaperlayerView exact revision on arXiv

Source-linked AI summary

DeepSearch-World: Self-Distillation for Deep Search Agents in a Verifiable Environment

Xinyu Geng, Xuanhua He, Sixiang Chen, Yanjing Xiao, Fan Zhang, Shijue Huang, Haitao Mi, Zhenwen Liang, Tianqing Fang, Yi R. Fung

arXiv:2607.07820v1cs.CL

TL;DR

Self-evolving tool-use agents remain difficult to train because existing supervision is static, sparse, or unreliable for long-horizon interactions. The paper introduces a deterministic, verifiable environment and self-distillation framework that iteratively trains agents on verified tool-use experience, yielding improvements over the Qwen3.5-9B backbone and competitive performance with strong open-source agents.

  • Problem

    Self-evolving tool-use agents lack reliable process-level supervision for improving from their own long-horizon interactions.

  • Method

    DeepSearch-Evolve combines the verifiable DeepSearch-World environment, scaffold teacher trajectories, and an iterative self-evolving training loop.

  • Results

    DeepSearch-World-9B substantially improves over its Qwen3.5-9B backbone and remains competitive with strong open-source agents across deep-search and reasoning benchmarks.

  • Takeaways & Limitations

    Deterministic verification and grounded process supervision support scalable improvement of web agents from self-generated experience.

  • Takeaways & Limitations

    The Wikipedia-based environment limits coverage and domain diversity, while the evolving-SFT update rule may restrict generalization and flexibility.

Abstract

from arXiv · show

Training tool-use agents to improve from their own experience remains challenging, as supervised fine-tuning relies on fixed teacher-distilled trajectories, while sparse-reward reinforcement learning provides weak supervision for long-horizon interactions. We present DeepSearch-Evolve, a self-distillation framework for web agents built on DeepSearch-World, a deterministic and verifiable environment with reproducible search and page-reading tools. DeepSearch-World contains 420K multi-hop QA tasks constructed from entity-level random walks and supports key agentic cognitive behaviors useful for self-evolving, including progress verification, grounded reflection, and failure recovery. DeepSearch-Evolve iteratively performs trajectory generation, filtering, data mixing, and fine-tuning to train stronger agents. Without distillation from more capable models, DeepSearch-World-9B achieves competitive performance compared with open-source agents, reaching 31.2% on BrowseComp, 61.5% on GAIA, and 93.4% on HotpotQA, showing that verifiable environments enable scalable self-evolution for long-horizon web agents. We will release the environment, 420K training pool, validation set, model, and code to facilitate future research on self-improving deep search agents.

1 Introduction

Existing self-evolving tool-use agents are constrained by static imitation, sparse rewards, or unreliable dense supervision, motivating a deterministic, verifiable environment for process-level feedback. DeepSearch-World and DeepSearch-Evolve provide this setting for iterative self-distillation from verified tool-use experience, with competitive open-source-agent performance without proprietary-model distillation.

  • Motivation: Autonomous improvement remains difficult because supervised fine-tuning saturates around backbone capability and trajectory diversity, while rollout optimization relies on sparse verifiable rewards.Agentic supervision must cover tool selection, evidence verification, query reformulation, and progress tracking, motivating deterministic and verifiable environments.
  • Contributions: DeepSearch-Evolve enables agents to iteratively improve from their own verified tool-use experience through scaffold process supervision for long-horizon interactions.The teacher tracks progress, evidence, failed attempts, and recovery-related information during tool use.
  • Contributions: DeepSearch-World is a deterministic, verifiable offline environment with search and browse tools and 420K aligned multi-hop QA tasks over Wikipedia.It provides reproducible observations and entity-level verification of intermediate tool-use progress, unlike live web environments.
  • Results: DeepSearch-World-9B achieves competitive performance among open-source agents without distillation from strong proprietary models, while sustaining longer tool-use interactions and using tools more effectively.Its performance also benefits from verified trajectory filtering and scaffold-to-ReAct conversion.

2 Related Work

Prior work studies tool-use agents, web navigation, trajectory tuning, and self-improvement, but sparse rewards and weak or unstable verification limit optimization. DeepSearch-Evolve addresses these limitations with a deterministic, verifiable Wikipedia-based environment and evolving SFT for stable self-distillation.

  • Tool-Use Agents and Environments: The paper contrasts prior tool-use training, web navigation, and trajectory-level tuning with a deterministic offline Wikipedia environment for scalable, reproducible agent trajectory generation.Tool-augmented agents extend capabilities through search, API invocation, and environment interaction (Yao et al., 2023; Schick et al., 2024; Chen et al., 2026b), while related training explores large-scale tool use and web navigation (Li et al., 2025a; Geng et al., 2026; Chen et al… ).
  • Self-evolving Agents: Self-improvement methods refine agent behavior through model-generated trajectories and verification signals, while GRPO-style group RL has become a dominant tool-use post-training paradigm.These approaches include Zelikman et al. (2022), Singh et al. (2024), and Yuan et al. (2023), alongside GRPO-style methods (Team et al., 2025c; Geng et al., 2025; Huang et al., 2026).
  • Self-evolving Agents: Sparse rewards and weak verifiability make GRPO-style optimization unstable, while OPSD methods rely on teacher policies that may provide unreliable or nonstationary target distributions.Recent OPSD work seeks to distill from on-policy generation (Hübotter et al., 2026; Zhao et al., 2026; Ye et al., 2026), but its teacher target is not always reliable or stationary.
  • Self-evolving Agents: DeepSearch-Evolve introduces evolving SFT with a verifiable tool environment to enable stable self-distillation.Its self-evolving loop uses trajectory generation, rejection sampling, importance-sampling-based data mixing, and SFT training.

3 Methodology

DeepSearch-Evolve combines a deterministic, verifiable offline search environment and scaffold teacher with iterative self-distillation into a deployable ReAct agent. Its design supports objective progress verification, grounded reflection, failure recovery, and training from verified tool-use experience.

  • 3.1 Overview: DeepSearch-Evolve integrates DeepSearch-World, a scaffold teacher, and an asynchronous iterative generation-and-training loop for self-distilling search agents.The framework has three components: verifiable tools and multi-hop QA construction, scaffolded trajectory generation, and iterative self-evolving training.
  • 3.1 DeepSearch-World: 420K multi-hop QA instances are constructed from entity-level random walks and grounded in an offline Wikipedia corpus of approximately 10 million searchable entries.The entity alignment ensures each question’s required evidence is retrievable and verifiable within DeepSearch-World.
  • 3.1 DeepSearch-World: DeepSearch-World provides deterministic search and visit tools, while order-free entity matching verifies objective progress after each call without expensive LLM judgments.Search returns ranked snippets and URLs, visit retrieves corresponding full articles, and failed calls trigger staged rule-based reflection.
  • 3.2 Scaffold Teacher: The scaffold teacher cycles through PLAN, ACT, and END to track subgoals, evidence, failed strategies, reflections, and grounded final answers.Its trajectories combine successful tool use with recoverable failures, while the final answer is generated from verified working memory.
  • 3.3 Self-Evolving Training: At each round, the current model generates scaffold trajectories, retains verified successful examples, converts them to ReAct supervision, and trains the next model from its own experience.Trajectory-level filtering removes redundant evidence, weak goal alignment, and inconsistent reasoning; scaffold-specific planning and reflection are distilled into a deployable ReAct policy.

4 Experiments

DeepSearch-World-9B consistently outperforms Qwen3.5-9B-Instruct across the evaluated benchmarks, with gains attributed to transferable tool-use behaviors such as query reformulation, evidence grounding, and multi-step synthesis. Additional experiments show that larger verified data pools, longer evidence-gathering interactions, reflection rewriting, and answer-correctness filtering strengthen self-distillation.

  • Experimental Setup: The evaluation covers seven deep-search and reasoning benchmarks against proprietary and open-source agents under two evaluation settings.The benchmarks are BrowseComp, BrowseComp-ZH, HLE, GAIA, xbench, HotPotQA, and Search-QA.
  • Main Results: +23.8 BrowseComp, +22.9 BrowseComp-ZH, +9.0 HLE, +37.6 GAIA, +29.0 xbench, and +48.1 HotpotQA over Qwen3.5-9B-Instruct demonstrate consistent cross-benchmark gains.The gains are attributed to transferable query reformulation, evidence grounding, and multi-step synthesis; the lower BrowseComp-ZH score reflects English-only training while still showing partial cross-lingual transfer.
  • Data Scale: The 420K training pool reaches a higher validation plateau, lower residual format error, and more stable tool-success and entity-hit-ratio gains than 100K.The larger pool preserves sufficient unseen instances across rounds, whereas the smaller pool is exhausted earlier and yields weaker supervision.
  • Tool-Use Behavior: 70% advanced capability versus 19% for Qwen3.5-9B-Instruct shows stronger long-horizon tool-use strategy.DeepSearch-World sustains 18.0 interaction rounds and 5.4 visit calls on average, compared with 4.7 rounds and 0.9 visit calls for the baseline.
  • Trajectory Conversion: 31.9 versus 25.0 on DeepSearch-Val shows that the full trajectory-conversion pipeline improves over vanilla SFT on 10K converted trajectories.Removing reflection rewriting reduces performance to 16.7 and 15.7, while state internalization provides smaller consistent gains.
  • Trajectory Filtering: 58.2 SearchQA with rejection sampling and quality filtering exceeds 54.9 with rejection sampling alone and 46.4 without the dominant safeguard.Rejection sampling supplies the largest gain by verifying answer correctness, while quality filtering removes redundant, weakly aligned, or inconsistent correct traces.

5 Conclusion

The paper introduces DeepSearch-World, a deterministic and verifiable offline-Wikipedia tool environment, and DeepSearch-Evolve, a self-distillation framework built on it. DeepSearch-World-9B substantially improves over Qwen3.5-9B while remaining competitive with strong open-source agents.

  • Environment: DeepSearch-World provides reproducible observations, entity-level progress verification, and grounded reflection signals for stable, verifiable tool-use trajectories.
  • Method: DeepSearch-Evolve performs self-distillation by generating verified trajectories and converting scaffold supervision into standard ReAct-format training.
  • Results: DeepSearch-World-9B substantially improves over Qwen3.5-9B across deep-search and reasoning benchmarks while remaining competitive with strong open-source agents.

Limitation

The environment’s Wikipedia basis limits coverage and domain diversity, while its evolving-SFT update rule leaves RL-style self-evolution and higher-level tool-use capabilities underexplored. Broader knowledge sources and improved RL training could expand generalization and the scope of verifiable self-evolution.

  • Limitation: Wikipedia-based environments limit coverage and domain diversity, motivating extensions to broader knowledge sources for improved generalization.
  • Limitation: The current update rule relies on evolving SFT, while RL-style or OPSD-based updates may improve generalization and flexibility.
  • Limitation: Injecting planning, error recovery, and tool-use strategies into RL training remains underexplored, limiting broader verifiable self-evolution for tool-use agents.

Ethical Considerations · A Details of Data and environment construction.

DeepSearch-World uses publicly available, licensed Wikipedia content in a fully offline, deterministic environment without private or newly generated personally identifiable information. Its 420K-instance multi-hop QA dataset is designed for genuine multi-step tool use, while acknowledging inherited societal biases and the need for broader data and stronger safety alignment.

  • Ethical Considerations: The work uses publicly available Wikipedia content licensed under CC-BY-SA and releases the environment and dataset for research with attribution to Wikipedia contributors.
  • Ethical Considerations: The dataset does not collect private or newly generated personally identifiable information, though it may include public names and biographical facts from Wikipedia.
  • Ethical Considerations: The environment is fully offline and deterministic, reducing risks from uncontrolled web access during training.
  • Ethical Considerations: Wikipedia may contain societal biases that models trained on it could inherit.Future work should explore more diverse data sources and stronger safety alignment for tool-use agents.
  • A Details of Data and environment construction.: 420K instances comprise a large-scale multi-hop QA dataset designed to require genuine multi-step tool use.The dataset construction pipeline has two phases.
  • A Details of Data and environment construction.: The construction pipeline for the 420K-instance dataset consists of two phases.

A.1 Data and Environment Construction … B Experimental Details

The paper constructs a deterministic Wikipedia-based environment from multi-hop knowledge trees and implements efficient search, page reading, and rollout mechanisms for reproducible agent interaction. Its experimental setup combines indexed offline retrieval, context management, and backend-agnostic text-based tool calling.

  • A.1 Data and Environment Construction: The environment builds knowledge trees by breadth-first exploration, selecting up to three relevant hyperlinks per level until four levels or six visited pages.Each tree edge records the semantic relationship between parent and child articles.
  • A.1 Data and Environment Construction: Questions are generated from randomly sampled 2–4 subtrees and require cross-referencing all entities across multiple Wikipedia articles.Feature fuzzification replaces specific values with approximate descriptors to force factual search and verification rather than memorization.
  • A.1 Data and Environment Construction: The offline corpus contains approximately 10 million English Wikipedia entries, stored in JSONL with SQLite offsets and indexed by Pyserini BM25 for keyword retrieval.Missing articles can be supplemented through rate-limited online scraping.
  • A.2 Tool Implementation Details: web_search_wiki returns the top five ranked articles with titles, captions, and deterministic URLs from a shared Pyserini BM25 index.The interface exposes search-style results rather than full documents, supporting reproducible retrieval across workers.
  • A.2 Tool Implementation Details: visit_wiki retrieves full articles through normalized-title URL lookup and two-level offset indexing, with shared-memory caching available for parallel workers.Long pages receive lightweight summaries or deterministic character-level truncation, preserving a browser-like schema while keeping retrieval efficient and deterministic.
  • A.3 Teacher Rollout Implementation Details: Teacher rollouts retain the most recent interaction steps in full while summarizing earlier steps into a progress state to control context length.This sliding-window strategy preserves recent observations and tool-use rationales within the interaction budget.
  • A.3 Teacher Rollout Implementation Details: Tool use follows a backend-agnostic text-based function-calling protocol with delimiter tokens for function names, arguments, results, and observations.Regular-expression parsing dispatches calls, while long observations are condensed or deterministically truncated when context requires it.

B.1 Benchmark Details · B.2 Baseline Details

The evaluation spans six search-intensive benchmarks covering multilingual browsing, expert reasoning, and search-based QA. It compares direct-reasoning models with similarly scaled open-source search agents, while noting that the latter comparison is practical rather than strictly parameter-matched.

  • B.1 Benchmark Details: The evaluation uses six search-intensive benchmarks covering English browsing, Chinese browsing, expert-level reasoning, and search-based QA.
  • B.1 Benchmark Details: BrowseComp tests persistent discovery of hard-to-find, entangled web information through 1,266 English browsing questions.
  • B.1 Benchmark Details: BrowseComp-ZH evaluates Chinese-web browsing with 289 native multi-hop questions spanning 11 domains.
  • B.1 Benchmark Details: GAIA-Text contains 103 text-only development questions requiring real-world multi-step web search and reasoning, while xBench-DeepSearch contains 100 Chinese deep-search questions.
  • B.1 Benchmark Details: Search-QA combines single-hop datasets such as NQ, TriviaQA, and PopQA with multi-hop datasets including HotpotQA, 2Wiki, MuSiQue, and Bamboogle.
  • B.2 Baseline Details: Direct Reasoning compares single-pass answers without retrieval or tools against GPT-5, Claude-4-Sonnet, Claude-3.7-Sonnet, OpenAI Deep Research, OpenAI-o3, and Qwen3.5-9B-Instruct.
  • B.2 Baseline Details: Open Source Agents compares recent search and deep-research systems at comparable scales, including R1-Searcher-7B, Search-R1-7B, ZeroSearch-7B, ASearcher-7B, DeepResearcher-7B, PokeeResearch-7B, WebSailor-7B, WebExplorer-8B, Marco-DR-8B, MiroThinker-v1.0-8B, and DeepDive-9B.
  • B.2 Baseline Details: Because exact 9B public deep-search agents are limited, the study includes nearby 7B–8B systems as a practical comparison within similar deployment budgets, not a strict parameter-matched contest.

B.3 Training Hyperparameter

The self-evolving loop runs for 15 rounds, using thresholded trajectory collection, importance-sampling mixing, exponential decay, and fixed fine-tuning settings.

  • B.3 Training Hyperparameter: 15 rounds define the self-evolving loop, with each round governed by the stated generation and training configuration.The loop is run for R=15 rounds.
  • B.3 Training Hyperparameter: 10,000-question generation batches trigger training after 4,000 correct trajectories, followed by importance-sampling mixing to 4,000 trajectories.The thresholds are B=10,000, Cmin=4,000, and Ntarget=4,000.
  • B.3 Training Hyperparameter: γ=0.5 applies exponential decay across rounds, while training uses one epoch per round, a 5 × 10−6 learning rate, cosine scheduling, and 10% warmup.Training also uses DeepSpeed ZeRO-2, BFloat16 precision, and a maximum sequence length of 32,768.
  • B.3 Training Hyperparameter: The qwen3_5_nothink template strips <think> tokens during training to focus learning on trajectory structure.This template is part of the stated training configuration.

B.4 Asynchronous Training Efficiency · B.5 Advanced Capability Score. · C Prompts

The system improves training efficiency through asynchronous generation and background training, while evaluating advanced capabilities across five dimensions. Its prompts specify reject-sampling judgments, trajectory quality checks, progress tracking, tool-use formats, final-answer constraints, and the agent’s interaction protocol.

  • B.4 Asynchronous Training Efficiency: Approximately 2–3 days are required for a complete 5-round self-evolving loop on a single 2-node H20 cluster.Generation processes approximately 10,000 questions per batch, while full SFT on 4,000 trajectories completes during the next generation batch.
  • B.4 Asynchronous Training Efficiency: Generation never blocks on training: trajectories are produced in the background, and model swaps occur only at round boundaries.This asynchronous orchestration enables generation and training to proceed concurrently.
  • B.5 Advanced Capability Score.: The advanced capability score averages pass rates for planning, memory, reasoning, self-correction, and evidence gathering.An LLM judge assesses whether agents decompose questions, maintain intermediate evidence, connect evidence across hops, revise failures, and retrieve sufficient support.
  • C.1 Trajectory Generation: Reject sampling uses an evaluator that classifies model answers as correct, incorrect, or not attempted, with rules for numerical near-matches, requested detail, and inferable omissions.The evaluator receives the question, standard answer, and prediction and must return only A, B, or C.
  • C.2 Quality Filter: The quality filter judges tool calls and reasoning for non-redundancy, goal alignment, and logical, accurate, evidence-based reasoning.A call is reasonable when it is necessary, query-driven, efficient, precise, non-redundant, and specification-compliant.
  • C.3 ReAct Trajectories: ReAct trajectories alternate concise thinking, one tool call, tool responses, further reasoning, and a final answer.The trajectory format explicitly represents repeated think–tool_call–tool_response cycles before answer generation.
  • C.4 Planning User Prompt; C.5 Action User Prompt: The planning prompt updates a JSON progress state by moving completed items, revising remaining tasks, and recording new information and experience.The action prompt separately requires either a tool call when information is missing or a direct answer only after confirmation, while enforcing the exact response format.
  • C.6 Ending User Prompt; C.7 System Prompt: Ending and system prompts require concise final outputs, confirmed facts from visited pages, one tool call at a time, and a mandatory search-then-visit workflow.The system prompt also distinguishes injected tool results, reflections, and state updates from the action module’s outputs.

D Case Study · E Large Language Model Usage

The case study demonstrates a verified, stepwise search that identifies L’Oracolo by Franco Leoni through a chain of institutional and production-history clues. The paper also states that Large Language Models were used only for grammar correction and LaTeX formatting.

  • D Case Study: The organization was identified as Glasgow Grand Opera Society, founded in 1906 from Moody-Manners Opera Company profits to support touring companies’ choruses.This founding mechanism resolved the question’s opening clue.
  • D Case Study: The trajectory uses progress lists, grounded reflection, targeted searches, and independent verification to resolve each remaining clue before returning the final answer.The agent explicitly tracks completed and pending subgoals while checking candidate evidence against later production-history clues.
  • D Case Study: The society’s first independent production was Gounod’s Faust in 1911, whose 1859 Paris premiere occurred at the Théâtre Lyrique.Verifying both production-history clues strengthened the institutional match.
  • D Case Study: Erik Chisholm, a composer-organist who founded the Active Society for the Propagation of Contemporary Music, led the society’s 1930s British premiere of Mozart’s Idomeneo.The premiere took place in 1934.
  • D Case Study: The search concludes that the Glasgow Grand Opera Society’s 1994 revival was L’Oracolo, composed by Franco Leoni.The work was a 1905 opera and the first UK production in 80 years.
  • D Case Study: From 1993, annual productions moved to the New Athenaeum Theatre at the Royal Conservatoire of Scotland, preceding the 1994 L’Oracolo performance.The venue move supplied the later institutional clue needed to complete the chain.
  • E Large Language Model Usage: The paper states that Large Language Models assisted only with grammar correction and LaTeX formatting, with no prompts requesting positive reviews.The authors explicitly limit the reported language-model assistance to editorial tasks.
Loading 2607.07820v1…