Dr. Zero: Self-Evolving Search Agents without Training Data

Zhenrui Yue; Kartikeya Upasani; Xianjun Yang; Suyu Ge; Shaoliang Nie; Yuning Mao; Zhe Liu; Dong Wang

Source-linked AI summary

Dr. Zero: Self-Evolving Search Agents without Training Data

Zhenrui Yue, Kartikeya Upasani, Xianjun Yang, Suyu Ge, Shaoliang Nie, Yuning Mao, Zhe Liu, Dong Wang

arXiv:2601.07055v2Jan 11, 2026cs.AI

TL;DR

Open-domain self-evolving search agents face limited question diversity and costly multi-step tool-use training without curated QA data. Dr. Zero addresses this with an external-search proposer–solver feedback loop and HRPO, and it achieves competitive or superior performance to supervised search agents on several benchmarks.

Problem
Open-domain self-evolving search agents are limited by homogeneous one-hop questions, insufficient difficulty adaptation, and inefficient nested sampling for multi-turn reasoning.
Method
Dr. Zero alternately trains a proposer and solver using external search, difficulty-guided rewards, and HRPO group baselines over structurally similar questions.
Results
Dr. Zero is competitive with supervised search agents and surpasses them on several question answering benchmarks without curated QA training data.
Takeaways & Limitations
Self-evolution can improve search and reasoning capabilities using externally grounded, autonomously generated questions without human-written questions or annotated answers.
Takeaways & Limitations
Grounding requirements and the absence of difficulty reward for zero-pass cases reduce but do not eliminate incorrect or ambiguous synthetic answers.

Abstract

from arXiv · show

As high-quality data becomes increasingly difficult to obtain, self-evolution without curated training data has emerged as a promising paradigm. This approach allows large language models (LLMs) to autonomously generate and solve complex problems, thereby improving their reasoning capabilities. However, multi-turn search agents struggle in this setting due to limited question diversity and the substantial compute required for multi-step reasoning and tool use. In this work, we introduce Dr. Zero, a framework that enables search agents to effectively self-evolve without human-annotated training data, relying solely on an external search engine as their knowledge environment. In particular, we design a self-evolution feedback loop where a proposer generates structurally diverse questions to train a solver initialized from the same base model. As the solver evolves, it incentivizes the proposer to produce increasingly difficult yet solvable tasks, thus establishing an automated curriculum to refine both agents. To enhance training efficiency, we also introduce hop-grouped relative policy optimization (HRPO). This method clusters structurally similar questions to construct group-level baselines, effectively minimizing the sampling overhead in evaluating each query's individual difficulty and solvability. Consequently, HRPO significantly reduces the compute requirements for proposer training and reward estimation without compromising performance or stability. Extensive experimental results demonstrate that Dr. Zero matches or surpasses fully supervised search agents on several question answering benchmarks, showing that strong agentic search and evidence-grounded reasoning can emerge solely through self-evolution.

1 Introduction

Dr. Zero targets open-domain self-evolving search without curated QA data, addressing limited question diversity, difficulty adaptation, and high multi-turn training cost. It alternates proposer and solver improvement with difficulty-guided rewards and HRPO, achieving competitive or superior performance without curated training data.

Open-domain self-evolving search remains under-explored because existing approaches often rely on human questions, extensive contexts, or ground-truth annotations.
Data-free proposer training tends toward simple one-hop questions and fails to match supervised baselines on complex multi-hop queries.Standard GRPO also increases compute through nested sampling of multiple questions and solver responses.
Dr. Zero uses an external search engine to improve proposer and solver performance without human-written training questions or answers.Its multi-turn tool-use pipeline and difficulty-guided reward produce complex, multi-hop, verifiable questions.
HRPO clusters structurally similar questions to provide group-level baselines for advantage estimation, eliminating expensive nested sampling during proposer training.
Without curated QA training data, Dr. Zero matches aggregate fully supervised performance at the 3B scale and outperforms supervised baselines on several individual benchmarks.

2 Related Work

Related work covers reinforcement-learning methods for language models, search and retrieval augmentation, and self-evolving LLMs. These lines of work provide the training, external-knowledge, and autonomous-improvement context for Dr. Zero.

Policy-gradient reinforcement learning improves agent policies by learning from experience and maximizing cumulative rewards.Actor-critic methods use learned value baselines to reduce gradient variance, while simpler offline alternatives also exist.
Search and retrieval augmentation integrates external knowledge into language modeling, while iterative retrieval improves performance on complex questions.IRCoT is cited as an example of multi-step retrieval for knowledge-intensive tasks.
Self-evolving LLMs iteratively generate and learn from their own experiences to enhance model capabilities without human annotations.Early approaches include self-play and self-rewarding loops in which models generate or evaluate their own outputs.
Dr. Zero’s feedback loop uses solver feedback to guide the proposer toward verifiable and challenging queries while improving the solver’s search and reasoning capabilities.

3 Methodology

Dr. Zero trains proposer and solver search agents in a self-evolution loop using an external search engine, without curated QA data. HRPO groups structurally similar questions for proposer optimization, while solver feedback promotes diverse, challenging, and verifiable queries.

3.1 Setup: The proposer and solver use an external search engine as their knowledge environment and maximize separate expected rewards.The framework avoids demonstrations, human-written questions, and annotated answers.
Self-evolution loop: Solver feedback creates a curriculum in which the proposer generates increasingly complex queries as the solver improves.The proposer learns from solver outcomes, while the solver trains on generated QA pairs.
3.2 Proposer Training: HRPO computes advantages by grouping structurally similar questions across prompts instead of repeatedly sampling responses for each seed prompt.Advantages are standardized over questions sharing the same hop complexity, reducing the instability of a global baseline.
3.2 Proposer Training: The proposer reward penalizes questions that receive either no correct solver solutions or universally correct solutions, targeting solvable but nontrivial tasks.The reward uses solver pass rate as a proxy for verifiability and difficulty, with an additional format reward for structural quality.
3.3 Solver Training: GRPO trains the solver on proposer-generated data using empirical group statistics and outcome-based rewards for final-answer correctness.The solver is refined without a separate value function as generated queries become more complex.
Evaluation: Dr. Zero matches or outperforms supervised search agents while requiring no curated QA training data.The reported comparison covers few-shot and supervised methods in Table 1.

4 Experiments

Dr. Zero is evaluated across one-hop and multi-hop open-domain QA benchmarks against few-shot, supervised, and data-free baselines. Results show strong benchmark performance, efficient self-evolution, and model-size- and question-distribution-dependent learning dynamics.

Experiment Settings: Dr. Zero is evaluated on three one-hop and four multi-hop open-domain QA datasets using Qwen2.5 3B and 7B base models.The evaluation covers Natural Questions, TriviaQA, PopQA, HotpotQA, 2WikiMQA, MuSiQue, and Bamboogle.
Overall Performance: Dr. Zero substantially outperforms few-shot baselines across benchmarks, achieving 0.397 EM on NQ with Qwen2.5-3B versus 0.106 for few-shot prompting.The comparison includes standard prompting, IRCoT, Search-o1, and RAG, alongside supervised baselines.
Overall Performance: On single-hop tasks, the 3B model outperforms supervised Search-R1 by 22.9%, 6.5%, and 18.4% on NQ, TriviaQA, and PopQA, respectively.On multi-hop tasks, the 7B variant reaches roughly 90% of Search-R1’s performance and surpasses it on 2WikiMQA.
Data-Free Baselines: Dr. Zero consistently performs best among evaluated data-free methods, with an average relative gain of 83.3% over R-Zero* across four multi-hop benchmarks.The comparison augments SQLM and R-Zero with multi-turn reasoning and search, producing SQLM* and R-Zero*.
Training Dynamics: Solvers reach a performance peak within approximately 50 steps, while the second iteration yields average benchmark gains of 4.93% for 3B and 1.64% for 7B.After the second iteration, the 3B model gains 2.2%, whereas the 7B model declines from 0.372 to 0.360 and later iterations provide marginal or no gains.
Question Distribution: For the 3B model, the 4:3:2:1 distribution of 1/2/3/4-hop questions achieves the best multi-hop performance, with an average 0.220 EM.Increasing the proportion of multi-hop questions does not itself improve the 3B model’s performance.

5 Conclusion

Dr. Zero enables language agents to improve reasoning and search without curated QA training data. Its proposer-solver loop and HRPO support diverse, challenging, verifiable questions, while future work targets stability, reward hacking, and bias amplification.

Conclusion: Dr. Zero enhances language-agent reasoning and search through self-evolution without curated QA training data.The framework autonomously generates diverse, increasingly challenging, short-form, verifiable open-domain questions without human-written questions or annotated answers.
Conclusion: The iterative proposer-solver paradigm generates diverse, increasingly challenging questions, while HRPO clusters structurally similar queries to estimate advantages efficiently.The framework is reported as competitive with supervised search agents and superior on several question-answering benchmarks.
Future Work: Future work will address performance plateaus, entropy collapse in larger models, reward hacking, and bias amplification.These goals concern extending self-evolution stability and safeguarding the feedback loop without human supervision.

Implementation

Dr. Zero alternates proposer and solver optimization: the proposer generates QA pairs, the solver computes rewards, and HRPO updates the proposer. Implementation uses structured format rewards, external retrieval, and exact-match scoring.

Optimization Loop: Dr. Zero alternates proposer and solver optimization, using generated QA pairs as solver inputs and solver rewards to update the proposer with HRPO.The proposer produces one response per prompt, from which a question and answer are extracted before solver reward computation.
Reward Design: HRPO format rewards require valid think structure, tool usage, extractable question tags, and extractable answer tags.The four components sum to a maximum format reward of 0.5, while the difficulty score ranges from 0 to 1.
Search and Scoring: The external search engine indexes a corpus with E5 embeddings, retrieves the top three documents using approximate nearest-neighbor search, and evaluation uses exact match.Retrieved passages are formatted as tool responses for training and inference.

Additional Results

Additional analyses show that Dr. Zero generates broader hop-structure coverage than data-free baselines, while HRPO reduces rollout requirements and key reward components support performance. Statistical testing further indicates that Dr. Zero can match or exceed supervised baselines.

Computational Efficiency: HRPO substantially decreases wall-clock time and GPU-hours per iteration while improving average GPU utilization relative to standard GRPO.The comparison uses a single 8xH100 node for 3B proposer training; solver training takes approximately 3 hours for both methods.
Ablation Results: Removing the format reward lowers average performance from 0.304 to 0.289, while the default difficulty-based reward outperforms a parabolic alternative.These ablations support the roles of structural validity and difficulty-oriented reward shaping in proposer training.
Computational Efficiency: HRPO uses fewer than one-third as many solver rollouts per prompt as GRPO by replacing 16 rollouts with 5.GRPO generates 4 questions with 4 rollouts each, while HRPO generates 1 question with 5 solver predictions.
Significance Testing: Additional significance testing finds that Dr. Zero can match or exceed supervised baselines, including consistent gains for the 3B backbone on knowledge-intensive tasks.The reported tests use greedy decoding and average results with standard deviations.
Generated Question Analysis: 93.4% and 95.6% of SQLM* and R-Zero* questions are one-hop, whereas Dr. Zero allocates 60% of seed prompts to two- through four-hop questions.The analysis defines diversity as coverage across question hop structures and reports Dr. Zero’s default 4:3:2:1 generation schedule.