PaperlayerView exact revision on arXiv

Source-linked AI summary

ZeroSearch: Incentivize the Search Capability of LLMs without Searching

Hao Sun, Zile Qiao, Jiayan Guo, Xuanbo Fan, Yingyan Hou, Yong Jiang, Pengjun Xie, Yan Zhang, Fei Huang, Jingren Zhou

arXiv:2505.04588v3cs.CL

TL;DR

LLM search training is challenged by unpredictable document quality and high live-search API costs. ZEROSEARCH simulates search with a supervised-fine-tuned retrieval module and progressively noisier rollouts during RL. It outperforms real-search-based models, generalizes across model types and sizes, and supports multiple RL algorithms, while requiring GPU infrastructure.

  • Problem

    Live-search RL faces unpredictable document quality and high API costs, while LLMs need external information to reduce hallucinated or outdated responses.

  • Method

    ZEROSEARCH uses supervised fine-tuning to create a retrieval module that generates useful or noisy documents, then increases retrieval difficulty through curriculum rollouts.

  • Results

    ZEROSEARCH outperforms real search-based models, generalizes across base and instruction-tuned LLMs of varying sizes, and supports a wide range of RL algorithms.

  • Takeaways & Limitations

    LLMs can be used as controllable substitutes for real search engines during RL training, with 7B simulation comparable to Google Search and 14B simulation surpassing it.

  • Takeaways & Limitations

    Deploying the simulated search LLM requires GPU servers and therefore introduces infrastructure costs despite being more cost-effective than commercial API usage.

Abstract

from arXiv · show

Effective information searching is essential for enhancing the reasoning and generation capabilities of large language models (LLMs). Recent research has explored using reinforcement learning (RL) to improve LLMs' search capabilities by interacting with live search engines in real-world environments. While these approaches show promising results, they face two major challenges: (1) Uncontrolled Document Quality: The quality of documents returned by search engines is often unpredictable, introducing noise and instability into the training process. (2) Prohibitively High API Costs: RL training requires frequent rollouts, potentially involving hundreds of thousands of search requests, which incur substantial API expenses and severely constrain scalability. To address these challenges, we introduce ZeroSearch, a novel RL framework that incentivizes the capabilities of LLMs to use a real search engine with simulated searches during training. Our approach begins with lightweight supervised fine-tuning to transform the LLM into a retrieval module capable of generating both useful and noisy documents in response to a query. During RL training, we employ a curriculum-based rollout strategy that incrementally degrades the quality of generated documents, progressively eliciting the model's reasoning ability by exposing it to increasingly challenging retrieval scenarios. Extensive experiments demonstrate that ZeroSearch effectively incentivizes the search capabilities of LLMs using a 3B LLM as the retrieval module. Remarkably, a 7B retrieval module achieves comparable performance to the real search engine, while a 14B retrieval module even surpasses it. Furthermore, it generalizes well across both base and instruction-tuned models of various parameter sizes and is compatible with a wide range of RL algorithms.

1 Introduction

LLMs need external information because pretrained knowledge is static, making them vulnerable to hallucinated or outdated responses. ZEROSEARCH addresses this by simulating search during RL training, controlling document quality while avoiding real search-engine interaction.

  • Static pretrained knowledge leaves LLMs vulnerable to hallucinated or outdated information, motivating access to external sources.
  • RL research has begun training LLMs to search for relevant information, including approaches that interact with commercial search engines.
  • ZEROSEARCH uses an LLM-based retrieval module to generate useful or noisy documents without interacting with real search engines.
  • A curriculum rollout progressively degrades generated-document quality, exposing the policy model to increasingly challenging retrieval scenarios.
  • Experiments cover in-domain and out-of-domain datasets and report superiority to real-search-engine models, zero API cost, broad generalization, and compatibility with multiple RL algorithms.

2 Related Work

Related work improves retrieval-augmented generation through prompting, supervised fine-tuning, test-time scaling, and reinforcement learning. Recent RL methods increasingly train models to search autonomously, including systems using live commercial search engines.

  • Early RAG methods guide query generation, query decomposition, and multi-turn retrieval through prompts.
  • Prompt-based approaches can be effective but require intricate prompt engineering and substantial reasoning capability.
  • Subsequent work uses supervised fine-tuning and test-time scaling, including MCTS, to improve retrieval efficiency or expand inference search.
  • RL-based studies train LLMs for information retrieval, with representative systems including Search-R1, R1-Searcher, and ReSearch.
  • DeepResearcher and WebThinker extend this direction by enabling live interaction with commercial search engines.

3 ZEROSEARCH

ZEROSEARCH replaces real search-engine interaction with an LLM simulation and structured multi-turn reasoning. It controls retrieval difficulty through curriculum-based noise, uses accuracy-focused rewards, and stabilizes training by masking externally generated document tokens.

  • 3.1 Reinforcement Learning without a Search Engine: ZEROSEARCH formulates RL without a search engine by using a fixed LLM to simulate search-engine results.
  • 3.2 Training Template: The policy model follows iterative <think>, <search>, and <answer> stages to structure reasoning, retrieval, and final answering.
  • 3.3 Search Simulation Tuning: Lightweight SFT trains the simulation LLM to generate useful and noisy documents, with prompt keywords controlling document quality.
  • 3.4 Rollout with Curriculum Search Simulation: Curriculum rollout increases the probability of noisy documents over training, moving from basic output structures toward harder retrieval scenarios.The schedule uses initial and final noise probabilities, training progress, and exponential base b, defaulting to 4.
  • 3.5 Reward Design: The reward uses F1 rather than exact match because exact match encouraged excessively long answers through reward hacking.F1 balances precision and recall; no separate format reward is added because outputs are consistently well formed.
  • ZEROSEARCH supports REINFORCE, PPO, and GRPO, allowing the framework to work with multiple RL optimization algorithms.
  • Loss masking excludes externally generated document tokens from gradient computation, stabilizing RL while preserving retrieval-augmented generation.

4 Main Results

ZEROSEARCH is evaluated against prompting, retrieval-augmented generation, and reinforcement-learning baselines across seven question-answering datasets. It consistently outperforms the baselines, including Search-R1, across both in-domain and out-of-domain settings and across model families and sizes.

  • The evaluation uses Exact Match for question-answering performance and F1 as the reward metric across reinforcement-learning methods.The experiments compare vanilla prompting, advanced RAG, and RL-tuning baselines, with Search-R1 as the real-search comparison.
  • ZEROSEARCH consistently outperforms all baseline methods across the evaluated datasets.Table 3 reports comparisons across seven datasets and different LLM backbones.
  • The performance advantage holds on both in-domain datasets, NQ and HotpotQA, and out-of-domain datasets.The out-of-domain datasets are TriviaQA, PopQA, 2WikiMultiHopQA, MuSiQue, and Bamboogle.
  • ZEROSEARCH outperforms methods that rely on real search engines, including Search-R1.The comparison identifies ZEROSEARCH as a potential alternative to real search engines for large-scale reinforcement learning.
  • ZEROSEARCH generalizes across base and instruction-tuned models, different parameter sizes, and different model families.Its performance also improves with larger models, indicating scalability within the evaluated settings.

5 Further Analysis

Further analyses show that ZEROSEARCH improves reward similarly to real-search training, benefits from larger fine-tuned simulation engines, and works across several RL algorithms. Its curriculum rollout improves performance over random rollout by progressively exposing policies to harder retrieval scenarios.

  • 5.1 Compare ZEROSEARCH with Real Search Engine: ZEROSEARCH and Search-R1 both steadily increase reward during training, while ZEROSEARCH eventually surpasses Search-R1 with less fluctuation.ZEROSEARCH initially lags behind but achieves a more pronounced reward improvement, which the authors associate with curriculum rollout.
  • 5.1 Compare ZEROSEARCH with Real Search Engine: ZEROSEARCH steadily improves reward for both base and instruction-tuned models.This result supports generalization across the two model types evaluated.
  • 5.2 Choice of Simulation LLMs: Fine-tuned simulation engines significantly outperform prompt-based simulation engines.The authors attribute the weaker prompt-based performance to a remaining distribution gap from real search-engine responses.
  • 5.2 Choice of Simulation LLMs: Performance improves consistently as simulation-model size increases, including better discrimination between useful and noisy documents.Larger simulation models thereby support more effective curriculum rollout during training.
  • 5.4 RL Algorithm Comparison: REINFORCE, GRPO, and PPO all enhance search ability within ZEROSEARCH, with REINFORCE achieving the best performance.The comparison uses Qwen-2.5-3B-Base as the policy model; repeated real-engine rollouts for REINFORCE and GRPO incur higher API costs.
  • 5.5 Curriculum Rollout Study: The easy-to-hard curriculum consistently outperforms random rollout across Qwen-2.5-3B-Base and LLaMA-3.2-3B-Base.The curriculum begins with better search results and progressively exposes the policy to more challenging scenarios.

6 Conclusion and Limitation Discussion

ZEROSEARCH enhances LLM search capabilities without real search-engine interaction during training and generalizes across model types and RL algorithms. Its main limitation is the need to deploy simulated-search LLMs on GPU servers, adding infrastructure costs.

  • ZEROSEARCH enhances LLM search capabilities without interacting with real search engines during training.
  • The framework generalizes across base and instruction-tuned LLMs of varying sizes and supports a wide range of RL algorithms.
  • Deploying the simulated-search LLM requires GPU servers, creating infrastructure costs despite eliminating commercial API expenses.

A Compare ZEROSEARCH with Real Search Engine

ZEROSEARCH produces smoother reward curves than a real search engine because it controls document difficulty during training. It also works with base and instruction-tuned models, although their learning trajectories differ initially.

  • Compare ZEROSEARCH with Real Search Engine: ZEROSEARCH consistently achieves smoother reward curves than a real search engine across the compared model sizes.Real-search document quality is uncontrollable during rollout, whereas ZEROSEARCH dynamically controls document difficulty.
  • Compare Base and Instruct LLMs: Instruction-tuned models initially achieve higher rewards because stronger instruction following helps them invoke the search engine more effectively.
  • Compare Base and Instruct LLMs: Base models steadily improve and reach performance levels comparable to instruction-tuned models.The results support ZEROSEARCH compatibility with both model types.
  • Compare Base and Instruct LLMs: Base models can acquire search capabilities through reinforcement learning without supervised fine-tuning as a warm-up.

C Effect of Document Token Loss Masking

Document-token loss masking stabilizes ZEROSEARCH training by excluding tokens not generated by the policy model. Removing the masking substantially lowers reward and model performance.

  • Effect of Document Token Loss Masking: Document tokens are excluded from the loss because they are not generated by the policy model and may introduce noise.
  • Effect of Document Token Loss Masking: Removing document-token loss masking substantially drops reward, indicating training instability and degraded learning effectiveness.Table 7 further reports a significant performance decline without loss masking.

D Cost Analysis

ZEROSEARCH is presented as a more cost-efficient alternative to commercial search APIs for RL training. Commercial API costs rise substantially with larger batches or more training steps, limiting scalability.

  • Cost Analysis: ZEROSEARCH offers a more cost-efficient alternative to commercial search APIs.The comparison estimates API costs using SerpAPI and GPU deployment costs using AWS.
  • Cost Analysis: Commercial search API costs increase substantially with larger batch sizes or more training steps, making RL training less scalable.
  • Cost Analysis: ZEROSEARCH requires GPU deployment, whose utilization is inconsistent between rollout and policy-update phases.

E Case Study

The case studies show that ZEROSEARCH elicits structured, multi-turn search behavior and supports controlled document-quality variation for reinforcement-learning rollouts.

  • Interaction Trajectory Study: The policy model consistently follows the required output format, despite format adherence not being explicitly reinforced by the reward design.
  • Interaction Trajectory Study: The policy model conducts multi-turn searches to reach a final answer, confirming that the method incentivizes search capabilities.
  • Simulated Document Study: Useful simulated documents consistently contain the correct answer, whereas noisy documents fail to do so.
  • Simulated Document Study: Precise control over document quality enables curriculum-based rollouts and contributes to stabilizing reinforcement-learning training.

F Implementation Details

Implementation uses separate GPU groups for simulation and RL training, lightweight SFT backbones of multiple sizes, and three reinforcement-learning algorithms.

  • Hardware and Training Setup: The simulation server runs on 4 H20 GPUs, while reinforcement-learning training uses another 4 H20 GPUs.
  • Simulation LLM: The simulation LLM is trained with lightweight SFT using Qwen-2.5-3B-Instruct, Qwen-2.5-7B-Instruct, and Qwen-2.5-14B-Instruct backbones.
  • Reinforcement Learning: ZEROSEARCH is trained with REINFORCE, GRPO, and PPO.
  • Framework Scope: The framework enhances LLM search capabilities without requiring interaction with real-world search engines and is intended to reduce RL training costs.
  • Reported Behavior: Table 9 reports that ZEROSEARCH policy models learn the correct output format and conduct multi-step searches to reach the final answer.
  • Simulated Documents: Table 10 compares useful and noisy outputs from 14B simulation LLMs, with correct answers highlighted in red.
Loading 2505.04588v3…