PaperlayerView exact revision on arXiv

Source-linked AI summary

Search-R1: Training LLMs to Reason and Leverage Search Engines with Reinforcement Learning

Bowen Jin, Hansi Zeng, Zhenrui Yue, Jinsung Yoon, Sercan Arik, Dong Wang, Hamed Zamani, Jiawei Han

arXiv:2503.09516v5cs.CLcs.AIcs.IR

TL;DR

LLMs need better ways to reason with current external information, while prompting them to use search may not teach effective search interaction. Search-R1 trains interleaved reasoning and multi-turn search with reinforcement learning, achieving average relative improvements of 41% and 20% over RAG baselines under matched settings.

  • Problem

    LLMs struggle with complex reasoning and effective retrieval of up-to-date external information, motivating methods that improve both capabilities.

  • Method

    Search-R1 models search as part of the RL environment, optimizing reasoning trajectories with interleaved multi-turn retrieval, retrieved-token masking, and outcome-based rewards.

  • Results

    41% and 20% average relative improvements over RAG baselines are reported for two LLMs under the same experimental setup.

  • Takeaways & Limitations

    Search-R1 demonstrates that a simple outcome-based reward can support search-and-reasoning scenarios, while experiments provide insights into RL methods, LLM choices, and response length.

  • Takeaways & Limitations

    Comparisons are controlled using the same retriever, retrieved-document count, knowledge corpus, training data, and pretrained LLMs.

Abstract

from arXiv · show

Efficiently acquiring external knowledge and up-to-date information is essential for effective reasoning and text generation in large language models (LLMs). Prompting advanced LLMs with reasoning capabilities to use search engines during inference is often suboptimal, as the LLM might not fully possess the capability on how to interact optimally with the search engine. This paper introduces Search-R1, an extension of reinforcement learning (RL) for reasoning frameworks where the LLM learns to autonomously generate (multiple) search queries during step-by-step reasoning with real-time retrieval. Search-R1 optimizes LLM reasoning trajectories with multi-turn search interactions, leveraging retrieved token masking for stable RL training and a simple outcome-based reward function. Experiments on seven question-answering datasets show that Search-R1 improves performance by 41% (Qwen2.5-7B) and 20% (Qwen2.5-3B) over various RAG baselines under the same setting. This paper further provides empirical insights into RL optimization methods, LLM choices, and response length dynamics in retrieval-augmented reasoning. The code and model checkpoints are available at https://github.com/PeterGriffinJin/Search-R1.

1 Introduction

LLMs struggle with complex reasoning and effective use of up-to-date external information. SEARCH-R1 addresses these challenges by training interleaved search and reasoning through reinforcement learning.

  • LLMs face challenges in complex reasoning and retrieving up-to-date information from external sources.
  • Existing search-integration approaches use retrieval-augmented generation or treat search engines as tools.RAG incorporates retrieved passages into the LLM context, while tool-use approaches enable more adaptive interaction.
  • Annotated trajectories and search non-differentiability make end-to-end optimization difficult for adaptive search approaches.
  • SEARCH-R1 models the search engine as part of the environment and interleaves LLM token generation with retrieval during RL rollouts.The framework is compatible with PPO and GRPO and uses retrieved token masking for stable optimization.
  • SEARCH-R1 supports multi-turn retrieval and reasoning through explicit search, information, thinking, and answer token markers.This structure enables iterative decision-making with multiple search calls during reasoning.
  • SEARCH-R1 uses an outcome-based reward function and reports average relative improvements of 41% and 20% over RAG baselines for two LLMs.The experiments use the same retrieval model, training data, and pretrained LLMs across comparisons.

2 Related Works

Related work integrates search with LLMs through retrieval augmentation, tool use, and reinforcement-learning-based tuning. These approaches differ in supervision, optimization, and efficiency trade-offs.

  • Search engines are integrated with LLMs to provide external information for models that lack domain-specific knowledge and are prone to hallucinations.
  • RAG and tool-use methods are the two primary approaches for integrating search engines with LLMs.
  • RL trains agents through sequential interaction with an environment and reward feedback to maximize cumulative reward.
  • PPO-based RLHF can be difficult to implement because PPO requires multiple rounds of LLM optimization.
  • DPO and SimPO simplify RL-based tuning but suffer from off-policy issues and do not consistently match pure RL performance.

3 Search-R1

SEARCH-R1 extends PPO and GRPO to optimize LLM reasoning trajectories that interleave generated text with multi-turn search-engine retrieval. It combines retrieved-token loss masking, structured rollout interactions, and final-outcome rewards for search-augmented reasoning.

  • 3.1 Reinforcement Learning with a Search Engine: SEARCH-R1 models the search engine as part of the environment, enabling RL trajectories that interleave LLM token generation with retrieval.The framework is compatible with PPO and GRPO and applies retrieved-token masking during optimization.
  • 3.1 Reinforcement Learning with a Search Engine: Retrieved-token loss masking updates only LLM-generated tokens, stabilizing training while preserving search-augmented generation.The masking operation assigns one to generated tokens and zero to retrieved tokens.
  • 3.1 Reinforcement Learning with a Search Engine: PPO optimizes clipped token-level policy objectives using advantage estimates from future rewards and a learned value function.The search setting applies the objective only to LLM-generated tokens rather than retrieved tokens.
  • 3.1 Reinforcement Learning with a Search Engine: GRPO uses the average reward of multiple sampled responses as a baseline, avoiding an additional learned value-function approximation.Its advantages are computed from relative rewards within each sampled group, with KL divergence added directly to the loss.
  • 3.2 Text Generation with Interleaved Multi-Turn Search Engine Call: During rollout, the LLM alternates between reasoning and search calls, enclosing queries in <search> and </search> tokens before retrieval results are inserted.The process repeats until an answer or end-of-sequence condition is reached, subject to a maximum action budget.
  • 3.3 Training Template: The training template structures each iteration as reasoning, a search-engine call, and an answer while avoiding content-specific problem-solving constraints.This design leaves the model’s learning dynamics during RL observable and unbiased.
  • 3.4 Reward Modeling: SEARCH-R1 uses only final outcome rewards that assess response correctness, such as exact string matching in factual reasoning tasks.It omits format rewards and neural reward models, citing existing structural adherence and added sensitivity, cost, and complexity.

4 Main Results

SEARCH-R1 is evaluated across seven general and multi-hop question-answering datasets against retrieval, inference, and fine-tuning baselines under matched experimental conditions. It consistently outperforms strong baselines, with gains across both in-distribution and out-of-distribution evaluation.

  • Evaluation setup: SEARCH-R1 is evaluated on seven datasets spanning general and multi-hop question answering.The benchmarks are NQ, TriviaQA, PopQA, HotpotQA, 2WikiMultiHopQA, Musique, and Bamboogle.
  • Evaluation setup: The comparison covers direct inference, CoT, RAG, IRCoT, Search-o1, supervised fine-tuning, and RL-based fine-tuning without search.These baselines cover zero-shot retrieval and learned retrieval settings.
  • Experimental conditions: The experiments use Qwen-2.5-3B and Qwen-2.5-7B variants, a 2018 Wikipedia dump, E5 retrieval, and three retrieved passages.Exact Match is used for evaluation, with training based on merged NQ and HotpotQA training sets.
  • Main findings: SEARCH-R1 achieves 24% and 20% average relative improvement with Qwen2.5-7B and Qwen2.5-3B, respectively.The gains hold on both in-distribution NQ and HotpotQA and out-of-distribution datasets.
  • Main findings: SEARCH-R1 consistently outperforms strong baselines and surpasses RL-based reasoning without retrieval.The reported gains include both in-distribution and out-of-distribution evaluation.

5 Analysis

The analysis compares RL algorithms, model variants, response lengths, search behavior, and retrieved-token masking. It shows trade-offs between convergence speed and stability, while masking improves training outcomes.

  • Different RL methods: PPO vs. GRPO: GRPO converges faster than PPO, but PPO provides greater training stability during extended training.The paper attributes PPO’s slower start to its critic warm-up and reports reward collapse for GRPO after many steps.
  • Response length and search behavior: The model learns to make more valid search calls as training proceeds.This trend is reported as part of the search-behavior dynamics study.
  • Base vs. Instruct LLMs: Instruction-tuned models converge faster and start with higher performance, while base and instruction-tuned models reach highly similar final rewards.The comparison uses Qwen2.5-3B and Qwen2.5-7B variants.
  • Response length and search behavior: Response length decreases early, then increases and stabilizes later alongside the training reward trajectory.During the first 100 steps, the base model removes excessive filler words while adapting to task requirements.
  • Retrieved-token loss masking: Retrieved-token loss masking consistently improves performance relative to training without masking.The paper reports that masking mitigates unintended optimization effects and supports more stable training.

6 Conclusions

SEARCH-R1 extends reinforcement learning to interleave LLM reasoning with real-time search, enabling autonomous query generation and strategic use of retrieved information. Experiments across seven datasets show improved performance on reasoning tasks requiring external knowledge.

  • Conclusion: SEARCH-R1 optimizes LLM rollouts through reinforcement learning while interleaving self-reasoning with real-time search interactions.The framework treats search as part of the environment rather than relying on extensive prompting or large-scale supervised tool-use trajectories.
  • Conclusion: The framework addresses the non-differentiability of search by optimizing sampled trajectories with reinforcement learning.Existing search-integrated approaches are difficult to scale because they rely on annotated trajectories and cannot use end-to-end gradient descent through search.
  • Conclusion: SEARCH-R1’s reported scope is complex reasoning tasks requiring real-time external knowledge, evaluated through extensive experiments on seven datasets.The conclusion presents broader search strategies, dynamic retrieval, additional tools, and multimodal reasoning as future directions.
  • RL formulation: The extended RL objective models trajectories as interleaved reasoning steps and retrieved content conditioned on the prompt and retrieval-augmented context.Its KL term keeps the learned policy aligned with a reference model over the joint response distribution.

B.2 Experimental Settings

The experiments use controlled retrieval and training settings across Qwen2.5 base and instruction-tuned models, with evaluation on seven datasets. PPO and GRPO runs share the same main hardware, batch, and sequence configurations.

  • Models and retrieval: Experiments use Qwen-2.5-3B and Qwen-2.5-7B in base and instruction-tuned variants.The retrieval source is the 2018 Wikipedia dump, using E5 as the retriever and three passages per retrieval-based method.
  • Data and evaluation: Training merges the NQ and HotpotQA training sets, while evaluation uses test or validation sets from seven datasets with Exact Match.Inference baselines use instruction-tuned models because base models fail to follow instructions; RL tuning evaluates both model types.
  • PPO settings: PPO training uses policy and value learning rates of 1e-6 and 1e-5, respectively, for 500 steps with GAE parameters λ = 1 and γ = 1.The policy and value models use warm-up ratios of 0.285 and 0.015.
  • Hardware and batching: Training runs on 8 H100 GPUs with total batch size 512, mini-batch size 256, micro-batch size 64, and maximum sequence length 4,096 tokens.The maximum response length and retrieved-content length are both 500 tokens.
  • Shared settings: Both methods use vLLM rollouts with temperature 1.0, top-p 1.0, KL coefficient β = 0.001, and clip ratio ϵ = 0.2.Gradient checkpointing and FSDP CPU offloading are also used.
  • GRPO settings: GRPO samples five responses per prompt and uses the same 8×H100 setup, batch sizes, and sequence-length configurations as PPO.GRPO is trained for 500 steps with a policy learning rate of 1e-6 and a warm-up ratio of 0.285.
  • Checkpointing and rewards: Checkpoints are saved every 100 steps, with the latest stable checkpoint used when training diverges; the maximum action budget is 4 and retrieval defaults to three passages.Outcome rewards are computed using Exact Match, and PPO is the default RL algorithm unless otherwise stated.

C Main Results on 14B LLM

SEARCH-R1 consistently outperforms baseline methods on Qwen2.5-14B models, and larger model size yields further performance gains. Retrieved-token loss masking also improves performance and stabilizes optimization.

  • Main Results on 14B LLM: SEARCH-R1 consistently outperforms all evaluated baseline methods on Qwen2.5-14B models.The experiments use Qwen2.5-14B models and report results in Table 5.
  • Main Results on 14B LLM: Larger model size produces consistent performance gains with SEARCH-R1.The reported results highlight benefits from scaling the LLM size within this approach.
  • Retrieved Token Loss Masking Study: Retrieved-token loss masking leads to more stable optimization and improved model performance.The comparison evaluates Qwen2.5-3B/7B-base models with and without masking.
  • Retrieved Token Loss Masking Study: SEARCH-R1 trained with retrieved-token loss masking consistently outperforms its unmasked counterpart.Quantitative results in Table 6 confirm the advantage under PPO training.

E Base vs. Instruct LLMs

Instruction-tuned models begin with higher performance and converge faster than base models, but both reach similar final performance. Across RL methods, GRPO converges faster while PPO is more stable, with comparable final rewards.

  • Base vs. Instruct LLMs: Instruction-tuned models converge faster and start from higher performance than base models.This pattern is reported across Qwen2.5-3B and Qwen2.5-7B variants.
  • Base vs. Instruct LLMs: Base and instruction-tuned models reach similar final performance after training.The early advantage of instruction tuning does not persist as a final-performance difference.
  • RL Method Comparison: GRPO converges faster than PPO across the evaluated settings.PPO requires critic-model warm-up steps before effective training begins.
  • RL Method Comparison: PPO provides more stable training, whereas GRPO can experience reward collapse during extended training.The comparison is reported across four LLMs and the two RL algorithms.
  • RL Method Comparison: PPO and GRPO achieve comparable final reward performance.The methods trade off convergence speed and stability while reaching similar final rewards.

G Number of Retrieved Passages Study in SEARCH-R1 Training

The number of retrieved passages affects convergence and stability during SEARCH-R1 training. top-k = 5 converges fastest initially but becomes unstable, while top-k = 3 achieves the best later reward and overall evaluation performance.

  • Retrieved Passage Count: top-k = 1, 3, and 5 exhibit similar overall training trajectories.The study evaluates these settings during SEARCH-R1 training.
  • Retrieved Passage Count: top-k = 5 reaches the highest training reward within the first 200 steps but later becomes unstable.Its reward gradually declines as training progresses.
  • Retrieved Passage Count: top-k = 3 ultimately achieves the highest reward after 500 steps.top-k = 1 and 3 show more consistent improvements throughout training.
  • Retrieved Passage Count: top-k = 3 yields the best overall evaluation performance at step 500.The authors attribute the trade-off to retrieval recall for top-k = 1 and noisy passages for top-k = 5.

H Group Size Study in SEARCH-R1 (GRPO) Training

Larger GRPO group sizes accelerate convergence but increase collapse risk, while smaller sizes improve stability and generalization. Case studies show that interleaved retrieval supports iterative reasoning and self-verification, but irrelevant or insufficient evidence can mislead the model.

  • Group Size Effects: Larger group sizes generally lead to faster convergence but increase the risk of training collapse.The study compares group sizes 1, 3, and 5, with size 1 reducing GRPO to REINFORCE.
  • Group Size Effects: Smaller group sizes enable more stable training and better generalization on unseen tasks.Larger groups can achieve higher training rewards while trading away stability.
  • Case Studies: SEARCH-R1 uses multi-turn retrieval to iteratively refine reasoning with retrieved passages.This contrasts with RL without search, which relies solely on internal model knowledge.
  • Case Studies: After sufficient information is gathered, SEARCH-R1 may perform an additional retrieval step to self-verify its conclusion.The case study observes this behavior after the second retrieval round.
  • Case Studies: Irrelevant or insufficient retrieved information can lead SEARCH-R1 to incorrect answers.The case-study summaries report failures from misleading passages and inadequate evidence.
Loading 2503.09516v5…