Source-linked AI summary
R1-Searcher: Incentivizing the Search Capability in LLMs via Reinforcement Learning
Huatong Song, Jinhao Jiang, Yingqian Min, Jie Chen, Zhipeng Chen, Wayne Xin Zhao, Lei Fang, Ji-Rong Wen
TL;DR
Large reasoning models may struggle when internal knowledge is insufficient for knowledge-intensive or time-sensitive questions. R1-Searcher trains autonomous external retrieval through two-stage outcome-based RL without distillation or cold-start fine-tuning, and reports stronger performance than existing RAG methods across multi-hop QA evaluations.
Problem
Internal-knowledge reliance limits LLM reasoning on knowledge-intensive and time-sensitive questions, motivating access to external information during reasoning.
Method
R1-Searcher uses two-stage outcome-based RL to learn retrieval invocation and effective use of an external search system without distillation or cold-start fine-tuning.
Results
R1-Searcher consistently and significantly surpasses existing RAG methods across four multi-hop datasets, with reported gains of up to 48.22% on HotpotQA and 21.72% on 2Wiki over ReARTeR.
Takeaways & Limitations
The method generalizes from in-domain training to out-of-domain datasets and online search, and is applicable to both base and instruction-tuned models.
Takeaways & Limitations
The study currently uses simple data mixing and plans to evaluate larger models beyond the current 7B configuration.
Abstract
from arXiv · showhide
Existing Large Reasoning Models (LRMs) have shown the potential of reinforcement learning (RL) to enhance the complex reasoning capabilities of Large Language Models~(LLMs). While they achieve remarkable performance on challenging tasks such as mathematics and coding, they often rely on their internal knowledge to solve problems, which can be inadequate for time-sensitive or knowledge-intensive questions, leading to inaccuracies and hallucinations. To address this, we propose \textbf{R1-Searcher}, a novel two-stage outcome-based RL approach designed to enhance the search capabilities of LLMs. This method allows LLMs to autonomously invoke external search systems to access additional knowledge during the reasoning process. Our framework relies exclusively on RL, without requiring process rewards or distillation for a cold start. % effectively generalizing to out-of-domain datasets and supporting both Base and Instruct models. Our experiments demonstrate that our method significantly outperforms previous strong RAG methods, even when compared to the closed-source GPT-4o-mini.
1 Introduction
R1-Searcher addresses the limits of internal-knowledge reasoning by training LLMs to autonomously use external retrieval during reasoning. Its two-stage, outcome-based RL framework improves performance across multi-hop QA benchmarks and supports generalization beyond training settings.
- Motivation: Internal-knowledge reliance can make reasoning models inaccurate on knowledge-intensive, private-information, and time-sensitive questions.The paper motivates access to external information during reasoning to address these limitations.
- Method: R1-Searcher uses a two-stage, outcome-based RL framework to enable autonomous retrieval during reasoning.The first stage rewards retrieval invocation, while the second rewards correctly using retrieved information to answer questions.
- Results: 48.22% improvement on HotpotQA and 21.72% on 2Wiki are reported over ReARTeR with GPT-4o-mini when using Qwen-2.5-7B-Base.The comparison is reported as a result of the experiments described in the introduction.
- Results: 11.4% improvement on Bamboogle is reported over Search-o1 with 32B parameters, using online search unseen during training.Bamboogle is used to assess generalization to an online-search setting.
- Results: Experiments on four multi-hop datasets show that R1-Searcher consistently and significantly surpasses existing RAG methods, including the closed-source GPT-4o-mini.The paper evaluates the method using multiple LLM backbones and benchmark settings.
- Method: The framework relies exclusively on RL, without distillation or cold-start instruction fine-tuning, and applies to both base and instruction-tuned models.The approach is also reported to generalize to out-of-domain datasets and online search scenarios.
2 Method
R1-Searcher trains LLMs to use an external retrieval system through two-stage outcome-based RL, progressing from search invocation to answer-focused reasoning. The method combines difficulty-aware data selection, staged rewards, retrieval-integrated rollouts, and masking of retrieved documents during loss calculation.
- 2.1 Data Selection: Data are selected from HotpotQA and 2WikiMultiHopQA and categorized by the number of rollouts needed for correct answers.The categories are easy, medium, and difficult, based on fewer than 10, 10–20, and more than 20 rollouts, respectively.
- 2.2 Training Method: The two-stage method first trains retrieval use, then trains the model to incorporate search into reasoning for accurate question solving.Stage-1 focuses on utilizing the external retrieval system, while Stage-2 combines search with answer solving.
- 2.2.1 Reward Design: Stage-1 uses retrieval and format rewards to encourage search without rewarding answer correctness.The retrieval reward depends on the number of retrieval invocations, while the format reward checks the required output and query structure.
- 2.2.1 Reward Design: Stage-2 removes the retrieval reward and uses answer and format rewards, with answer reward calculated from the F1 score between predicted and reference answers.The final Stage-2 reward is the sum of answer and format rewards; PN, RN, and IN denote predicted-answer length, reference-answer length, and their word-count intersection.
- 2.2.2 RAG-based Rollout: During RAG rollouts, query tags pause generation for retrieval, retrieved documents are inserted as environment observations, and document tokens are masked during loss calculation.The model continues reasoning using retrieved documents, while masking prevents it from being trained to generate those documents.
3 Experiment
The experiments evaluate R1-Searcher across four multi-hop QA benchmarks, using in-domain and out-of-domain settings, multiple backbones, and several retrieval baselines. R1-Searcher achieves strong improvements over existing methods, supports training from a base model without cold start, and generalizes beyond its training datasets.
- Datasets and evaluation: Evaluation covers HotpotQA, 2WikiMultiHopQA, Musique, and Bamboogle, with the first two treated as in-domain and the latter two as out-of-domain benchmarks.Parts of the HotpotQA and 2WikiMultiHopQA training sets are used for reinforcement learning, while Musique and Bamboogle assess generalization.
- Datasets and evaluation: Performance is measured with Cover Exact Match (ACC_R) and LLM-as-Judge (ACC_L), which assess answer inclusion and semantic correctness, respectively.ACC_L uses GPT-4o-mini to judge whether predictions fully align with the golden answer's meaning and key information.
- Baselines and setup: The comparison includes naive generation, standard RAG, branching, summarization-based, adaptive retrieval, RAG-CoT, test-time scaling, and reasoning-model baselines.The baselines span methods that alter prompting, document compression, retrieval adaptivity, reasoning paths, or test-time search.
- Main results: R1-Searcher improves over ReARTeR by 48.2% on HotpotQA, 21.7% on 2WikiMultiHopQA, and 4.0% on Bamboogle under LLM-as-Judge.Using the same Llama-3.1-8B-Instruct backbone, the method is reported to facilitate accurate retrieval invocations with lower overhead than MCTS-based exploration.
- Main results: Qwen-2.5-7B-Base training from scratch achieves the best performance on most in-domain and out-of-domain datasets, surpassing GPT-4o-mini without a cold start.The result is presented as evidence that the two-stage RL method can guide learning directly from a powerful base model.
- Generalization: With only 8148 RL training samples, R1-Searcher performs well on Musique and Bamboogle and generalizes to online search.The paper attributes this to learning retrieval and reasoning integration through exploration during RL training.
4 Further Analysis
Further analysis examines training algorithms, reward metrics, format constraints, data difficulty and diversity, and generalization to online search. Results indicate that RL and carefully designed training signals improve retrieval behavior, performance, and generalization.
- Basic Training Methods: GRPO produces longer responses and more frequent retrievals than Reinforce++ despite similar training rewards.The comparison uses Llama-3.1-8B-Instruct trained on HotpotQA and 2Wiki.
- Basic Training Methods: RL outperforms SFT on both in-domain and out-of-domain test sets, indicating stronger retrieval capability and generalization.The comparison includes Qwen-2.5-7B-Base and Llama-3.1-8B-Instruct.
- Reward Design: F1-based answer rewards yield longer responses and better final results than CEM- and EM-based rewards.F1 achieves up to a 52.6% average performance improvement over the EM-based reward, while EM produces shorter responses and poorer test performance.
- Reward Design: Format rewards are iteratively constrained to prevent abnormal outputs and reward hacking, including fabricated external documents and format violations.The observed issues include generating document tags without query tags, nonsensical outputs under Base-model training with KL set to 0, and retrieval bypass when Llama omits Stage-1 training.
- Training Data: Training on more difficult data increases generation length and retrieval frequency and improves evaluation performance by 3.4% average CEM over easier-data training.The comparison uses datasets with and without questions requiring more than 20 rollouts.
- Training Data: Mixed HotpotQA and 2Wiki training increases retrievals, response length, and test performance, with up to 10.9% average CEM improvement.Training solely on 2Wiki gives higher rewards but lower average performance across three datasets, possibly because of lower diversity and overfitting.
- Generalization to Online Search: With online search unseen during RL training, R1-Searcher improves performance by 18.2% over local retrieval and 11.4% over Search-o1.The online-search evaluation uses Bamboogle, Google API retrieval, and GPT-4o-mini summaries of retrieved pages.
5 Case Study
The case studies compare RL with SFT and compare RL-trained Qwen and Llama models on Bamboogle. RL improves retrieval behavior and question decomposition, with model-specific differences in multi-step reasoning.
- RL invokes external retrieval when the model lacks an answer, whereas SFT may rely on incorrect internal knowledge.The comparison is reported in the Bamboogle case study contrasting Qwen-2.5-7B-Instruct models trained with RL and SFT.
- After RL, Qwen-2.5-7B-Base more effectively decomposes complex questions than Llama-3.1-8B-Instruct.The reported mechanism involves breaking multi-step problems into subproblems and sequentially retrieving relevant information.
- Sequential retrieval of relevant information increases the likelihood of finding pertinent documents and enhancing accuracy.
6 Conclusion
R1-Searcher integrates retrieval-augmented generation with two-stage outcome-supervised reinforcement learning so models learn to invoke external search during reasoning. It uses only RL, generalizes to unseen datasets and online search, supports base and instruction-tuned models, and outperforms traditional RAG and other reasoning approaches across multiple datasets.
- R1-Searcher integrates RAG with a two-stage outcome-supervised RL approach for invoking external search during reasoning.A designed reward mechanism supports autonomous exploration without instruction fine-tuning for cold start.
- The framework uses only RL and supports autonomous learning through exploration without instruction fine-tuning for cold start.
- R1-Searcher generalizes from in-domain training datasets to out-of-domain tests, switches to online search for up-to-date information, and supports base and instruction-tuned models.
- Across multiple datasets, R1-Searcher outperforms traditional RAG methods and other reasoning approaches.
7 Future Work
Future work targets training methodology through more sophisticated data curricula and larger models. The authors observed that training-data distribution and difficulty influence learning, while current experiments use simple data mixing and a 7B configuration.
- More sophisticated data curricula are planned because training-data distribution and difficulty significantly influence learning.The current approach uses only simple data mixing, which may be improved with a more structured curriculum.
- Current future-work priorities are refining data curricula and evaluating larger model scales.
- The authors plan to scale beyond the current 7B configuration to investigate larger models such as 32B.This is intended to better assess the effectiveness of the approach.