Source-linked AI summary
ReSearch: Learning to Reason with Search for LLMs via Reinforcement Learning
Mingyang Chen, Linzhuang Sun, Tianpeng Li, Haoze Sun, Yijie Zhou, Chenzheng Zhu, Haofen Wang, Jeff Z. Pan, Wen Zhang, Huajun Chen, Fan Yang, Zenan Zhou, Weipeng Chen
TL;DR
Complex multi-hop questions require LLMs to coordinate reasoning with multiple search operations, yet existing methods often depend on heuristics or costly reasoning-step labels. ReSearch integrates thinking, search queries, and retrieval results into reinforcement-learning rollouts without supervised reasoning data. Across multi-hop question answering benchmarks, it improves over baselines and generalizes beyond its single training set, while naturally eliciting reflection and self-correction.
Problem
Complex multi-hop RAG requires reasoning to steer multiple retrieval operations, while existing methods use labor-intensive heuristics or costly labeled reasoning steps.
Method
ReSearch trains LLMs with reinforcement learning from scratch, integrating text-based thinking, search queries, and retrieval results into the reasoning chain without supervised reasoning data.
Results
ReSearch achieves significant improvements over baselines across multiple multi-hop question answering benchmarks and generalizes after training on one specific dataset.
Takeaways & Limitations
ReSearch naturally elicits reflection and self-correction during reinforcement learning without predefined heuristics.
Takeaways & Limitations
The framework primarily targets concise, objectively verifiable answers and has not explored specialized or domain-specific knowledge bases beyond Wikipedia.
Abstract
from arXiv · showhide
Large Language Models (LLMs) have shown remarkable capabilities in reasoning, exemplified by the success of OpenAI-o1 and DeepSeek-R1. However, integrating reasoning with external search processes remains challenging, especially for complex multi-hop questions requiring multiple retrieval steps. We propose ReSearch, a novel framework that trains LLMs to Reason with Search via reinforcement learning without using any supervised data on reasoning steps. Our approach treats search operations as integral components of the reasoning chain, where when and how to perform searches is guided by text-based thinking, and search results subsequently influence further reasoning. We train ReSearch on Qwen2.5-7B(-Instruct) and Qwen2.5-32B(-Instruct) models and conduct extensive experiments. Despite being trained on only one dataset, our models demonstrate strong generalizability across various benchmarks. Analysis reveals that ReSearch naturally elicits advanced reasoning capabilities such as reflection and self-correction during the reinforcement learning process.
1 Introduction
ReSearch addresses the challenge of training LLMs to perform multi-step reasoning with external search without labeled reasoning traces. It integrates thinking, search, and retrieved information into a reinforcement-learning framework and reports generalizable benchmark improvements.
- Motivation: Complex multi-hop RAG requires reasoning to decide when and how to conduct multiple retrieval steps, but robust strategies remain challenging.Existing approaches often rely on labor-intensive, non-scalable prompts or heuristics, while labeling reasoning steps is costly and time-consuming.
- Framework: ReSearch trains LLMs to reason with search via reinforcement learning without supervised data on reasoning steps.The framework treats text-based thinking, search queries, and retrieval results as components of one reasoning chain.
- Framework: Text-based thinking steers when and how searches occur, while retrieved results influence subsequent text-based thinking.This makes search operations interactive with the model’s reasoning rather than a separate retrieval stage.
- Results: 8.9% to 22.4% absolute improvements over baselines were obtained across multi-hop question answering benchmarks.The models were trained from scratch on Qwen2.5-7B(-Instruct) and Qwen2.5-32B(-Instruct) using one specific training set, then evaluated on multiple benchmarks.
- Results: Training on different model scales and one dataset produced models with significant generalizability across multi-hop question answering benchmarks.The reported experiments also indicate potential for more realistic scenarios.
- Analysis: ReSearch progressively elicits reflection and self-correction during reinforcement learning without predefined heuristics.The analysis attributes these capabilities to the training process rather than supervised reasoning-step demonstrations.
2 Method
ReSearch uses reinforcement learning to optimize reasoning-with-search rollouts, in which text generation, search queries, and retrieved results interact iteratively. Its training uses GRPO with masked retrieval tokens and rule-based answer and format rewards.
- Reinforcement Learning: ReSearch samples multiple reasoning-with-search rollouts and optimizes the language-model policy toward rollouts receiving higher rewards.The framework is trained with reinforcement learning rather than labeled reasoning traces.
- Reinforcement Learning: GRPO estimates a baseline from a group of rollouts instead of training a separate critic model as in PPO.The objective uses an existing policy, a reference policy, normalized rollout advantages, clipping, and a KL penalty.
- Rollout with Search: Each rollout iteratively alternates text-based thinking, search queries, and retrieved results until an end-of-sentence token is generated.Encountering the closing search tag transfers the query to the search environment, and the returned results are appended to the next model input.
- Rollout with Search: Retrieval-result tokens are masked during loss calculation because they come from the search environment rather than the training policy.Only tokens from text-based thinking and search queries contribute to the loss.
3 Experiments
ReSearch is evaluated on multi-hop question answering using several Qwen2.5 model sizes, with training restricted to MuSiQue and evaluation spanning multiple benchmarks. It improves over baselines, generalizes across question types, and progressively learns more search-based reasoning during reinforcement learning.
- Experimental Setup: ReSearch is trained on Qwen2.5 models with 7B and 32B parameters, using base and instruction-tuned variants.Training uses reinforcement learning from scratch without supervised fine-tuning.
- Experimental Setup: The evaluation covers HotpotQA, 2WikiMultiHopQA, MuSiQue, and Bamboogle multi-hop question answering benchmarks.The tasks require multi-step reasoning and multiple information retrieval, with Wikipedia-based retrieval used as the environment.
- Main Results: ReSearch improves over the best baseline by 15.81% EM and 17.56% LJ for 7B models, and by 14.82% EM and 15.46% LJ for 32B models.The authors report improvements across all evaluated benchmarks.
- Main Results: Instruction-tuned models outperform corresponding base models after ReSearch training across benchmarks and model sizes.Both base and instruction-tuned models are trained from scratch with reinforcement learning, without supervised fine-tuning.
- Main Results: Training on MuSiQue generalizes to other benchmarks with different question types and structures.The authors characterize the learned search-and-reasoning ability as independent of specific knowledge or multi-hop patterns.
- Further Analysis: Response length generally increases during training while the average number of search operations grows consistently.Response length excludes retrieval-result tokens and is interpreted as reasoning cost; the increasing search count indicates progressively more iterative retrieval for complex questions.
- Further Analysis: Training reward rises sharply during the first 20 steps and then increases gradually, with higher cold-start reward for instruction-tuned models.For 7B models, rewards converge to a similar level, whereas 32B instruction-tuned models achieve higher training reward than base models.
- Case Study: A case study shows the model decomposing a complex question, using search iteratively, and correcting an ineffective query through self-elicited reflection.The model explicitly recognizes a failed search and generates a more effective query without reflection being explicitly designed or trained.
4 Related Work
Related work develops reinforcement-learning methods for LLM reasoning and retrieval-augmented generation, including iterative approaches for complex multi-hop questions.
- Reinforcement learning has become important for aligning LLMs with human values and enhancing their reasoning capabilities.
- Retrieval-augmented generation research covers retriever optimization, query refinement, and self-reflection mechanisms.
- Iterative RAG methods alternate retrieval-enhanced generation with generation-enhanced retrieval for more complex multi-hop questions.
5 Conclusion
ReSearch integrates search into reinforcement-learned reasoning without supervised reasoning-step data and improves performance across multi-hop benchmarks. Training analysis also finds naturally emerging reflection and self-correction.
- ReSearch trains LLMs to reason with search via reinforcement learning without supervised data on reasoning steps.
- Search operations form part of the reasoning chain: text-based thinking guides searches, and retrieved results influence subsequent reasoning.
- ReSearch significantly improves results over baseline methods across multiple multi-hop question-answering benchmarks.
- Training analysis indicates that ReSearch naturally elicits reflection and self-correction without predefined heuristics.
A Prompt for LLM-as-a-Judge
The prompt evaluates whether a predicted answer semantically matches at least one ground-truth answer. It requests a rationale followed by a binary correctness judgment in JSON format.
- The evaluation receives a question, a ground-truth answer list, and a predicted answer.
- The judge must first provide a rationale and then label the prediction correct or incorrect.
- Exact wording is unnecessary if the prediction is semantically equivalent to at least one ground-truth answer.
- The requested output is a JSON object containing rationale and judgement fields.
B Implementation Details
The implementation-details section points to Table 4 for important ReSearch training parameters and labels that table as the implementation-details summary.
- Training uses 8 × 8 Nvidia H800 GPUs with full parameter optimization and gradient checkpointing.
- Table 4 presents important implementation details for ReSearch.
C Limitation
ReSearch is evaluated primarily on concise-answer settings with objectively verifiable outcomes, and its use of Wikipedia limits evidence about specialized knowledge bases.
- The framework primarily targets questions whose answers are relatively concise and objectively verifiable using simple metrics such as F1 score.
- Its effectiveness on longer, more nuanced responses remains uncertain because those tasks require more sophisticated reward modeling.
- The study uses Wikipedia as the primary knowledge base for retrieval-augmented generation operations.
- The framework has not yet been evaluated with specialized or domain-specific knowledge bases.
D Broader Impact
ReSearch is presented as a framework that could improve the accuracy and reliability of AI systems in knowledge-intensive applications by combining reasoning with external information verification.
- The framework could enhance AI-system accuracy and reliability in knowledge-intensive tasks.
- Potential beneficiaries include education, scientific research, and fact-checking.
- Breaking complex questions into manageable steps while verifying information through external sources could help reduce misinformation.