R-Search: Empowering LLM Reasoning with Search via Multi-Reward Reinforcement Learning

• Complex tasks need external knowledge integrated into reasoning, but one-time search or reasoning alone can fall short and produce suboptimal trajectories.Source: §1 · pp. 1–2
• R-Search lets the LLM interleave multi-step reasoning and token-level retrieval, distill retrieved documents into evidence, and optimize trajectories with answer, evidence, and format rewards.Source: §1 · p. 2
• Across seven datasets, R-Search outperforms vanilla and advanced RAG baselines by up to 32.2%.Source: §1 · p. 2
• RSTool makes high-quality reasoning evidence transferable, enabling costly reasoning–search interactions to be offloaded to local models.Source: §1 · p. 2

2.1 Retrieval-Augmented Generation (RAG)

• Existing multi-turn RAG approaches (Jeong et al., 2024; Jiang et al., 2023; Cheng et al., 2024; Trivedi et al., 2023; Shao et al., 2023; Chan et al., 2024; Asai et al., 2024) remain prompt-dependent and limit interaction depth.Source: §2.1 · p. 2

2.2 Reinforcement Learning for RAG

• Most RL approaches emphasize internal reasoning; Search-R1 applies PPO and GRPO to autonomous search, but reward design and reasoning–search interaction remain challenges (Jin et al., 2025).Source: §2.2 · pp. 2–3

3.1 RL on RAG Paradigms

• R-Search models advanced RAG as a partially observable decision process and optimizes interleaved reasoning-search trajectories with a policy maximizing expected reward.At each step, the model may search, continue reasoning, or generate the final answer.Source: §3.1 · p. 3

3.2.1 Rollout: evidence-augmented iterative reasoning and search

• The rollout prompts chain-of-thought, permits search during reasoning, appends retrieved documents, then derives global factual evidence before generating the answer.The question is appended at the end of the system template.Source: §3.2.1 · p. 4, Table 1 · p. 4

3.2.2 Multi-reward modeling

• Multi-stage rewards include answer F1 against gold and an evidence reward from a frozen different-family model independently answering from shared evidence.The cross-family answer uses the same evidence as the policy model, and its answer reward is used to reflect evidence quality.Source: §3.2.2 · pp. 4–5
• The format reward requires evidence and the final answer to appear in exactly one designated box each, alongside answer and evidence rewards.The reward definitions identify indicators for retrieval, answer formatting, and evidence formatting.Source: §3.2.2 · pp. 4–5

3.2.3 Mask and non-mask strategy

• Training masks retrieved-document tokens from policy loss but leaves generated evidence trainable, aiming to reduce gradient noise while improving evidence integration.The stated rationale is to avoid noisy or misaligned gradients on retrieved text and to train the model to select, understand, and integrate external knowledge.Source: §3.2.3 · p. 5

4.1 Datasets & Metrics, and & Search Tools

• Setup covers four multi-hop and three single-hop QA datasets, evaluates F1 and stricter Exact Match, and uses E5 dense retrieval with dataset-specific Wikipedia corpora.Single-hop datasets and Bamboogle use the 2018 Wikipedia dump; the other multi-hop datasets use aligned Wikipedia corpora.Source: §4.1 · p. 5

4.2 Baselines and Backbone LLMs

• Comparisons include naive generation, vanilla, branching, multi-step, adaptive RAG, and Search-R1; RL-trained backbones are Qwen-2.5-3B/7B, with Llama-3.2-3B for cross-family evidence rewards.Search-R1 training parameters and datasets are aligned with R-Search for a fair comparison.Source: §4.2 · pp. 5–6

4.3 Implementation Details

• Evaluation fixes retrieval top-k at 5; RL training uses only 2WikiMQA training data on 8×A100 80GB GPUs, with at most 195 steps and batch size 256.Training retrieval top-k is 3; rollouts sample five responses per prompt.Source: §4.3 · p. 6

5.1 Overall Performance

• Up to 37.2% improvement over Vanilla RAG and LLM w/o search on complex multi-hop QA, and up to 45.2% over branching RAG on long-chain tasks.R-Search also surpasses Search-R1 by 5.6% on MuSiQue.Source: §5.1 · p. 6, Table 2 · p. 7
• The larger R-Search model generally outperforms the smaller one, while the smaller model still exceeds other RAG methods in most cases, especially on multi-hop QA.Source: §5.1 · p. 7

5.3 Analysis

• As training progresses, models trigger more retrieval steps, enabling more rounds of reasoning–search interaction and deeper exploration of external knowledge.Source: Figure 3 · p. 8, §5.3 · p. 8

5.2 Ablation Study

• Removing evidence and its associated rewards causes a clear performance drop, especially on multi-hop datasets; evidence helps select relevant knowledge and prioritize reliable, complete intermediate facts.The evidence-free variant removes evidence from rollout and disables evidence-related rewards in the evidence and format reward.Source: §5.2 · pp. 7–8, Table 3 · p. 8

5.3 Analysis

• GRPO generally outperforms PPO, particularly on larger models and complex multi-hop tasks, and converges faster to a higher reward ceiling.The authors attribute PPO’s difficulty on complex tasks partly to sparse, delayed rewards and its critic’s instability and warm-up phase.Source: Table 4 · p. 8, §5.3 · p. 8

• Across seven benchmarks, R-Search demonstrates superiority while autonomously optimizing reasoning-search trajectories through multi-reward signals.Source: §6 · p. 8
• R-Search explicitly generates evidence when invoking external search, enabling it to function as a modular search tool.Source: §6 · p. 8
• The design allows resource-intensive search processes to be efficiently offloaded to local models.Source: §6 · p. 8

• Training used only the 2WikiMQA dataset, while performance improvements were achieved on both in-domain and out-of-domain tasks.Source: Limitations · p. 9
• The authors suggest that incorporating more high-quality knowledge from diverse domains may further enhance performance.Source: Limitations · p. 9
• Future work should explore integrating more diverse, high-quality knowledge sources to improve R-Search’s effectiveness.Source: Limitations · p. 9

A.1 Datasets

• The evaluation uses four multi-hop datasets—HotpotQA, 2WikiMultiHopQA, MuSiQue, and Bamboogle—and three single-hop datasets—NQ, PopQA, and TriviaQA.Source: §A.1 · p. 12
• Multi-hop QA tests whether R-Search handles complex logic- and knowledge-intensive questions through intermediate sub-questions and a final answer.Source: §A.1 · p. 12
• Single-hop QA evaluates knowledge-intensive factual answering and robustness across questions with varying complexity.Source: §A.1 · p. 12
• The first three multi-hop datasets use 500 test samples each, Bamboogle uses all 125 provided test samples, and single-hop evaluation randomly samples 500 examples per dataset.Source: §A.1 · p. 12

A.2 Results

• The section includes all evaluation metrics for both analyses, but these passages do not provide their values.Source: §A.2 · p. 12
• Tables 6 and 7 present evaluation metrics for the ablation study and the PPO-versus-GRPO comparison, respectively.Source: §A.2 · p. 12
• The reported trends for EM are consistent with those for F1.Source: §A.2 · p. 12

A.3 More Implementation Details

• Inference uses a temperature of 0.1 across all models to reduce uncertainty.Source: §A.3 · p. 12

B.1 System Template

• The rollout template covers reasoning, retrieval, evidence integration, and answer generation, while allowing the model to skip search when external knowledge is unnecessary.It guides reasoning before each action without imposing a fixed reasoning format, and uses tags to mark functional segments such as search, observations, evidence, and answers.Source: §B.1 · p. 12

B.2 Evidence Template

• Before the final answer, the model generates evidence from the question and all retrieved texts to identify key facts from a global perspective.The evidence is integrated into the reasoning process to deepen interaction between reasoning and retrieved knowledge.Source: §B.2 · p. 12

C Case Study

• In 2-hop and 4-hop examples, the model plans, retrieves when needed, and uses intermediate conclusions to guide later retrieval.R-Search applies multi-stage, multi-type rewards in these examples; the tables identify MuSiQue and 2WikiMQA cases, respectively.Source: C Case Study · p. 12, Table 8 · p. 15, Table 9 · p. 16
• The generated evidence is described as clear, well-structured, and informative, making it easy to transfer to downstream models for final answer generation.Source: C Case Study · p. 13
• The appendix includes an ablation study and a PPO-versus-GRPO performance comparison, but the captions do not state their outcomes.Source: Table 6 · p. 14, Table 7 · p. 14