PaperlayerView exact revision on arXiv

Source-linked AI summary

LeTS: Learning to Think-and-Search via Process-and-Outcome Reward Hybridization

Qi Zhang, Shouqing Yang, Lirong Gao, Hao Chen, Xiaomeng Hu, Jinglei Chen, Jiexiang Wang, Sheng Guo, Bo Zheng, Haobo Wang, Junbo Zhao

arXiv:2505.17447v1cs.CL

TL;DR

Outcome-supervised RL for RAG often overlooks the correctness of intermediate think-and-search steps. LeTS adds rule-based process rewards and rescales outcome advantages; across RAG benchmarks, it reports a 2.61% performance gain alongside fewer generated tokens and search times.

  • Problem

    Outcome-level rewards can be applied uniformly across a rollout, overlooking the relative importance of individual reasoning steps.

    Source: §3.4 · p. 5

  • Method

    LeTS combines rule-based process rewards with outcome-level rewards by rescaling outcome advantages at each reasoning step according to process rewards.

    Source: §1 · p. 2, §3.4 · p. 5

  • Results

    LeTS reports a 2.61% performance gain across various RAG benchmarks, with generated tokens reduced by 11.15% and search times by 30.85%.

    Source: §1 · p. 2

  • Takeaways & Limitations

    The results highlight the potential of process- and outcome-level reward hybridization to boost LLM reasoning via RL in other scenarios.

    Source: §1 · p. 2

    Cost constraints prevented validation on language models larger than the Qwen-7B and Qwen-3B series.

    Source: Limitations · p. 9

Abstract

from arXiv · show

Large language models (LLMs) have demonstrated impressive capabilities in reasoning with the emergence of reasoning models like OpenAI-o1 and DeepSeek-R1. Recent research focuses on integrating reasoning capabilities into the realm of retrieval-augmented generation (RAG) via outcome-supervised reinforcement learning (RL) approaches, while the correctness of intermediate think-and-search steps is usually neglected. To address this issue, we design a process-level reward module to mitigate the unawareness of intermediate reasoning steps in outcome-level supervision without additional annotation. Grounded on this, we propose Learning to Think-and-Search (LeTS), a novel framework that hybridizes stepwise process reward and outcome-based reward to current RL methods for RAG. Extensive experiments demonstrate the generalization and inference efficiency of LeTS across various RAG benchmarks. In addition, these results reveal the potential of process- and outcome-level reward hybridization in boosting LLMs' reasoning ability via RL under other scenarios. The code will be released soon.

1 Introduction

Outcome-supervised think-and-search methods can miss errors in intermediate retrieval steps. LeTS combines process-level and outcome-level rewards, with reported performance and inference-efficiency gains across RAG benchmarks.

1 Introduction

  • Vanilla RAG can be insufficient for multi-hop queries, while prompting- and training-based approaches face generalization or supervision limitations.Prompting methods depend on the model’s inherent capacity; training-based approaches are described as having limited generalization and lacking robust reasoning supervision.Source: §1 · pp. 1–2
  • Outcome-level RL rewards can overlook intermediate reasoning quality, contributing to redundant retrieval and irrelevant content that misleads the model.The paper identifies repeated similar or unnecessary retrieval and irrelevant content introduced during intermediate steps as two issues.Source: §1 · p. 2
  • LeTS combines outcome-level feedback with two rule-based process rewards: one penalizes redundant knowledge, and the other rewards correct actions in weaker rollouts.The framework integrates these process rewards through advantage rescaling.Source: §1 · p. 2
  • LeTS reports a 2.61% performance gain across RAG benchmarks, with average generated tokens and search times reduced by 11.15% and 30.85%, respectively.The introduction presents these as performance and inference-efficiency results.Source: §1 · p. 2
  • The reported performance and inference efficiency motivate the paper’s proposed potential for process- and outcome-level reward hybridization to support LLM reasoning via RL in other scenarios.The paper frames benefits beyond RAG as potential, rather than reporting experiments in those other scenarios here.Source: §1 · p. 2

2 Related Work

Prior RAG methods progress from fixed retrieval to iterative and learned retrieval, while RL approaches often rely on outcome-level rewards. LeTS distinguishes itself with retrieval-specific process supervision combined with outcome feedback.

2.1 Retrieval-Augmented Generation

  • RAG work moves from fixed retrieve-then-generate methods to iterative retrieval and learned retrieval policies, with ReSearch using RL to decide when to retrieve.The passage notes that iterative approaches can depend on prompt engineering and large black-box LLMs; ReSearch relies only on outcome-level rewards.Source: §2.1 · p. 2

2.2 Large Language Models and Reinforcement Learning

  • Prior retrieval RL methods use coarse outcome rewards, while S2R adds process supervision for verification and correction but does not explicitly supervise retrieval behavior.LeTS instead designs retrieval-specific process rewards and combines them with outcome-level feedback.Source: §2.2 · p. 3

3 Methodology

LeTS extends GRPO for think-and-search RAG by combining outcome rewards with rule-based stepwise process rewards. Its formulation selectively retrieves evidence, scores answer and format correctness, and uses process signals to refine step-level learning.

3 Methodology

  • LeTS uses GRPO, which estimates value baselines from a group of rollouts rather than a separately trained critic.The methodology section identifies GRPO as the framework underlying LeTS.Source: §3 · p. 3

3.1 Think-and-Search Formulation

  • The model selectively retrieves when external information is needed, appends retrieved documents to the context, and continues reasoning before producing a final answer.The process consists of think-and-search steps followed by a final reasoning trace and answer.Source: §3.1 · pp. 3–4

3.2 Outcome-Level Reward Modeling

  • Outcome-level reward combines reasoning-format correctness with final-answer accuracy, measured using F1 against the gold answer.The format check covers the think-and-search chain and final-answer structure.Source: §3.2 · p. 4

3.3 Process-Level Reward Modeling

  • Process rewards address overlooked intermediate-step quality with rollout-level penalties for redundant retrieval and group-level matching of weaker steps to stronger rollouts.Malformed-format rollouts omit process rewards; the knowledge-match reward is omitted when no valid outperforming rollout is available.Source: §3.3 · pp. 4–5

3.4 Process-and-Outcome Reward Hybridization

  • Stepwise outcome-level advantage rescaling adjusts each step’s outcome advantage according to its process signal while preserving the trajectory-level reward sign.Rescaling is not applied to the final reasoning step or rollouts excluded from process-reward computation.Source: §3.4 · p. 5

3.5 Reinforcement Learning Formulation

  • The RL objective applies each rescaled step advantage to its tokens, clips policy updates, regularizes against a reference model, and masks retrieved tokens during training.The KL regularization weight is β; retrieved-token masking is intended to prevent retrieval-content noise from skewing the learning signal.Source: §3.5 · pp. 5–6

4 Experiments

LeTS is evaluated against retrieval and RL baselines using MusiQue training data, with results across multi-hop and single-hop RAG benchmarks. It improves over same-backbone RL methods and also performs well on smaller instruct models.

4.1 Experimental Settings

  • LeTS is trained for two epochs on MusiQue’s training split, and ReSearch is the primary strong baseline among the compared retrieval and RL methods.The experiments use Qwen2.5-7B, Qwen2.5-7B-Instruct, and Qwen2.5-3B-Instruct; MusiQue is selected for its diverse multi-hop questions and quality control.Source: §4.1 · p. 6

4.2 Main Results

  • LeTS-Qwen-7B-Instruct improves over same-backbone RL methods by 2.61% EM and 2.33% LJ; LeTS-Qwen-7B gains 1.67% EM and 1.99% LJ.The authors also report performance comparable to instruct-tuned methods for Qwen2.5-7B-Base and good performance on Qwen2.5-3B-Instruct; Qwen2.5-3B-Base is excluded after RL degenerates due to reward hacking.Source: §4.2 · p. 7

4.3 Generalization on Single-Hop Benchmarks

  • LeTS outperforms other baselines on NQ, PopQA, and TriviaQA, extending its reported effectiveness to single-hop settings.Table 2 compares methods on single-hop benchmarks using Qwen-7B-Instruct as the backbone.Source: §4.3 · p. 7, Table 2 · p. 7

5 Ablation Study and Analysis

The analysis reports reduced inference costs and redundant or irrelevant searching, alongside improved training reward dynamics when process-level rewards are added. These findings cover reward ablations, search behavior, and inference efficiency.

5.1 Impact of Process-Level Reward

  • Both rollout-level knowledge redundancy reward and group-level knowledge match reward reduce average search time while maintaining strong overall performance.The ablation is on MusiQue; vanilla GRPO uses only outcome-level supervision.Source: §5.1 · p. 7, Table 3 · p. 7
  • Training reward improves with either process-level component, while using both produces a more stable and consistent reward trajectory during RL training.Figure 4 shows training reward for the four methods.Source: §5.1 · p. 7, Figure 4 · p. 7
  • Methods using the in-rollout knowledge redundancy reward—w/. KR and LeTS—show lower redundancy frequencies across training.Knowledge redundancy frequency is the rate of identical documents retrieved multiple times within one rollout; Figure 5 tracks it across four methods.Source: §5.1 · pp. 7–8, Figure 5 · p. 8

5.2 Alleviation of the Redundant Searching Phenomenon

  • Baseline models retrieve more redundant information when their generations fail, while LeTS reduces in-rollout redundancy across four multi-hop benchmarks.Table 5 separates redundancy frequencies for correct and incorrect generations.Source: §5.2 · p. 8, Table 5 · p. 8

5.3 Alleviation of the Irrelevant Searching Phenomenon

  • LeTS consistently produces more concise and relevant intermediate search queries than other methods.The comparison is reported in Table 10.Source: §5.3 · p. 8

5.4 Efficiency Analysis

  • LeTS reduces generated tokens and search steps; average search times fall by approximately 30% without compromising performance across four multi-hop benchmarks.Table 4 compares LeTS with ReSearch and excludes tokens from retrieved documents from its token counts.Source: §5.4 · p. 8, Table 4 · p. 8

6 Conclusion

The conclusion presents two process-level rewards and LeTS, which combines process- and outcome-based reward through advantage rescaling. It reports generalization and inference efficiency across language models and single- and multi-hop RAG benchmarks.

6 Conclusion

  • LeTS hybridizes process- and outcome-based rewards through advantage rescaling, with reported generalization and inference efficiency across language models and single- and multi-hop RAG benchmarks.The paper frames the rewards as addressing two drawbacks of outcome-supervised RL for RAG.Source: §6 · p. 8

Limitations

The authors have not validated LeTS on larger language models because of cost constraints.

Limitations

  • Cost constraints limited validation to Qwen-7B and Qwen-3B series; larger language models remain untested.The authors describe their results on the tested series as significant.Source: Limitations · p. 9

A Experimental Settings

The evaluation covers multi-hop and single-hop question answering, using Exact Match and an LLM-as-a-Judge assessment of semantic correctness.

A Experimental Settings

  • Four multi-hop benchmarks use development sets for HotpotQA, 2WikiMultiHopQA, and MusiQue, and the test set for Bamboogle.The respective sample counts are 7,405, 12,576, 2,417, and 125.Source: A Experimental Settings · p. 11
  • The single-hop evaluation uses NQ, PopQA, and TriviaQA.Source: A Experimental Settings · p. 11
  • Evaluation uses Exact Match and LLM-as-a-Judge, with gpt-4o-mini judging semantic correctness of final answers.Exact Match counts a prediction as correct only when it exactly matches the ground truth.Source: A Experimental Settings · p. 11

B Implementation Details

The appendix specifies LeTS training hardware and describes prompts for structured think-and-search training and semantic answer evaluation.

B Implementation Details

C Prompt Templates

  • The appendix provides prompt templates to support consistency and reproducibility across experimental stages.Source: C Prompt Templates · p. 11

C.1 Prompt Templates for Training

  • Training prompts guide base and instruction-tuned models to follow structured reasoning and retrieval conventions.The base prompt frames interaction as a conversation and defines reasoning, search, and answer tags; the instruction-tuned prompt uses similar conventions.Source: §C.1 · p. 11, Table 7 · p. 11

C.2 Prompt Templates for LLM-as-a-Judge Evaluation

  • The LLM-as-a-Judge evaluation prompt standardizes semantic correctness judgments and their output format.Predictions count as correct when semantically equivalent to at least one ground-truth answer.Source: §C.2 · pp. 11–12, Table 9 · p. 12

D Learning to Search via Reinforcement Learning on Base Version of Small Language Models

With Qwen2.5-3B-Base, outcome-level F1 reward does not stimulate think-and-search behavior; the appendix also outlines LeTS’s process-and-outcome GRPO procedure.

D Learning to Search via Reinforcement Learning on Base Version of Small Language Models

  • As reward rose to around 0.3, Qwen2.5-3B-Base search time fell to zero, and the authors conclude F1-based outcome reward fails to stimulate its think-and-search ability.They attribute the collapse in search time to reward hacking and leave this as an open problem.Source: D Learning to Search via Reinforcement… · p. 12, Figure 6 · p. 12

E Pseudo Code for LeTS

  • LeTS samples rollouts, computes outcome rewards, and assigns stepwise process rewards to well-formatted rollouts in the outperforming group before computing GRPO loss.The pseudocode identifies the process reward as a group-match reward.Source: E Pseudo Code for LeTS · p. 13

F Case Studies

A comparative case study contrasts ReSearch’s incorrect answer after redundant and irrelevant searches with LeTS’s correct answer.

F Case Studies

Loading 2505.17447v1…