PaperlayerView exact revision on arXiv

Source-linked AI summary

Process vs. Outcome Reward: Which is Better for Agentic RAG Reinforcement Learning

Wenlin Zhang, Xiangyang Li, Kuicai Dong, Yichao Wang, Pengyue Jia, Xiaopeng Li, Yingyi Zhang, Derong Xu, Zhaocheng Du, Huifeng Guo, Ruiming Tang, Xiangyu Zhao

arXiv:2505.14069v3cs.IR

TL;DR

Agentic RAG needs more adaptable reasoning than traditional static workflows provide, while outcome-supervised RL suffers from inefficient exploration, gradient conflict, and sparse rewards. ReasonRAG addresses these issues by using MCTS and SPRE to construct RAG-ProGuide for process-supervised policy optimization across query generation, evidence extraction, and answer generation. It achieves superior performance across five benchmark datasets using 5k training instances, compared with 90k required by Search-R1.

  • Problem

    Outcome-supervised agentic RAG provides delayed, sparse final-answer feedback, limiting exploration efficiency and potentially creating gradient conflict during multistep reasoning.

  • Method

    ReasonRAG uses MCTS and SPRE to construct RAG-ProGuide, a process-level supervision dataset for optimizing query generation, evidence extraction, and answer generation.

  • Results

    ReasonRAG achieves superior performance across five benchmark datasets using 5k training instances, compared with 90k required by Search-R1.

  • Takeaways & Limitations

    Fine-grained process-level rewards support agentic RAG policy optimization with stronger performance and lower training-data requirements than the compared outcome-supervised approach.

  • Takeaways & Limitations

    Process-level annotation requires exploring process actions and incurs higher data-rollout time costs than outcome supervision.

Abstract

from arXiv · show

Retrieval-augmented generation (RAG) enhances the text generation capabilities of large language models (LLMs) by integrating external knowledge and up-to-date information. However, traditional RAG systems are limited by static workflows and lack the adaptability required for multistep reasoning and complex task management. To address these limitations, agentic RAG systems (e.g., DeepResearch) have been proposed, enabling dynamic retrieval strategies, iterative context refinement, and adaptive workflows for handling complex search queries beyond the capabilities of conventional RAG. Recent advances, such as Search-R1, have demonstrated promising gains using outcome-based reinforcement learning, where the correctness of the final answer serves as the reward signal. Nevertheless, such outcome-supervised agentic RAG methods face challenges including low exploration efficiency, gradient conflict, and sparse reward signals. To overcome these challenges, we propose to utilize fine-grained, process-level rewards to improve training stability, reduce computational costs, and enhance efficiency. Specifically, we introduce a novel method ReasonRAG that automatically constructs RAG-ProGuide, a high-quality dataset providing process-level rewards for (i) query generation, (ii) evidence extraction, and (iii) answer generation, thereby enhancing model inherent capabilities via process-supervised reinforcement learning. With the process-level policy optimization, the proposed framework empowers LLMs to autonomously invoke search, generate queries, extract relevant evidence, and produce final answers. Compared to existing approaches such as Search-R1 and traditional RAG systems, ReasonRAG, leveraging RAG-ProGuide, achieves superior performance on five benchmark datasets using only 5k training instances, significantly fewer than the 90k training instances required by Search-R1.

1 Introduction

Traditional RAG uses static workflows that struggle with complex, multistep reasoning, while outcome-supervised RL provides sparse, delayed feedback. ReasonRAG addresses these limitations with process-level rewards, MCTS exploration, and the RAG-ProGuide dataset for agentic RAG.

  • Motivation: Traditional RAG systems use linear, static workflows that limit complex multistep reasoning, contextual integration, and iterative response refinement.Agentic RAG introduces dynamic retrieval strategies, enhanced contextual understanding, and iterative refinement.
  • Limitations of Outcome-Supervised RL: Outcome-supervised RL suffers from inefficient exploration because rewards arrive only after the model generates a complete reasoning chain.Intermediate errors are not identified when they occur, limiting the feedback available during exploration.
  • Limitations of Outcome-Supervised RL: Late mistakes can penalize correct early actions, creating gradient conflict, while final-answer-only feedback produces sparse rewards.These limitations increase the training data and steps needed for convergence.
  • Process-Supervision Challenges: Process-supervised RL for RAG must design efficient intermediate rewards while avoiding the prohibitive cost of manually annotating complex retrieval steps.Rewards should favor short, helpful paths to correct answers and penalize unnecessarily circuitous reasoning.
  • ReasonRAG: ReasonRAG combines MCTS exploration with SPRE reward assignment to identify high-reward intermediate reasoning steps for agentic RAG.The framework automatically constructs process-level annotations rather than relying solely on final-answer correctness.
  • Contributions: ReasonRAG introduces RAG-ProGuide, an off-policy process-level annotation dataset, and evaluates process- versus outcome-supervised RL across five benchmark datasets.The reported experiments use Qwen2.5-7B-Instruct and emphasize both comparative performance and training efficiency.

2 ReasonRAG Framework

ReasonRAG builds process-level supervision for agentic RAG by combining SPRE reward estimation, MCTS exploration, and preference optimization. Its workflow supports adaptive reasoning through query generation, evidence extraction, and answer generation, while RAG-ProGuide captures diverse intermediate decisions.

  • Shortest Path Reward Estimation (SPRE): SPRE evaluates partial reasoning paths through simulated continuations, combining final-answer correctness with penalties for unnecessarily long trajectories.The method repeats rollouts until a final answer is obtained and estimates rewards with step-based penalties.
  • Monte Carlo Tree Search (MCTS) for Process-level Exploration: MCTS explores agentic RAG’s large decision space by selecting promising paths, expanding sampled actions, and backpropagating SPRE-estimated rewards.States encode the question, prior reasoning outputs, and workflow stage; actions include query generation, evidence extraction, and answer generation.
  • RAG-ProGuide Dataset: RAG-ProGuide is constructed from MCTS trajectories and filtered preference pairs, producing 4,603 questions and 13,289 distinct preference pairs.The questions are sampled from PopQA, HotpotQA, and 2WikiMultihopQA, and branches without final answers are pruned.
  • RAG-ProGuide Dataset: RAG-ProGuide covers single-hop and multi-hop reasoning with diverse comparisons across query generation, evidence extraction, and answer generation stages.The dataset reports balanced distributions of three reasoning actions and varied iteration counts, response lengths, and reward gaps.
  • Process-Supervised Preference Optimization: DPO uses process-supervised preference data to tune the policy toward preferred reasoning trajectories, while inference adaptively invokes search according to question complexity.ReasonRAG cycles through Reasoning, Grounding, and Terminal states during agentic RAG inference.

3 Experiments

Experiments compare ReasonRAG with outcome-supervised, supervised, and other RAG baselines across five benchmarks. Results indicate stronger data efficiency, multi-hop performance, training efficiency, adaptive retrieval depth, and sensitivity to retrieved-context quantity.

  • Main Results: ReasonRAG trained on 5k queries outperforms Search-R1 trained on 90k queries across the evaluated benchmarks.This comparison is reported as a data-efficiency advantage over the Search-R1 baseline.
  • Main Results: ReasonRAG achieves 48.9% F1 on HotpotQA, exceeding AutoRAG at 43.7% and Search-R1 at 47.0%.The compared models were trained on larger datasets.
  • Main Results: ReasonRAG achieves higher F1 scores than other baselines on the out-of-domain Bamboogle and MuSiQue benchmarks.The paper describes this as strong generalization to out-of-domain data.
  • Training Efficiency: ReasonRAG reaches stronger EM with fewer GPU hours than Search-R1, while the efficiency gap is especially notable on multi-hop tasks.On PopQA, both models improve comparably rapidly; on 2WikiMultiHopQA and HotpotQA, ReasonRAG improves faster and more substantially.
  • Optimization Strategies: ReasonRAG with process-level rewards outperforms its outcome-level and SFT variants across in-domain and out-of-domain datasets.Outcome-level training requires substantially more data to match comparable process-level performance, while SFT overfits multi-hop paths and generalizes poorly.
  • Impact of Search: Performance improves with more retrieval iterations before saturating: PopQA converges within 2–3 steps, whereas multi-hop datasets require 3–5.The base model without preference optimization struggles in multi-hop settings.
  • Impact of Search: Increasing retrieved documents from k = 1 to k = 3 improves performance across datasets, while k = 5 adds no gain on PopQA and HotpotQA.2WikiMultiHopQA continues benefiting from richer retrieved context.

4 Related Works

Related work progresses from prompt-engineered agentic RAG and supervised fine-tuning toward reinforcement-learning approaches for sequential reasoning. These approaches target interaction with external information and complex reasoning capabilities.

  • Prompt-Based Agentic RAG: Prompt-based agentic RAG manually decomposes tasks into retrieval judgment, query generation, evidence extraction, and answer generation.Related approaches also explore personalization, graph-based retrieval, and reranking.
  • RL-Based Agentic RAG: RL-based agentic RAG builds on reinforcement learning for sequential decision-making and the demonstrated potential of outcome-supervised reasoning.The cited discussion presents outcome-supervised RL as an increasingly prominent paradigm for end-to-end optimization.

5 Conclusion

ReasonRAG combines MCTS with process-level supervision to optimize agentic RAG policies for query generation, evidence extraction, and answer synthesis. Its search procedure selects, expands, and backpropagates candidate reasoning paths while favoring efficient paths to correct answers.

  • 5 Conclusion: ReasonRAG integrates MCTS with agentic RAG to generate RAG-ProGuide, a dataset of process-level supervision prioritizing shortest correct reasoning paths.RAG-ProGuide supports preference-based policy optimization for autonomous query generation, evidence extraction, and answer synthesis.
  • 5 Conclusion: ReasonRAG achieves superior performance on five benchmark datasets with 5k training instances, compared with 90k required by Search-R1.The reported comparison attributes this efficiency to high-quality process-level rewards in RAG-ProGuide.
  • 5 Conclusion: MCTS represents each reasoning state with visit count, intermediate correctness value, and stage, then iteratively performs selection, expansion, and backpropagation.State transitions append each selected action to the current state, while Q(s) records an intermediate annotation in [0, 1].
  • 5 Conclusion: Node selection balances exploration and exploitation using state values and visit frequencies, progressively favoring nodes with higher Q-values and fewer visits.The UCT score uses c_uct to control the exploration degree.
  • 5 Conclusion: Expansion samples one RAG reasoning step, creates a child node, and evaluates the resulting intermediate process to initialize its reward.LLM-based evaluation avoids time-consuming iterative simulations during exploration.

B Process Evaluation

The process evaluations compare partial, incorrect, and complete RAG reasoning traces, showing how retrieval and reasoning quality affect answer construction. A case study contrasts pre-optimization failures with process-supervised behavior that performs the main agentic RAG steps autonomously.

  • B Process Evaluation: The partial case retrieves relevant evidence but misses required entities, producing incomplete inference.The trace is categorized as partially correct because retrieval succeeds without covering all information needed for the answer.
  • B Process Evaluation: The incorrect case misidentifies directors, causing the reasoning chain to rely on factual errors.The comparison concerns the question of which film has the director who died later.
  • B Process Evaluation: The fully correct case decomposes the question into sequential sub-queries, retrieves biographical information, and synthesizes the answer through structured comparison.The reasoning trace shows factual grounding and answer justification despite minor query-formulation imperfections.
  • B Process Evaluation: Before policy optimization, the model generates an inappropriate query and is misled by irrelevant information; afterward, it autonomously performs query generation, evidence extraction, and answer generation.The comparison is illustrated on the 2WikiMultihopQA dataset.

D Evaluation Dataset Details

The evaluation uses process-annotated questions for training and held-out or official development splits for testing across PopQA, HotpotQA, and 2WikiMultiHopQA. The datasets cover factual recall and multi-hop reasoning, and performance is measured with EM and F1.

  • D Evaluation Dataset Details: 4,603 process-level annotated questions form the policy-optimization training set: 704 from PopQA, 2,843 from HotpotQA, and 1,056 from 2WikiMultiHopQA.The test data use remaining unlabeled PopQA samples and official development splits of HotpotQA and 2WikiMultiHopQA.
  • D Evaluation Dataset Details: HotpotQA and 2WikiMultiHopQA evaluate multi-hop reasoning by requiring information to be combined across multiple passages.HotpotQA includes sentence-level supporting facts, while 2WikiMultiHopQA constructs explicit reasoning paths from Wikidata.
  • D Evaluation Dataset Details: PopQA evaluates open-domain factual recall across a broad range of factual knowledge.Its remaining unlabeled samples are used as the test set.
  • D Evaluation Dataset Details: Exact Match measures the percentage of predictions exactly matching a reference answer, while F1 measures token-level overlap with the ground truth.The evaluation follows the official metrics implementation in the FlashRAG toolkit.

E Implementation Details

The implementation records experiment artifacts and their resource links in a dedicated table. The supplied implementation details identify datasets, models, baselines, and external knowledge resources as included artifacts.

  • E Implementation Details: Table 5 summarizes the datasets, models, baselines, external knowledge base, and other artifacts used in the experiments.The table provides resource links for these experimental materials.

E.1 Implementation Details of ReasonRAG

The implementation uses Wikidump 2018 augmented with dataset content as its knowledge source, with BGE retrieving the top three documents. Qwen2.5-7B-Instruct serves as the baseline for methods without fine-tuning.

  • Wikidump 2018 is augmented with relevant content from PopQA, HotpotQA, and 2WikiMultiHopQA.The corpus follows the FlashRAG toolkit setup.
  • BGE consistently retrieves the top 3 documents to support retrieval.
  • Qwen2.5-7B-Instruct is used as the baseline model for inference when methods require no fine-tuning.

E.2 Implementation Details of Baselines

Baseline comparisons use FlashRAG reproductions for multiple RAG methods, while the implementation supplies prompts for reasoning, grounding, and process evaluation. RAG-ProGuide is constructed from three FlashRAG datasets and released for academic, non-commercial research.

  • Baseline implementations: FlashRAG reproductions provide baseline implementations for Naïve Generation, Standard RAG, FLARE, Iter-Retgen, RECOMP, LongLLMLingua, and Selective-Context.Self-RAG uses the checkpoint provided in the FlashRAG reproduction, while AdaptiveRAG uses its router with Qwen2.5-7B-Instruct as the reasoning model.
  • Prompting: The prompts cover reasoning, grounding in the agentic RAG workflow, and process evaluation for node expansion.The prompt details are shown in Figures 12, 13, and 14.
  • RAG-ProGuide: RAG-ProGuide is constructed from PopQA, HotpotQA, and 2WikiMultiHopQA in the FlashRAG dataset.The source datasets use the CC-BY-SA-4.0 license.
  • Prompting: Figure 12 presents the system prompt for Reasoning.
  • Prompting: Figure 13 presents the system prompt for Evidence Extraction.
  • Prompting: Figure 14 presents the system prompt for Process Evaluation.

H Limitations

Process-supervised RL requires more time to collect process-level annotations than outcome supervision, although the resulting data enables greater performance gains with fewer samples. The framework also carries risks from uncontrollable or harmful generated content retrieved from search engines.

  • Data-collection cost: Process-level annotation rollouts incur higher time costs than outcome-supervised data collection.The added cost comes from exploring process-level actions for fine-grained reward annotation.
  • Data efficiency: Despite the greater annotation cost, the process-supervised data achieves greater performance gains with fewer training samples.The paper attributes this to superior data quality observed in training efficiency.
  • Safety: LLMs may generate uncontrollable responses when retrieved search content includes racist or harmful information.The authors advise using secure search engines or knowledge corpora and evaluating in safe environments and datasets.
Loading 2505.14069v3…