PaperlayerView exact revision on arXiv

Source-linked AI summary

Search-P1: Path-Centric Reward Shaping for Stable and Efficient Agentic RAG Training

Tianle Xia, Ming Xu, Lingxiang Hu, Yiding Sun, Wenwei Li, Linfang Shang, Liqun Liu, Peng Shu, Huan Yu, Jie Jiang

arXiv:2602.22576v1cs.CLcs.IRcs.LG

TL;DR

Agentic RAG needs better training signals because outcome-based reinforcement learning can ignore intermediate reasoning quality and assign no useful credit to failed trajectories. SEARCH-P1 introduces path-centric reward shaping with dual-track scoring and soft outcome scoring, and reports a 7.7-point average accuracy gain over existing methods across public and industrial QA settings.

  • Problem

    Outcome-based RL for agentic RAG provides sparse feedback, wastes partially correct trajectories, and can converge slowly because intermediate reasoning quality is ignored.

  • Method

    SEARCH-P1 evaluates reasoning paths with dual-track self-consistency and reference-alignment scoring, order-agnostic step matching, and soft outcome scoring for partial credit.

  • Results

    7.7 points: SEARCH-P1's average accuracy gain over existing methods is reported across public QA benchmarks and an internal advertising dataset.

  • Takeaways & Limitations

    Path-centric rewards provide fine-grained supervision for diverse multi-step reasoning and improve accuracy and efficiency across academic and industrial settings.

Abstract

from arXiv · show

Retrieval-Augmented Generation (RAG) enhances large language models (LLMs) by incorporating external knowledge, yet traditional single-round retrieval struggles with complex multi-step reasoning. Agentic RAG addresses this by enabling LLMs to dynamically decide when and what to retrieve, but current RL-based training methods suffer from sparse outcome rewards that discard intermediate signals and low sample efficiency where failed samples contribute nothing. We propose Search-P1, a framework that introduces path-centric reward shaping for agentic RAG training, comprising two key components: (1) Path-Centric Reward, which evaluates the structural quality of reasoning trajectories through order-agnostic step coverage and soft scoring that extracts learning signals even from failed samples, and (2) Dual-Track Path Scoring with offline-generated reference planners that assesses paths from both self-consistency and reference-alignment perspectives. Experiments on multiple QA benchmarks demonstrate that Search-P1 achieves significant improvements over Search-R1 and other strong baselines, with an average accuracy gain of 7.7 points.

1 Introduction

SEARCH-P1 targets weaknesses in outcome-reward training for agentic RAG by evaluating reasoning paths and retaining signals from partially unsuccessful trajectories. Across public and industrial QA settings, it reports improved accuracy and consistent gains across model scales.

  • Single-round retrieval is insufficient for complex multi-step reasoning across multiple knowledge domains.
  • Outcome-based RL for agentic RAG suffers from sparse rewards, low sample efficiency, and slow convergence.These limitations arise because intermediate reasoning quality is ignored and partially correct trajectories can receive zero reward.
  • SEARCH-P1 uses dual-track path scoring to evaluate trajectories through self-consistency and reference-alignment perspectives with order-agnostic matching.
  • 7.7 points: SEARCH-P1's average accuracy gain over existing methods is reported across public QA benchmarks and an internal advertising dataset.The paper also reports consistent improvements across models and settings.
  • Path-centric reward shaping extracts learning signals from failed trajectories through path-level reward and soft outcome scoring.

2 Related Work

Prior agentic RAG work uses prompting or reinforcement learning to support multi-step retrieval, but these approaches remain dependent on instruction following or largely binary outcome feedback. SEARCH-P1 instead emphasizes denser path-centric training signals.

  • Prompt-Based Agentic RAG: Prompt-based agentic RAG methods interleave reasoning with retrieval or use sophisticated retrieval strategies for multi-step search.Their effectiveness depends heavily on the base model's instruction-following ability.
  • RL-Based Agentic RAG: RL-based agentic RAG methods add auxiliary signals for training stability or search efficiency, while some process-reward methods still rely primarily on binary outcome feedback.SEARCH-P1 proposes path-centric reward shaping to provide denser training signals.

3 Methodology

SEARCH-P1 restructures agentic RAG training around explicit plans and path-centric rewards, evaluating trajectories beyond final-answer correctness. Its dual-track scoring and soft outcome scoring provide denser signals, including for incorrect trajectories.

  • Problem formulation: Agentic RAG trajectories interleave reasoning, search actions, observations, and a final answer, while SEARCH-P1 adds an explicit planner exposing the intended reasoning structure.The planner also provides a self-declared plan against which execution can be evaluated.
  • Path-centric reward: SEARCH-P1 combines path-centric rewards with soft outcome scoring instead of relying solely on final-answer correctness.The complete reward includes path, outcome, and format components, with balancing coefficients.
  • Reference planner generation: Offline reference planners are distilled by generating candidate trajectories, filtering correct answers, voting, and extracting the essential steps shared across successful trajectories.The resulting reference reasoning path is used for subsequent path evaluation.
  • Dual-track path scoring: Dual-track scoring evaluates self-consistency with the model’s own plan and reference alignment through order-agnostic coverage of essential steps.The self-consistency track assesses plan execution, while the reference-alignment track counts accomplished reference steps regardless of order.
  • Dual-track path scoring: Efficiency ratios discourage redundant actions, and the final path reward takes the maximum track score so a suboptimal reference plan does not dilute a better model strategy.This design also permits the reference-alignment track to dominate when it better represents trajectory quality.
  • Soft outcome scoring: Soft outcome scoring assigns useful learning signals to incorrect trajectories by combining partial answer correctness with reasoning quality.This converts previously zero-reward failed samples into training signals based on path quality.

4 Experiments

Experiments evaluate SEARCH-P1 across public and industrial QA benchmarks, comparing reward components and showing higher accuracy, faster convergence, and benefits from soft scoring.

  • Experimental Setup: The evaluation covers seven public QA benchmarks spanning general and multi-hop QA, plus the internal AD-QA dataset, using Accuracy as the primary metric.Experiments use Qwen2.5-7B-Instruct and Qwen2.5-3B-Instruct with Wikipedia retrieval and E5.
  • Main Results: SEARCH-P1 achieves the highest average accuracy across model sizes, outperforming baselines by 7.7 Avg. ACC over Search-R1 on 7B and 7.9 on 3B.On AD-QA, the 7B model gains 20.6 points over Search-R1.
  • Format Reward: Soft Format converges faster and reaches higher stable rewards than Strict Format and Without Format baselines.Strict Format assigns zero reward for invalid formatting, whereas the buffered design provides continuous feedback.
  • Path-Centric Reward: Removing reference-alignment lowers accuracy by 5.3%, while removing self-consistency lowers it by 3.1%; the full dual-track model performs best.These ablations support the complementary value of reference guidance and internal consistency.

5 Analysis

The analysis examines reward-weight sensitivity, training and inference efficiency, model–algorithm robustness, evaluator sensitivity, and qualitative trajectory improvements. SEARCH-P1 reaches target accuracy faster, produces more concise reasoning, and maintains more consistent turn counts across outcomes.

  • Reward sensitivity: The optimal reward weights λp=0.3 and λa=0.6 balance accuracy as the primary objective with reasoning quality as a regularizer.Too little path weight weakens supervision, while excessive path weight causes reward overfitting; excessive accuracy weight neglects reasoning quality and leads to reward hacking.
  • Training efficiency: SEARCH-P1 reaches Search-R1’s final accuracy of ∼40% within 60 training steps, compared with over 150 for Search-R1.Its interaction turns also steadily decrease, whereas Search-R1’s turns remain flat or increase.
  • Inference efficiency: Both methods require more turns for complex adversarial queries, but SEARCH-P1 keeps successful and unsuccessful turn counts consistent.Search-R1 shows larger success–failure gaps on multi-hop (+60%) and adversarial (+47%) tasks.
  • Base models and RL algorithms: Path-centric rewards improve all tested base-model and RL-algorithm combinations, indicating orthogonality to those choices.Qwen2.5-3B-Instruct slightly outperforms Llama-3.2-3B-Instruct, while GRPO is marginally more accurate and PPO has lower training variance.
  • Evaluator sensitivity: Qwen3-32B produces comparable accuracy (−0.8) and human agreement to the default evaluator, whereas Qwen3-8B lowers accuracy by 3.2 points and outcome-scoring agreement to 78.5%.The evaluator is used during training, but no evaluator calls are needed at inference time.
  • Qualitative analysis: A representative multi-hop QA case study shows more structured decomposition, precise query formulation, and effective information synthesis under path-centric rewards.The comparison is presented qualitatively against baseline reasoning trajectories.

6 Conclusion

SEARCH-P1 introduces path-centric reward shaping for agentic RAG training, evaluating entire reasoning paths to provide fine-grained supervision. Experiments on public QA benchmarks and an internal advertising dataset report improvements in accuracy and efficiency across academic and industrial settings.

  • The framework introduces path-centric reward shaping for agentic RAG training.
  • SEARCH-P1 evaluates the structural quality of entire reasoning paths rather than isolated elements.This provides fine-grained supervision while accommodating diverse multi-step reasoning.
  • Experiments on public QA benchmarks and an internal advertising dataset demonstrate improvements in accuracy and efficiency.The reported evaluation spans both academic and industrial settings.

Ethics Statement

The work uses publicly available datasets for training and evaluation and anonymizes the internal AD-QA dataset before use. It also links improved agentic RAG efficiency to potentially lower deployment resource requirements.

  • The study uses publicly available datasets for training and evaluation.
  • The internal AD-QA dataset is fully anonymized, with personally identifiable information removed before use.
  • Improved agentic RAG efficiency could reduce computational resources required for deployment.The passage frames this as a potential contribution to more sustainable AI.

A AD-QA Dataset

AD-QA is a fully anonymized advertising-domain benchmark with 1,000 multi-hop test instances. Its questions require synthesizing information across at least two knowledge domains, with expert-curated and cross-validated answers.

  • AD-QA contains 1,000 test instances requiring multi-step reasoning in a real-world advertising domain.The benchmark covers campaign configuration, bidding strategies, audience targeting, and conversion tracking.
  • Each AD-QA question requires synthesizing information from at least two distinct knowledge domains.This makes the benchmark challenging for multi-hop reasoning in enterprise settings.
  • AD-QA answers are curated by domain experts and verified through cross-validation.

B.1 Training Configuration

SEARCH-P1 trains agentic RAG with offline reference planning, iterative retrieval, structured outputs, and combined format, path-centric, and outcome rewards. The configuration uses explicit sampling, action-budget, and evaluation procedures for training and inference.

  • Reward Computation: Self-consistency scoring samples three independent reasoning paths and measures pairwise agreement with Jaccard similarity on extracted evidence spans.
  • Reward Computation: Soft outcome scoring applies a decay factor of 0.5 to partial matches when incorrect answers have high-quality reasoning paths.
  • Reference Planner Generation: Reference planners are generated offline with HY 2.0-Instruct and cached before training to avoid runtime overhead.The planners are produced for 90K training samples, averaging 1.91 LLM calls per sample.
  • Agentic RAG Inference: Agentic RAG iteratively generates reasoning, issues search queries, and receives retrieved passages until the action budget is exhausted or an answer is produced.
  • Reward Computation: The reward combines format, dual-track path-centric, and soft outcome signals.The default weights are λf = 0.1, λp = 0.3, and λa = 0.6.
  • Prompt and Output Structure: Training and inference prompts require decomposed subtasks, iterative searches, and structured <reasoning>, <tool_call>, and <answer> outputs.

E.1 Multi-Hop Reasoning Comparison

SEARCH-P1 improves multi-hop reasoning by combining planning-driven retrieval with path-based reward computation and dual-track scoring. Its trajectories identify the correct entity where Search-R1 retrieves irrelevant information, while successful multi-hop cases typically use two search iterations.

  • Reward computation: The reward computation combines format, path, and outcome rewards into a total reward R(y).The algorithm includes validity checks, planner-derived scores, self-consistency and reference-efficiency terms, and exact-answer evaluation.
  • Trajectory comparison: SEARCH-P1 identifies “Bang Bang Rock & Roll” as a complete album title, whereas Search-R1 retrieves information about the wrong entity.The comparison attributes the difference to SEARCH-P1’s planning-driven query versus Search-R1’s imprecise query.
  • Reward components: Format reward improves the model’s ability to produce properly structured responses with parseable answers.The format-reward component is evaluated through format compliance and contributes to the reward design.
  • Search iterations: Multi-hop datasets concentrate successful cases at two search iterations, while failed cases consistently show higher rates of three or more iterations.General QA successes are mostly single-iteration, and the 3B model requires slightly more iterations than the 7B model.
  • Retrieval configuration: Retrieving three documents per search achieves the best average performance, while too few documents may miss evidence and too many may add noise and context length.Some datasets benefit from retrieving five documents, but three is the overall best configuration.

F.8 Detailed LLM Evaluator Analysis

Detailed analyses show that evaluator quality varies by model and dataset, especially when judging reasoning-step coverage in multi-hop tasks. Soft outcome scoring and path-reward components provide larger benefits where partial credit and external guidance are more important.

  • LLM evaluator analysis: Qwen3-8B evaluators show larger drops on multi-hop datasets, including −4.0 on MuSiQue and −4.0 on 2Wiki, because covered-step counting is more challenging.Qwen3-32B shows a smaller average degradation of −1.1.
  • Evaluation design: Dual-track evaluation assesses model trajectories using both self-consistency and reference-alignment criteria.The evaluator compares the trajectory against offline-generated reference planning signals.
  • Path-reward ablation: Removing reference-alignment causes larger drops on multi-hop datasets, while removing self-consistency affects general QA more.The ablation associates external guidance with multi-hop performance and model-generated planning with general QA performance.
  • Base models and RL algorithms: GRPO achieves slightly higher accuracy than PPO across datasets, and Qwen2.5 consistently outperforms Llama-3.2.These comparisons are reported across different base models and RL algorithms.
  • Soft outcome scoring: Soft outcome scoring improves multi-hop QA accuracy by +3.0–3.7%, compared with +1.1–1.5% for general QA and +8.8–11.0% for the internal AD-QA dataset.The results indicate that partial-credit signals are particularly useful for complex enterprise queries.
Loading 2602.22576v1…