Source-linked AI summary
ReARTeR: Retrieval-Augmented Reasoning with Trustworthy Process Rewarding
Zhongxiang Sun, Qipeng Wang, Weijie Yu, Xiaoxue Zang, Kai Zheng, Jun Xu, Xiao Zhang, Song Yang, Han Li
TL;DR
Complex multi-step reasoning remains a challenge for RAG systems, and existing approaches face limitations in PRM explanations, training data, early-step scoring, and reasoning-specific post-training. ReARTeR combines PRM scoring and PEM explanations at test time with MCTS-guided preference-data collection during post-training. It significantly improves reasoning across multiple public multi-step RAG datasets.
Problem
RAG systems remain challenged by complex multi-step reasoning, while existing PRM approaches have limitations in explanations, training data, early-step accuracy, and reasoning-specific post-training.
Source: §1 · p. 1Method
ReARTeR combines PRM scoring and PEM explanations for test-time refinement with MCTS-guided step-level preference-data collection for post-training.
Source: §1 · p. 2Results
ReARTeR significantly improves reasoning on multiple public multi-step reasoning RAG datasets, including reasoning-path quality, search accuracy, and generator refinement ability.
Source: §1 · p. 2Takeaways & Limitations
The reported results validate the feasibility of enhancing RAG reasoning through ReARTeR’s post-training and inference-time scaling.
Source: §1 · p. 2
Abstract
from arXiv · showhide
Retrieval-Augmented Generation (RAG) systems for Large Language Models (LLMs) hold promise in knowledge-intensive tasks but face limitations in complex multi-step reasoning. While recent methods have integrated RAG with chain-of-thought reasoning or test-time search using Process Reward Models (PRMs), these approaches encounter challenges such as a lack of explanations, bias in PRM training data, early-step bias in PRM scores, and insufficient post-training optimization of reasoning potential. To address these issues, we propose Retrieval-Augmented Reasoning through Trustworthy Process Rewarding (ReARTeR), a framework that enhances RAG systems' reasoning capabilities through post-training and test-time scaling. At test time, ReARTeR introduces Trustworthy Process Rewarding via a Process Reward Model for accurate scalar scoring and a Process Explanation Model (PEM) for generating natural language explanations, enabling step refinement. During post-training, it utilizes Monte Carlo Tree Search guided by Trustworthy Process Rewarding to collect high-quality step-level preference data, optimized through Iterative Preference Optimization. ReARTeR addresses three core challenges: (1) misalignment between PRM and PEM, tackled through off-policy preference learning; (2) bias in PRM training data, mitigated by balanced annotation methods and stronger annotations for challenging examples; and (3) early-step bias in PRM, resolved through a temporal-difference-based look-ahead search strategy. Experimental results on multi-step reasoning benchmarks demonstrate significant improvements, underscoring ReARTeR's potential to advance the reasoning capabilities of RAG systems.
1 INTRODUCTION
Complex multi-step reasoning remains challenging for RAG systems, while existing PRM-based approaches face explanation, training-data, early-step accuracy, and post-training optimization challenges. ReARTeR combines test-time process scoring and explanations with MCTS-guided post-training, and reports improved reasoning on multi-step RAG benchmarks.
1 INTRODUCTION
- RAG systems remain challenged by complex multi-step reasoning, and prior RAG–CoT approaches have not fully leveraged LLM reasoning potential.PRM methods also face unexplainable scores, biased training data, weaker accuracy on earlier steps, and a lack of reasoning-specific post-training optimization.Source: §1 · pp. 1–2
- At test time, a PRM scores reasoning steps while a PEM explains scores to support refinement; post-training uses MCTS to collect step-level preference data for generator optimization.MCTS is guided by Trustworthy Process Rewarding, which combines the scoring and explanation models.Source: §1 · p. 2
- Off-policy preference learning aligns PEM explanations with PRM scores, balanced annotations and stronger difficult-example labels address training-data bias, and TD look-ahead search targets early-step bias.The preference labels are positive when refinement after an explanation improves the PRM score, and negative otherwise.Source: §1 · p. 2
- ReARTeR significantly improves reasoning on multiple public multi-step reasoning RAG datasets.The reported improvements include reasoning-path quality, search accuracy, and generator refinement ability.Source: §1 · p. 2
2 RELATED WORK
Prior work explores post-training and test-time search for reasoning, as well as retrieval-augmented approaches to multi-step tasks. The paper positions ReARTeR as combining trustworthy process rewards with both scaling strategies.
2.1 Learning and Search for Reasoning
- Prior post-training methods use reinforcement fine-tuning or iterative preference optimization, while ReARTeR uses MCTS for step-level preferences and KTO for optimization.The cited discussion identifies collecting step-level reasoning preferences as a challenge for earlier approaches.Source: §2.1 · p. 2
- Test-time work includes self-refinement and PRM-based search, while ReARTeR combines PRM-aligned PEM explanations with step-level refinement and improves PRM training for shallow nodes.The described PRM improvements build on balanced examples and add stronger generators for difficult problems with TD-based look-ahead search.Source: §2.1 · p. 3
2.2 Retrieval-Augmented Reasoning
- RAG–CoT methods include follow-up-question reasoning, interleaved retrieval, and iterative demonstrations, but the paper identifies limited use of learning and search to improve RAG reasoning.It also characterizes CR-Planner as using PRMs for search without addressing PRM trustworthiness.Source: §2.2 · p. 3
3 METHOD
ReARTeR combines retrieval-grounded reasoning with PRM scoring and PEM explanations, while training the PRM, aligning the PEM, and post-training the policy to improve reasoning and refinement.
3.1 Overview
- Each reasoning step combines a sub-query, optional retrieved knowledge, and a generator-produced thought; PRM scoring selects a candidate, while low-scoring steps receive PEM explanations for refinement.The policy uses the completed reasoning process to generate the final answer.Source: §3.1 · p. 3
3.2 Process Reward Model Training
- Monte Carlo rollouts can bias PRM supervision toward high scores and leave difficult questions with zero-valued signals; OmegaPRM balances positive and negative examples for mixed-score questions.Questions with all-correct or all-incorrect outcomes, even after stronger-generator reasoning, are discarded as non-discriminative.Source: §3.2 · pp. 3–4
3.3 Reducing Early-Step Bias for PRM
- TD-based lookahead updates PRM scores for shallow reasoning steps, stopping when reward changes stabilize or the step limit is reached to balance bias reduction and computation.The approach is motivated by the variance introduced by conventional lookahead rollouts.Source: §3.3 · p. 5
3.4 Process Explanation Model
- The PEM generates refinement explanations, then off-policy preference learning uses PRM feedback to align those explanations with scores; KTO optimizes the binary preference data.The dynamic negative-example loss weight uses PRM scores to reflect the degree of loss aversion.Source: §3.4 · p. 5
3.5 Post-Training Scaling of ReARTeR
- Warm-up reasoning-step data precedes iterative MCTS collection of step-level preferences and KTO policy updates, and the resulting post-trained policy can be combined with test-time scaling.Compared with ReFT’s PPO approach [22], the method sacrifices some exploration but achieves more stable updates.Source: §3.5 · pp. 5–6
4 EXPERIMENTS
The experiments are organized around ReARTeR’s reasoning performance, test-time components, post-training iterations, and PEM–PRM alignment.
4 EXPERIMENTS
- The first research question asks how ReARTeR affects reasoning in closed-source and open-source RAG models.The section frames this as an evaluation of reasoning capabilities.Source: §4 · p. 6
- The second research question examines how ReARTeR’s components affect test-time scaling.The section names component effects as an experimental question.Source: §4 · p. 6
- The third research question asks how the number of post-training iterations affects performance.The section explicitly identifies iteration count as a variable of interest.Source: §4 · p. 6
- The fourth research question evaluates how effectively ReARTeR aligns the PEM and PRM.The section treats model alignment as a separate evaluation question.Source: §4 · p. 6
4.1 Experimental Settings
The experiments evaluate ReARTeR on five multi-step QA datasets using two generators and compare it with retrieval, reasoning, and test-time-scaling baselines. They specify answer evaluation, training data construction, and implementation settings.
4.1 Experimental Settings
- Experiments use HotpotQA, 2WikiMultiHopQA, Musique, Bamboogle, and StrategyQA, with Wikipedia passages as the retrieval corpus and dataset-specific test samples.The sampled test sets contain 500 examples for HotpotQA, 2WikiMultiHopQA, Musique, and StrategyQA, and all 125 Bamboogle test examples.Source: §4.1.1 · p. 6
4.1.1 Datasets.
- PRM and post-training data are constructed from 200 training examples per dataset, yielding 167,716 PRM examples, 769 PCM examples, 548 warm-up examples, and an average of 27,822 preference examples per iteration.The passage identifies these totals as outputs of the paper's PRM and PCM data-construction strategies.Source: §4.1.1 · p. 6
4.1.2 Evaluation Metrics.
- Accuracy is measured by whether the golden answer appears in the prediction or by GPT4-o judge assessment; ACC_R is used for reward feedback during PRM training and post-training.Exact match is unsuitable because reasoning-optimized systems often produce longer answers with supplementary information.Source: §4.1.2 · p. 6
4.1.3 Backbone and Baseline Models.
- Evaluation uses GPT4o-mini for test-time scaling and LLaMA3.1-8B for post-training and test-time scaling, comparing ReARTeR with retrieval, RAG-CoT, and test-time-scaling baselines.The baselines also include branching, summarization-based, adaptive-retrieval, and general reasoning-optimized models.Source: §4.1.3 · p. 6
4.1.4 Implementation Details.
- ReARTeR’s test-time reasoning-step setting is 3; CoT is capped at T=5, with the first 3 steps shallow and the rest deep, while iterative baselines and ReARTeR retrieve Top 1 document per step.The PRM is fine-tuned from skywork-reward-llama-3.1-8b-v0.2 with LoRA; the PEM is fine-tuned from Llama-3.2-3B-Instruct.Source: §4.1.4 · p. 7
4.2 RQ1: Overall Performance
Across five multi-step QA datasets and two generator setups, ReARTeR improves RAG reasoning performance and outperforms the compared retrieval and reasoning baselines. Its comparisons include both test-time-only use and post-training with test-time scaling.
4.2 RQ1: Overall Performance
- ReARTeR significantly improves RAG reasoning performance across closed- and open-source generator setups and five multi-step QA datasets.For GPT4o-mini, only test-time scaling is applied because fine-tuning is not feasible.Source: §4.2 · p. 7
- ReARTeR outperforms branching, summarization-based, and adaptive-retrieval methods on the reported multi-step QA comparisons.The authors relate these comparisons to multi-path CoT reasoning, CoT followed by summarization, and generator-directed retrieval, respectively.Source: §4.2 · pp. 7–8
- ReARTeR exceeds RAG-CoT methods and CR-Planner, which the authors attribute to its post-training and test-time scaling and Trustworthy Process Rewarding.The reported comparisons also include reasoning models optimized through general-domain training and test-time scaling.Source: §4.2 · p. 8
- ReARTeR surpasses the compared general reasoning models Marco-o1 and Skywork-o1 in the RAG reasoning scenarios reported.The authors conclude that those generally optimized models are less effective in these RAG-specific scenarios than their framework.Source: §4.2 · p. 8
4.3 RQ2: Ablation Study of ReARTeR
Ablations show that removing any tested component reduces ReARTeR's overall performance. The reported explanations connect reliable PRM rewards, TD-based lookahead, and PEM-generated explanations to stronger refinement feedback.
4.3 RQ2: Ablation Study of ReARTeR
- The ablations test removing refinement, replacing PEM explanations with PRM scores, and removing TD-based lookahead.These variants examine refinement, explanation-based feedback, and mitigation of PRM early-step bias.Source: §4.3 · p. 8
- Further ablations replace the proposed PRM data with traditional Monte Carlo data and disable beam search by sampling a single reasoning path.The data variant evaluates the quality of the PRM trained with the proposed collection method.Source: §4.3 · p. 8
- Removing any tested component negatively affects ReARTeR's overall performance, and PEM explanations improve refinement more than PRM scores alone.The authors also report that reliable PRM rewards combined with TD-based lookahead enhance feedback during refinement.Source: §4.3 · pp. 8–9
4.4 RQ3: Post-training iterations analysis.
With LLaMA-3.1-8B, performance improves significantly and consistently as offline-reinforcement iterations increase across four multi-step reasoning datasets. Scalability to larger datasets or more iterations was not verified because of resource constraints.
4.4 RQ3: Post-training iterations analysis.
- The iteration study runs three offline-reinforcement iterations and evaluates the system on four multi-step reasoning datasets.It analyzes the effect of Step-Level Offline Reinforcement during post-training scaling.Source: §4.4 · p. 9
- Performance improves significantly and stably across iterations on four multi-step reasoning datasets using LLaMA-3.1-8B.The authors describe these results as a promising scaling property, while noting that larger datasets and additional iterations were not tested due to resource constraints.Source: §4.4 · p. 9
- The study does not verify scaling to larger datasets or additional iterations because of resource constraints.Potential further improvements under resource-abundant conditions are presented as a possibility, not a tested result.Source: §4.4 · p. 9
4.5 RQ4: The effective of RARTPR in aligning PEM and PRM.
The section evaluates PEM–PRM alignment and illustrates the aligned PEM’s feedback with a reasoning-error case study. Alignment improves low-score-step refinement and multi-hop query accuracy across datasets.
4.5 RQ4: The effective of RARTPR in aligning PEM and PRM.
- The suggested fixes are to refine sub-question decomposition, improve retrieval, and check answer accuracy and completeness.The aligned PEM’s case study presents these as ways to improve the example reasoning process.Source: §4.5 · pp. 9–10
- PEM–PRM alignment increases low-score-step reward improvement and consistently improves multi-hop query accuracy across datasets.Figure 5 compares the alignment conditions; the passage gives no exact accuracy values.Source: Figure 5 · p. 9, §4.5 · p. 9
- The case study identifies errors in question decomposition, retrieval relevance, and answer relevance or completeness.Its example flags an unrelated follow-up sub-question, inadequate retrieved information, and an irrelevant or incomplete answer.Source: §4.5 · pp. 9–10
5 CONCLUSION
ReARTeR combines test-time process scoring and explanation-based refinement with MCTS-guided post-training. The conclusion reports that it outperforms existing methods on multi-step reasoning benchmarks.
5 CONCLUSION
- Experiments on multi-step reasoning benchmarks show that ReARTeR outperforms existing methods.Source: §5 · p. 10
- ReARTeR enhances RAG multi-step reasoning through both post-training and test-time scaling.Source: §5 · p. 10
- Its Trustworthy Process Rewarding combines a Process Reward Model for scoring with a Process Explanation Model for explanation-based refinement.Source: §5 · p. 10
- MCTS-guided step-level offline reinforcement fine-tuning generates preference data to optimize the generator.Source: §5 · p. 10
- Off-policy preference learning, balanced annotations, and temporal-difference look-ahead search address PEM–PRM misalignment and PRM data and early-step biases.Source: §5 · p. 10