PaperlayerView exact revision on arXiv

Source-linked AI summary

R-PRM: Reasoning-Driven Process Reward Modeling

Shuaijie She, Junxiao Liu, Yifeng Liu, Jiajun Chen, Xin Huang, Shujian Huang

arXiv:2503.21295v1cs.CL

TL;DR

LLMs make intermediate mathematical reasoning errors, while existing PRMs directly score steps and face scarce annotated data. R-PRM generates reasoning-based evaluations, bootstraps training from limited annotations, applies preference optimization, and scales inference across multiple analyses. It improves PRM benchmarks and policy-model mathematical reasoning across challenging datasets, with reported gains of 11.9 and 8.5 F1 points on ProcessBench and PRMBench.

  • Problem

    Existing PRMs directly output step evaluations, limiting learning efficiency and interpretability, while process-level annotations remain scarce and costly.

  • Method

    R-PRM uses stronger LLMs to generate seed reasoning data, preference optimization without additional annotations, and inference-time scaling over multiple evaluation processes.

  • Results

    R-PRM improves F1 by 11.9 points on ProcessBench and 8.5 points on PRMBench over the strongest same-data baseline, while improving guided mathematical reasoning across six challenging datasets.

  • Takeaways & Limitations

    R-PRM provides more comprehensive, robust, and generalizable process evaluation, with performance continuing to improve as inference reasoning budgets increase.

  • Takeaways & Limitations

    The approach has not been verified on larger models such as 70B, and advanced search algorithms including MCTS and Beam Search remain underexplored.

Abstract

from arXiv · show

Large language models (LLMs) inevitably make mistakes when performing step-by-step mathematical reasoning. Process Reward Models (PRMs) have emerged as a promising solution by evaluating each reasoning step. However, existing PRMs typically output evaluation scores directly, limiting both learning efficiency and evaluation accuracy, which is further exacerbated by the scarcity of annotated data. To address these issues, we propose Reasoning-Driven Process Reward Modeling (R-PRM). First, we leverage stronger LLMs to generate seed data from limited annotations, effectively bootstrapping our model's reasoning capabilities and enabling comprehensive step-by-step evaluation. Second, we further enhance performance through preference optimization, without requiring additional annotated data. Third, we introduce inference-time scaling to fully harness the model's reasoning potential. Extensive experiments demonstrate R-PRM's effectiveness: on ProcessBench and PRMBench, it surpasses strong baselines by 11.9 and 8.5 points in F1 scores, respectively. When applied to guide mathematical reasoning, R-PRM achieves consistent accuracy improvements of over 8.5 points across six challenging datasets. Further analysis reveals that R-PRM exhibits more comprehensive evaluation and stronger generalization capabilities, thereby highlighting its significant potential.

1 Introduction

LLMs can produce erroneous intermediate reasoning, while existing PRMs face costly data scarcity and limited interpretability from direct score prediction. R-PRM addresses these issues by generating reasoning-based evaluations, using limited annotations to bootstrap training, preference optimization, and inference-time scaling.

  • Motivation: LLMs’ intermediate reasoning errors can undermine solution reliability even when chain-of-thought reasoning improves mathematical problem solving.This motivates evaluating reasoning processes rather than only final answers.
  • Motivation: PRMs evaluate each reasoning step more finely than Outcome Reward Models, but human process-level annotation is expensive and automated alternatives can require substantial computation.The data-scarcity challenge constrains PRM development.
  • Motivation and Approach: R-PRM generates comprehensive reasoning analyses for each intermediate step instead of directly outputting evaluation scores.The framework uses stronger LLMs and limited human-annotated labels to construct seed data for training.
  • Method: R-PRM further improves its evaluation process through preference optimization without additional annotated data and inference-time scaling across multiple reasoning processes.The framework aims to improve learning efficiency, evaluation robustness, and coverage.
  • Results: 11.9 and 8.5 F1 points are gained over the strongest same-data baseline on ProcessBench and PRMBench, respectively.These results establish the main benchmark improvements reported for R-PRM.

2 Related Work

Related work frames mathematical reward modeling around improving reasoning data, learning, and search, while distinguishing outcome-level from process-level evaluation. PRMs offer finer-grained feedback but remain limited by scarce and costly annotations.

  • Mathematical Reasoning: Even correct final answers can contain critical intermediate errors that reduce process reliability and hinder future model improvements.This motivates process-level supervision beyond final-answer correctness.
  • Inference-Time Reasoning: Increasing inference-time computation through multiple reasoning attempts and majority voting has produced notable accuracy improvements.This line of work motivates scalable inference strategies for reasoning systems.
  • Reward Models: Reward models improve mathematical reasoning by enhancing training data quality, guiding model learning, and steering policy reasoning with Best-of-N and Guided-Search methods.These roles connect reward modeling to both training and inference.
  • Reward Models: PRMs evaluate each reasoning step, whereas ORMs assess only whether the final answer is correct.Prior work reports stronger results for PRMs, but their process-level data is extremely scarce and costly to annotate.

3 Method

R-PRM replaces direct step scoring with generated analyses and yes/no judgments, trains this behavior from stronger-LLM seed data with preference optimization, and aggregates multiple inference trajectories for robust rewards.

  • 3.1 Reasoning for Process Reward Modeling: The framework illustration distinguishes initial models from models after training while showing partial analytical reasoning trajectories.White robots denote initial models; colored robots denote trained models.
  • 3.1 Reasoning for Process Reward Modeling: R-PRM analyzes each reasoning step across history, objectives, data sources, coherence, and computational transformations before producing a yes/no correctness judgment.The analysis and judgment are generated in one process.
  • 3.1 Reasoning for Process Reward Modeling: Stronger-LLM-generated tuples are filtered by agreement with human labels, then used as target sequences for supervised fine-tuning.The target combines the analytical reasoning process and its score.
  • 3.2 Preference Optimization: Preference optimization treats label-consistent evaluation trajectories as preferred and inconsistent trajectories as disfavored.DPO is used to encourage reasoning processes that yield correct judgments.
  • 3.3 Inference Time Scaling Strategy: At inference time, R-PRM samples K independent analytical processes for each step and averages the probability of yes judgments as the reward.Multiple trajectories are intended to reduce stochastic inconsistencies in LLM reasoning.

4 Experiment

R-PRM is evaluated on process-level benchmarks and as a reward model for inference-time policy guidance. It improves benchmark evaluation, multi-dimensional assessment, out-of-domain generalization, and guided mathematical reasoning.

  • Preference Optimization: DPO raises performance to 70.4 and 66.8 on the two benchmarks and surpasses Qwen2.5-Math-PRM trained with 1.8M data points on PRMBench.The result supports preference optimization within the generative evaluation paradigm without introducing additional annotation claims beyond the supplied evidence.
  • Multi-dimensional Evaluation: R-PRM improves three PRMBench evaluation dimensions by 7.0, 9.0, and 4.4 points over Qwen2.5-Math-7B-PRM800K and surpasses GPT-4 in completeness and sensitivity.The reported dimensions characterize more comprehensive process evaluation.
  • Error Detection: Progressive contextual analysis checks each step against preceding steps, supporting detection of superficially valid but logically flawed reasoning and stronger deception resistance than o1-mini.The soundness analysis covers empirical validity, step consistency, and domain alignment.
  • Generalization: R-PRM achieves F1 scores above 60 on all reported out-of-domain datasets, whereas most comparison models score between 10 and 40 on such datasets.Qwen2.5-Math-7B-PRM800K is the stated exception, reaching 50.7 F1 on OlympiadBench.
  • Policy Guidance: 8.4% and 8.6% average accuracy improvements over Pass@1 are achieved with Guide Search and Best-of-N, respectively, across six challenging math datasets.The evaluation uses Qwen2.5-7B-Instruct and compares PRM-guided inference strategies with policy-model baselines.

5 Analysis

The analysis examines how R-PRM responds to training-data scale, inference-time compute, threshold variation, and challenging reasoning errors. Across these settings, it shows stronger scaling behavior, robustness, and step-level error analysis.

  • 5.1 Effective Data Scaling: At 64k training samples, R-PRM surpasses Qwen2.5-Math-7B-PRM800K, trained on 265k samples, by 1.6 F1 points.With only 12.8k samples, R-PRM already reaches an F1 score of 52.6.
  • 5.1 Effective Data Scaling: Preference optimization raises performance from 65.2 to 70.4 using 269K existing preference pairs without additional labeled data.This demonstrates improved data utilization without collecting further annotations.
  • 5.2 Inference-Time-Scaling: F1 improves from 62.8 to 67.6 when evaluation trajectories increase from 2 to 4 on ProcessBench.Increasing trajectories also consistently improves performance across all four datasets.
  • 5.3 Threshold Robustness in Model Evaluation: R-PRM maintains minimal F1 fluctuations across thresholds on ProcessBench, whereas Qwen2.5-Math-7B-PRM800K shows greater threshold sensitivity.On the out-of-domain OlympiadBench subset, R-PRM remains robust while the baseline exhibits a larger threshold shift associated with misclassifying incorrect solutions as correct.
  • 5.4 Case Study: R-PRM detects an omitted verification of number 19 by analyzing adjacent steps, assigning Step 8 a reward score of 0.05.Qwen2.5-Math-PRM and Qwen2.5-Math-7B-PRM800K instead assign Step 8 scores of 0.99 and 0.86.

6 Conclusion

R-PRM combines stronger-LLM seed-data construction, preference optimization, and inference-time scaling to improve process reward modeling. Experiments show gains on ProcessBench and PRMBench, effective guidance of mathematical reasoning, and increasingly comprehensive, robust, and generalizable evaluation with more inference.

  • 6 Conclusion: R-PRM combines stronger-LLM seed data, preference optimization without additional annotations, and inference-time scaling.These components are intended to strengthen comprehensive process evaluation and exploit the model’s reasoning capabilities.
  • 6 Conclusion: R-PRM improves ProcessBench and PRMBench performance while effectively guiding language-model reasoning.Its evaluation capabilities become more comprehensive, robust, and generalizable as inference increases.

Limitations

The evaluation is limited to 7B models because computational constraints prevented verification on larger models such as 70B.

  • Limitations: Computational constraints limited verification to 7B models, leaving performance on larger models such as 70B untested.The authors hypothesize that larger models could achieve higher modeling accuracy with this methodology.

A Detailed Description of PRMBench Subcategories

PRMBench evaluates multiple dimensions of process reasoning beyond basic step correctness. Its subcategories cover efficiency, logical validity, consistency, domain use, confidence, prerequisites, deception, and multiple solutions.

  • PRMBench Subcategories: Non-Redundancy evaluates whether unnecessary reasoning steps can be identified and eliminated without sacrificing correctness.
  • PRMBench Subcategories: Non-Circular Logic assesses detection of reasoning loops in which conclusions are reintroduced as premises.
  • PRMBench Subcategories: Empirical Soundness measures rejection of steps that contradict established facts or real-world knowledge.
  • PRMBench Subcategories: Step Consistency and Domain Consistency assess logical flow across steps and correct application of domain-specific knowledge.
  • PRMBench Subcategories: Confidence Invariance, Prerequisite Sensitivity, Deception Resistance, and Multi-Solution Consistency test calibrated confidence, missing conditions, misleading information, and consistency across valid solution paths.

B Threshold Robustness on PRMBench

R-PRM shows stronger threshold robustness on PRMBench, while Qwen2.5-Math-7B-PRM800K is more sensitive to threshold choice.

  • An 8.2-point performance gap separates Qwen2.5-Math-7B-PRM800K at the 0.5 threshold from its optimal performance, whereas R-PRM shows significant robustness advantages.The comparison is reported for PRMBench threshold experiments.

C Additional Case

A MATH case study shows R-PRM detecting and explaining a calculation error that two comparison models miss with high confidence.

  • R-PRM checks the calculation, explains the error, and assigns a score around 0.3, while both comparison models assign correctness scores around 0.9.The case concerns an error step in a MATH-subset ProcessBench example.
  • The work uses Transformers and trl as software artifacts for downloading, training, and reinforcement-learning-based language-model development.Both libraries are identified as Apache-2.0-licensed artifacts.
  • PRM800K provides 800,000 step-level correctness labels, while PRMBench contains 6,216 fine-grained instances for evaluating error-type identification.These artifacts support training and evaluation of process reward models.

E Prompt for Construction Data

The construction-data prompt presents an LCM problem and a worked solution whose final answer is t = 48.

  • The construction prompt asks for the smallest positive integer t satisfying lcm[12, t]3 = (12t)2.The same problem statement appears in both the prompt and its solution setup.
  • The worked solution begins by taking cube roots and simplifying the equation to lcm[12, t] = 12t2/3.The passage presents this as Step 1 of the solution.
  • The solution concludes that t = 48 satisfies the original equation and is the smallest possible value.The construction-data material is associated with a prompt-construction table.
Loading 2503.21295v1…