PaperlayerView exact revision on arXiv

Source-linked AI summary

R-PRM: Reasoning-Driven Process Reward Modeling

Shuaijie She, Junxiao Liu, Yifeng Liu, Jiajun Chen, Xin Huang, Shujian Huang

arXiv:2503.21295v1cs.CL

TL;DR

Existing process reward models directly score steps, constraining learning efficiency and failing to explain why steps are incorrect. R-PRM reasons over intermediate steps and reports 11.9- and 8.5-point F1 gains on ProcessBench and PRMBench over the strongest same-data baseline.

  • Problem

    Existing process reward models directly evaluate steps, constraining learning efficiency and failing to explain errors needed for constructive feedback.

    Source: §1 · p. 1

  • Method

    R-PRM uses stronger LLMs to create seed data, preference optimization without additional annotations, and inference-time scaling, while reasoning over each intermediate step to judge correctness.

    Source: §1 · p. 1, §6 · p. 9

  • Results

    11.9- and 8.5-point F1 gains over the strongest same-data baseline on ProcessBench and PRMBench accompany accuracy improvements across six challenging math datasets.

    Source: §1 · p. 2

  • Takeaways & Limitations

    Analysis reports comprehensive evaluation coverage, generalization across diverse datasets, and accuracy that rises with larger reasoning budgets.

    Source: §1 · p. 2

    Experiments demonstrate effectiveness on 7B models, but the approach has not yet been verified on larger models.

    Source: Limitations · p. 9

Abstract

from arXiv · show

Large language models (LLMs) inevitably make mistakes when performing step-by-step mathematical reasoning. Process Reward Models (PRMs) have emerged as a promising solution by evaluating each reasoning step. However, existing PRMs typically output evaluation scores directly, limiting both learning efficiency and evaluation accuracy, which is further exacerbated by the scarcity of annotated data. To address these issues, we propose Reasoning-Driven Process Reward Modeling (R-PRM). First, we leverage stronger LLMs to generate seed data from limited annotations, effectively bootstrapping our model's reasoning capabilities and enabling comprehensive step-by-step evaluation. Second, we further enhance performance through preference optimization, without requiring additional annotated data. Third, we introduce inference-time scaling to fully harness the model's reasoning potential. Extensive experiments demonstrate R-PRM's effectiveness: on ProcessBench and PRMBench, it surpasses strong baselines by 11.9 and 8.5 points in F1 scores, respectively. When applied to guide mathematical reasoning, R-PRM achieves consistent accuracy improvements of over 8.5 points across six challenging datasets. Further analysis reveals that R-PRM exhibits more comprehensive evaluation and stronger generalization capabilities, thereby highlighting its significant potential.

1 Introduction

R-PRM uses reasoning over intermediate steps, seed-data generation, preference optimization, and inference-time sampling to address process-evaluation limitations. It reports gains on process benchmarks and when guiding mathematical reasoning.

1 Introduction

  • LLMs still make reasoning errors; unlike outcome-only models, PRMs assess steps, but direct scoring can constrain learning efficiency and obscure why errors occur (Lightman et al., 2023).The errors undermine solution reliability; stepwise evaluation is presented as a way to identify and mitigate process errors.Source: §1 · p. 1
  • Human process-label annotation is costly, while Monte Carlo correctness estimation (Wang et al., 2024b,a; Luo et al., 2024b) is resource-intensive and stronger-LLM filtering (Zhang et al., 2025) can remain noisy or biased.These approaches leave the challenge of sufficient high-quality training data unresolved.Source: §1 · p. 1
  • R-PRM bootstraps stepwise evaluation from stronger-LLM-generated seeds based on limited human labels, preference-optimizes without additional data generation, and samples multiple evaluations at inference time.The seed data is used to finetune Qwen2.5-Math-7B-Instruct for comprehensive, transparent analyses; multiple sampled processes provide a more comprehensive and robust assessment perspective.Source: §1 · pp. 1–2
  • 11.9 and 8.5 points of F1 score improvement over the strongest baseline trained on the same data on ProcessBench and PRMBench, respectively.Further analysis reports comprehensive multi-dimension evaluation coverage, stronger generalization across diverse datasets, and progressive accuracy improvement with increased reasoning budgets.Source: §1 · p. 2
  • 8.6 and 8.4 points of average accuracy improvement over the Pass@1 baseline with Best-of-N and Guided Search, respectively, across six challenging math datasets.The approach also outperforms majority voting and existing PRM baselines.Source: §1 · p. 2

2 Related Work

Prior work uses step-by-step reasoning and reward models to improve mathematical problem solving, but intermediate-step errors and scarce, costly PRM annotations remain challenges (Wei et al., 2023; Lightman et al., 2023).

2.1 Mathematical Reasoning

2.2 Reward Modeling of Reasoning

3 Method

R-PRM trains a model to analyze each reasoning step before judging it, optimizes its evaluation process using label-consistent preferences, and improves robustness at inference by aggregating multiple analytical trajectories.

3.1 Reasoning for Process Reward Modeling

  • R-PRM generates multidimensional analyses of each step before a yes/no judgment, training on stronger-LLM PRM800K examples retained for human-label consistency and predicting both in one pass.Source: §3.1 · p. 3

3.2 Process Reward Modeling Meta-Optimization

  • DPO improves evaluation reasoning by preferring sampled trajectories whose final predictions match annotations over inconsistent trajectories, using the SFT model as a frozen reference policy.Source: §3.2 · p. 3

3.3 Inference Time Scaling Strategy

  • At inference, R-PRM samples K independent analytical processes per step and averages their probability of a yes judgment to mitigate reasoning inconsistencies and stochastic variation.Source: §3.3 · p. 4

4 Experiment

R-PRM improves process-reward evaluation on ProcessBench and PRMBench, and its scores can guide policy-model reasoning to higher accuracy. Its results also indicate stronger evaluation coverage and generalization.

4.1 Experiment Settings

  • The experiments assess step-error identification on ProcessBench’s 3,400 annotated cases, granular PRM evaluation across PRMBench’s three dimensions, and policy guidance with Best-of-N and Greedy Guide Search.Policy-model tests include six math benchmarks; Qwen2.5-7B-Instruct supplies eight candidate steps at temperature 1.Source: §4.1 · p. 4

4.2 Experiment Results

  • R-PRM’s capabilities were enhanced through DPO, reaching scores of 70.4 and 66.8 on two benchmarks and surpassing Qwen2.5-Math-PRM, trained with 1.8M data points, on PRMBench.Source: §4.2 · pp. 5–6
  • F1 scores of 65.2 on ProcessBench and 64.9 on PRMBench make R-PRM-SFT stronger than state-of-the-art baselines.Against Qwen2.5-Math-7B-PRM800K, it gains 6.7 and 6.6 F1 points on ProcessBench and PRMBench; on ProcessBench, it gains 7.2 points over LLaMA3.3-70B-Instruct and 3.3 over GPT-4o.Source: Table 1 · p. 5, Table 2 · p. 5, §4.2 · pp. 5–6
  • R-PRM raises average accuracy over Pass@1 by 8.4% in Guide Search and 8.6% in Best-of-N, outperforming Qwen2.5-MATH-PRM and Majority Voting.Its gains over Qwen2.5-MATH-PRM are 2.6 points in Guide Search and 1.6 points in Best-of-N.Source: Table 3 · p. 6, §4.2 · p. 6, Table 4 · p. 7
  • R-PRM-DPO improves over Qwen2.5-Math-7B-PRM800K by 7.0, 9.0, and 4.4 points across PRMBench’s three core dimensions.It also surpasses GPT-4 in completeness and sensitivity; the authors connect its soundness to empirical validity, step consistency, domain alignment, and contextual checks for logical errors, with particular deception-resistance advantage over o1-mini.Source: §4.2 · p. 6
  • R-PRM achieves F1 scores above 60 on all out-of-domain datasets, while most listed open-source PRMs score below 50 on OlympiadBench.The passage notes Qwen2.5-Math-7B-PRM800K as an exception at 50.7 on OlympiadBench and suggests R-PRM learned a dataset-independent reasoning pattern.Source: §4.2 · p. 6

5 Analysis

The analysis finds that R-PRM benefits from more training data, preference optimization, and increased inference-time compute, while remaining robust to threshold changes and detecting an error missed by two baselines.

5.1 Effective Data Scaling

  • On ProcessBench, increasing training data to 285k samples yields F1 65.2; at 64k samples, R-PRM outperforms Qwen2.5-Math-7B-PRM800K, trained on 265k samples, by 1.6 points.With 12.8k training samples, R-PRM already achieves F1 52.6 and surpasses most existing open-source PRMs.Source: §5.1 · p. 7, Figure 2 · p. 7
  • Using 269K preference pairs without additional labeled data, R-PRM improves from F1 65.2 to 70.4.The 65.2 score is the result before preference optimization.Source: §5.1 · p. 7

5.2 Inference-Time-Scaling

  • On ProcessBench, scaling evaluation trajectories from 2 to 4 raises F1 from 62.8 to 67.6, with gains as trajectories increase across all four datasets.The paper reports this as consistent performance improvement across the datasets.Source: §5.2 · p. 7, Figure 3 · p. 7

5.3 Threshold Robustness in Model Evaluation

  • R-PRM shows minimal performance fluctuations across thresholds on ProcessBench and remains robust on OlympiadBench’s out-of-domain subset, where Qwen2.5-Math-7B-PRM800K exhibits a larger threshold shift.The paper interprets that shift as a tendency to misclassify incorrect solutions as correct.Source: §5.3 · p. 8

5.4 Case Study

  • In a ProcessBench MATH case, R-PRM detects the skipped verification of 19 at Step 7 and scores Step 8 at 0.05, while Qwen2.5-Math-PRM and Qwen2.5-Math-7B-PRM800K assign 0.99 and 0.86.R-PRM reasons that the numbers must be verified in ascending order, then resumes Step 7 to identify the discrepancy.Source: §5.4 · p. 8, Figure 5 · p. 8

6 Conclusion

R-PRM combines stronger-LLM-generated seed data, preference optimization without additional annotations, and inference-time scaling. Experiments report improved benchmark performance and effective guidance of LLM reasoning, while further analysis finds comprehensive, robust, and generalizable evaluation that improves with increased inference.

6 Conclusion

  • R-PRM uses stronger LLMs to construct seed data, preference optimization without additional annotated data, and inference-time scaling to harness its reasoning capabilities.The seed data enables comprehensive evaluation.Source: §6 · p. 9
  • Experiments report significant improvements on ProcessBench and PRMBench, and show that R-PRM effectively guides LLM reasoning.Source: §6 · p. 9
  • Further analysis finds R-PRM’s evaluation capabilities comprehensive, robust, and generalizable, with performance continuing to improve as inference increases.Source: §6 · p. 9

Limitations

R-PRM’s evaluation is limited to 7B models, and its exploration of advanced inference-time search methods remains incomplete.

Limitations

  • Due to computational constraints, the approach has not been verified on models larger than 7B, including 70B models.The authors hypothesize that larger models’ stronger reasoning could improve modeling accuracy with their methodology.Source: Limitations · p. 9
  • Although Best-of-N and Guided Search were tested, advanced methods such as MCTS and Beam Search remain underexplored.The authors suggest these methods could better leverage PRM characteristics and improve generation results.Source: Limitations · p. 9

A Detailed Description of PRMBench Subcategories

PRMBench evaluates reasoning across efficiency, logical and factual validity, consistency, confidence, prerequisite completeness, deception resistance, and handling of multiple valid solutions.

A Detailed Description of PRMBench Subcategories

  • Non-Redundancy measures removal of unnecessary steps without sacrificing correctness, while Non-Circular Logic checks for conclusions reused as premises.These categories assess efficiency and detection of logical loops.Source: A Detailed Description of PRMBench… · p. 10
  • Empirical Soundness rejects steps contradicting established facts, while Deception Resistance rejects misleading information containing subtle errors.The former concerns real-world factual consistency; the latter concerns deceptively plausible errors.Source: A Detailed Description of PRMBench… · p. 10
  • Step Consistency checks logical flow between steps, while Confidence Invariance tests appropriate confidence under detected errors or uncertainty and Prerequisite Sensitivity detects missing conditions.Together, these categories examine consistency and completeness within a reasoning process.Source: A Detailed Description of PRMBench… · p. 10
  • Domain Consistency checks correct use of domain-specific knowledge, while Multi-Solution Consistency checks agreement across different valid reasoning paths.These categories address domain boundaries and problems with multiple valid solutions.Source: A Detailed Description of PRMBench… · pp. 10–11

B Threshold Robustness on PRMBench

On PRMBench, R-PRM shows significant threshold robustness advantages; Qwen2.5-Math-7B-PRM800K has an 8.2-point gap between its score at threshold 0.5 and its optimal performance.

B Threshold Robustness on PRMBench

C Additional Case · D Used Scientific Artifacts

An additional case shows R-PRM identifying a calculation error missed by two PRM baselines. The paper also lists the scientific artifacts used and their intended-use ethics statement.

C Additional Case

  • R-PRM explains the step, checks the calculation, and finds the error with a final score around 0.3, while both baselines miss it with correctness scores around 0.9.Source: C Additional Case · p. 11

D Used Scientific Artifacts

E Prompt for Construction Data

The section identifies the prompt for constructing data and includes a ProcessBench MATH case study of error-step evaluation.

E Prompt for Construction Data

  • Figure 7 presents a ProcessBench MATH case study, with blue text highlighting our model’s critique and score for the error step.Red text denotes the error step and other models’ scores.Source: Figure 7 · p. 12
  • Table 5 is identified as the prompt for constructing data.The supplied passage gives the table title but not the prompt contents.Source: Table 5 · p. 13

Related papers

Loading 2503.21295v1…