The Lessons of Developing Process Reward Models in Mathematical Reasoning

Zhenru Zhang; Chujie Zheng; Yangzhen Wu; Beichen Zhang; Runji Lin; Bowen Yu; Dayiheng Liu; Jingren Zhou; Junyang Lin

Source-linked AI summary

The Lessons of Developing Process Reward Models in Mathematical Reasoning

Zhenru Zhang, Chujie Zheng, Yangzhen Wu, Beichen Zhang, Runji Lin, Bowen Yu, Dayiheng Liu, Jingren Zhou, Junyang Lin

arXiv:2501.07301v2Jan 13, 2025cs.CLcs.AIcs.LG

TL;DR

PRM development for mathematical reasoning is limited by costly process annotation and evaluation methods that may not faithfully measure intermediate-step correctness. The paper combines MC estimation with LLM-as-a-judge through consensus filtering and evaluates PRMs at both response and step levels. Experiments report improved performance and data efficiency, while remaining gaps include the BoN upper bound, reinforcement-learning practices, and use of human annotations.

Problem
PRM development faces significant challenges in annotating reasoning-process correctness and evaluating process verification.
Method
The paper uses consensus filtering to retain data only when MC estimation and LLM-as-a-judge agree on erroneous-step locations, alongside response-level BoN and step-level evaluation.
Results
Consensus filtering improves performance and data efficiency, while the trained PRMs achieve stronger erroneous-step identification than other open-source models.
Takeaways & Limitations
Combining MC estimation with LLM-as-a-judge and supplementing BoN with step-level evaluation provides practical guidance for PRM training and assessment.
Takeaways & Limitations
A considerable performance gap remains between the PRMs and the BoN upper bound, and reinforcement-learning best practices remain unexplored.

Abstract

from arXiv · show

Process Reward Models (PRMs) emerge as a promising approach for process supervision in mathematical reasoning of Large Language Models (LLMs), which aim to identify and mitigate intermediate errors in the reasoning processes. However, the development of effective PRMs faces significant challenges, particularly in data annotation and evaluation methodologies. In this paper, through extensive experiments, we demonstrate that commonly used Monte Carlo (MC) estimation-based data synthesis for PRMs typically yields inferior performance and generalization compared to LLM-as-a-judge and human annotation methods. MC estimation relies on completion models to evaluate current-step correctness, leading to inaccurate step verification. Furthermore, we identify potential biases in conventional Best-of-N (BoN) evaluation strategies for PRMs: (1) The unreliable policy models generate responses with correct answers but flawed processes, leading to a misalignment between the evaluation criteria of BoN and the PRM objectives of process verification. (2) The tolerance of PRMs of such responses leads to inflated BoN scores. (3) Existing PRMs have a significant proportion of minimum scores concentrated on the final answer steps, revealing the shift from process to outcome-based assessment in BoN Optimized PRMs. To address these challenges, we develop a consensus filtering mechanism that effectively integrates MC estimation with LLM-as-a-judge and advocates a more comprehensive evaluation framework that combines response-level and step-level metrics. Based on the mechanisms, we significantly improve both model performance and data efficiency in the BoN evaluation and the step-wise error identification task. Finally, we release a new state-of-the-art PRM that outperforms existing open-source alternatives and provides practical guidelines for future research in building process supervision models.

1 Introduction

PRMs target unreliable intermediate reasoning in mathematical LLM outputs, but their development is constrained by costly annotation and weaknesses in MC-based supervision and BoN evaluation. The paper identifies these problems and proposes consensus filtering plus combined response- and step-level evaluation.

LLMs can reach correct final answers while relying on flawed calculations or fabricated reasoning steps, undermining process reliability.
PRM data annotation is expensive and time-consuming, motivating automated process-correctness assessment methods such as Monte Carlo estimation.
MC-estimated PRMs show inferior performance and generalization because completion models can produce correct answers from incorrect steps or incorrect answers from correct steps.
Conventional BoN evaluation can misalign response selection with process verification when policy models produce correct answers through flawed reasoning.
The paper proposes consensus filtering that combines MC estimation with LLM-as-a-judge and advocates combining response-level and step-level evaluation metrics.

2 Preliminary Trials

The preliminary trials compare MC-based PRM training with human-annotated supervision and evaluate both downstream response selection and erroneous-step identification. MC-based models provide no clear advantage and perform worse on process-error localization despite larger training data.

MC estimation-based PRMs do not show noticeable advantages over human-annotated PRMs and lag significantly in identifying specific erroneous reasoning steps.
The evaluation measures both downstream utility and the ability to identify specific erroneous reasoning steps.
Evaluation Setup: BoN selects the highest-scored response from N candidates according to a PRM; here N = 8, with maj@8 as a baseline and pass@8 as an upper bound.
Evaluation Setup: PROCESSBENCH requires models to identify the first erroneous reasoning step or conclude that all steps are correct.
Evaluation Results: None of the evaluated PRMs exceeds maj@8 in Best-of-8, while MC-hard and MC-soft models localize erroneous steps significantly worse than the human-annotated PRM800K baseline.

3 The lessons

The experiments expose weaknesses in MC-based PRM training and response-level Best-of-N evaluation, then show that consensus filtering and step-level assessment improve process supervision. They also find that hard labels outperform soft labels after filtering and that optimal BoN scoring depends on the PRM’s training data.

MC estimation limitations: MC estimation produces inferior performance and generalization compared with LLM-as-a-judge and human annotation, due to noisy step-correctness estimates and inaccurate error localization.Its dependence on future outcomes and policy-model behavior can misclassify deterministic current-step correctness.
Consensus filtering: Consensus filtering retains only samples where MC estimation and LLM-as-a-judge agree on error locations, preserving approximately 40% of the original data.The reduced dataset significantly outperforms MC estimation on PROCESSBENCH and reaches comparable performance to LLM-as-a-judge with 40% of the data.
Hard versus soft labels: Hard labels substantially outperform soft labels on both Best-of-8 and PROCESSBENCH after data filtering.Before filtering, high noise makes the performance difference between label types insignificant.
BoN evaluation bias: BoN scores can be inflated when PRMs assign high scores to responses with correct answers but flawed reasoning processes.MC-trained PRMs show limited process-verification capability despite inflated BoN results on such cases.
BoN evaluation bias: Optimizing solely for BoN shifts PRMs from process-based assessment toward outcome-oriented behavior, with final-step scores often determining selection.Several open-source PRMs place more than 40% of minimum scores at final answer steps, motivating supplementary step-level evaluation.
Scoring strategies: The last-score strategy is more effective for MC-trained PRMs, whereas product and minimum scoring are more appropriate for LLM-as-a-judge and human-annotated PRMs.Thus, the preferred BoN scoring strategy varies with the PRM’s annotation method.

4 Our PRMs

This section describes the construction and evaluation of the paper’s PRMs, using consensus-filtered training data and both response-level and step-level tests. The resulting models outperform comparable PRMs in Best-of-N evaluation and show strong process-error identification.

Method: The PRM data pipeline expands MC-estimated examples, filters disagreements with Qwen2.5-Instruct-72B as an LLM judge, and trains binary step classifiers.Only instances with consensus between MC and LLM labels are retained; models are trained at 7B and 72B scales.
Evaluation: Evaluation combines response-level Best-of-N with the step-level PROCESSBENCH process-error identification task.The experiments validate both answer selection and intermediate-error detection.
Best-of-N Results: 1.4% average improvement: Qwen2.5-Math-PRM-7B outperforms maj@8 across all 7 tasks in the reported Best-of-8 evaluation.It also surpasses other PRMs at equivalent model scale.
Best-of-N Results: Qwen2.5-Math-PRM-72B performs slightly better overall than Qwen2.5-Math-RM-72B, with particularly significant gains on Minerva Math and MMLU STEM.The comparison concerns the reported response-level evaluation.
PROCESSBENCH Results: On PROCESSBENCH, both trained PRMs substantially outperform existing PRMs, while the 7B model surpasses all open-source models and GPT-4o-0806 but trails o1-mini.The 72B ORM also shows considerable step-error identification capability.

5 Related Work

Prior work distinguishes outcome-level scoring from process-level verification and highlights a trade-off between human annotation quality and cost. Automated step evaluation provides an alternative but follows multiple methodological approaches.

Reward Models: Outcome Reward Models score complete solutions, whereas Process Reward Models evaluate individual reasoning steps.PRMs are intended to assess the reasoning process rather than only the final answer.
Step Verification: Human annotation can produce high-quality step-verification data but requires substantial cost and effort.This motivates research into automated evaluation methods.
Step Verification: Automated reasoning-step evaluation is presented as a major research direction alongside human annotation.The paper organizes existing automated methods into distinct methodological categories.

6 Conclusion

The paper concludes that MC-based PRM data construction and vanilla BoN evaluation have important weaknesses, and proposes consensus filtering plus complementary evaluation to improve PRM development. It also identifies remaining optimization and data-utilization gaps.

Conclusion: Extensive experiments show that MC-based data construction performs worse and generalizes less effectively than LLM-as-a-judge and human annotation.The conclusion frames this as a central lesson for PRM development.
Conclusion: Vanilla response-level BoN can inaccurately assess PRMs and bias optimization toward outcome-oriented rather than process-oriented verification.The paper therefore evaluates both response-level BoN and step-level PROCESSBENCH.
Conclusion: Consensus filtering combines MC estimation with LLM-as-a-judge and significantly improves data efficiency and model performance.The strategy retains examples where both methods agree on erroneous-step locations.
Limitations: A considerable gap remains between the PRMs and the BoN upper bound pass@8, indicating substantial optimization potential.Best practices for using PRMs in reinforcement learning also remain unexplored.
Limitations: Efficiently using existing high-quality human annotations remains largely underexplored, including gradual dataset expansion through weak supervision.The paper presents this as a future research direction.

A PRM Guided Search

The PRM-guided search procedure expands multiple candidate steps, scores them with a PRM, and greedily selects the highest-scoring candidate. The paper cautions that local step correctness may not yield the best final answer.

Search Procedure: At each step, greedy search generates 8 candidates, scores them with a PRM, and selects the highest-scoring candidate for subsequent expansion.The comparison includes an ORM in the Best-of-8 setting.
Search Limitation: PRM scores represent current-step correctness, but locally optimal choices may not lead to the correct final answer.The paper contrasts rewards with value models that estimate future success probability.
Future Directions: Depth-First Search with backtracking or score-constrained search may be more suitable than purely greedy selection.The authors present these approaches as future possibilities rather than established results.

B.1 The BoN Evaluation on Qwen2.5-Math-72b-Instruct

On Qwen2.5-Math-72B-Instruct, the 72B PRM surpasses maj@8 under prm@8, while the 7B PRM remains below maj@8.

Qwen2.5-Math-PRM-72B surpasses maj@8 in prm@8 and is comparable with Qwen2.5-Math-RM-72B in orm@8.
Qwen2.5-Math-7B-PRM outperforms other PRMs of equivalent model scale but remains inferior to maj@8.This indicates challenges when using the 7B PRM to supervise responses generated by the 72B policy model.

B.2 The BoN Evaluation with Various Scoring Strategies

The experiments compare last-step, minimum-step, and product-based aggregations for converting step scores into solution-level scores in BoN evaluation.

Three scoring strategies—last, product, and minimum—are used to derive solution-level scores from step scores.
The scoring-strategy comparisons are conducted with Qwen2.5-Math-7B-Instruct and Qwen2.5-Math-72B-Instruct policy models.Results are reported in Table 13 and Table 14, respectively.

B.3 The BoN Evaluation on Chinese Benchmarks

BoN evaluation on Chinese benchmarks uses Qwen2.5-Math-7B-Instruct and Qwen2.5-Math-72B-Instruct policy models, with three scoring strategies.

The Chinese-benchmark evaluation covers CMATH, GaoKao Math Cloze, and GaoKao Math QA.

B.4 BoN with Larger N Values

Additional BoN experiments test larger candidate sets across diverse mathematical tasks and compare PRM scoring and LLM-as-a-judge procedures.

The PRMs maintain superior performance compared with other PRMs across MATH500, AIME24, AMC23, Minerva Math, GaoKao 2023 En, and OlympiadBench.
Best-of-64 evaluation is reported for Qwen2.5-Math-7B-Instruct.The trained models are marked with a star in Table 10.
LLM-as-a-judge BoN evaluation uses pairwise single-elimination tournaments or pointwise scoring of each response.Pairwise comparison uses N−1 comparisons for N responses; pointwise selection uses the highest proportion of correct steps.
The LLM-as-a-judge comparisons are conducted with Qwen2.5-Math-7B-Instruct and Qwen2.5-Math-72B-Instruct policy models.Results are listed in Tables 11 and 12, respectively.
The LLM-as-a-judge prompt reviews solution paragraphs sequentially and stops after detecting an error, returning an Incorrect conclusion.The prompt requests verification, detailed critique, and correction guidance for the detected error.
Additional BoN comparisons use last, product, and minimum scoring strategies for both policy-model scales and Chinese benchmarks.These comparisons are reported in Tables 13–16.