Source-linked AI summary
GenPRM: Scaling Test-Time Compute of Process Reward Models via Generative Reasoning
Jian Zhao, Runze Liu, Kaiyan Zhang, Zhimu Zhou, Junqi Gao, Dong Li, Jiafei Lyu, Zhouyi Qian, Biqing Qi, Xiu Li, Bowen Zhou
TL;DR
Existing process reward models have limited supervision and generalization, rely on scalar scoring, and cannot scale test-time computation. GenPRM uses generative chain-of-thought reasoning, code verification, and relative progress estimation, outperforming prior PRMs across ProcessBench and mathematical reasoning tasks while benefiting from test-time scaling.
Problem
Existing process reward models have limited supervision and cross-task generalization, rely on scalar classification, and cannot leverage test-time scaling.
Method
GenPRM reframes process supervision generatively through explicit chain-of-thought reasoning, code verification, Relative Progress Estimation, and synthesized rationale data.
Results
GenPRM outperforms prior PRMs across ProcessBench and mathematical reasoning tasks, with test-time scaling enabling smaller models to surpass larger PRMs.
Takeaways & Limitations
GenPRM supports scalable process supervision and effective critic-based policy refinement beyond conventional PRM scoring.
Takeaways & Limitations
Generative reasoning adds inference computation, and GenPRM is evaluated mainly on mathematical reasoning rather than coding or general reasoning tasks.
Abstract
from arXiv · showhide
Recent advancements in Large Language Models (LLMs) have shown that it is promising to utilize Process Reward Models (PRMs) as verifiers to enhance the performance of LLMs. However, current PRMs face three key challenges: (1) limited process supervision and generalization capabilities, (2) dependence on scalar value prediction without leveraging the generative abilities of LLMs, and (3) inability to scale the test-time compute of PRMs. In this work, we introduce GenPRM, a generative process reward model that performs explicit Chain-of-Thought (CoT) reasoning with code verification before providing judgment for each reasoning step. To obtain high-quality process supervision labels and rationale data, we propose Relative Progress Estimation (RPE) and a rationale synthesis framework that incorporates code verification. Experimental results on ProcessBench and several mathematical reasoning tasks show that GenPRM significantly outperforms prior PRMs with only 23K training data from MATH dataset. Through test-time scaling, a 1.5B GenPRM outperforms GPT-4o, and a 7B GenPRM surpasses Qwen2.5-Math-PRM-72B on ProcessBench. Additionally, GenPRM demonstrates strong abilities to serve as a critic model for policy model refinement. This work establishes a new paradigm for process supervision that bridges the gap between PRMs and critic models in LLMs. Our code, model, and data will be available in https://ryanliu112.github.io/GenPRM.
1. Introduction
Existing PRMs limit test-time scaling by providing weak, poorly generalizing process supervision through scalar classification. GenPRM reframes process supervision as generative CoT reasoning with code verification and Relative Progress Estimation, achieving stronger results and enabling smaller models to surpass larger PRMs through TTS.
- 1. Introduction: Test-time scaling motivates the need for high-quality verifiers because scaling compute has demonstrated effectiveness for improving LLM reasoning performance.
- 1. Introduction: Existing PRMs provide limited process supervision, generalize poorly across models and tasks, and commonly ignore LLMs’ natural-language generation abilities by predicting scalar values.
- 1. Introduction: GenPRM reframes process supervision as generative reasoning, combining explicit CoT, code verification, and final step judgment instead of discriminative scalar scoring.Relative Progress Estimation is introduced to improve conventional hard-label estimation.
- 1. Introduction: GenPRM outperforms prior classification-based PRMs on ProcessBench and mathematical reasoning tasks, while smaller GenPRM models surpass larger PRMs through test-time scaling.
2. Preliminaries
This section formulates PRM-based test-time scaling as a Markov Decision Process and introduces majority voting and Best-of-N as the considered scaling methods.
- MDP Formulation: PRM-based test-time scaling is formulated as an MDP with state and action spaces, transition dynamics, rewards, and a discount factor.The process starts from a prompt-derived initial state, after which the policy model generates an action.
- Test-Time Scaling Methods: The study considers two test-time scaling methods: majority voting, which selects the most frequent answer, and Best-of-N, which selects the best answer from N candidates.These methods provide complementary answer-selection strategies over sampled solutions.
3. Method
GenPRM combines explicit CoT reasoning with code verification to judge each reasoning step, then uses the resulting verifier or critic for test-time scaling. Its training pipeline generates labeled paths, estimates relative progress, synthesizes verified rationales, and applies consensus filtering.
- GenPRM: GenPRM generates a rationale for each step, executes verification code, incorporates execution feedback, and predicts the final process reward from the verified reasoning.The code-verified formulation is the GenPRM variant used throughout the paper.
- Test-Time Scaling: GenPRM scales policy-model inference by sampling multiple reasoning verification paths, averaging rewards through majority voting, and using the scores to rank responses or judge step correctness.It can operate as a parallel verifier for selecting among sampled answers or as a sequential critic for multi-turn refinement.
- Data Generation: To balance MC-estimation precision and efficiency, GenPRM dynamically selects the number of completion trajectories based on estimated Pass@1.Up to 2048 paths are sampled for both hard and easy problems to obtain sufficient correct and incorrect solutions, discarding problems lacking either type.
- Relative Progress Estimation: Relative Progress Estimation compares the next state’s MC score with the current state’s score, masks steps after the first error, and applies a threshold to derive reward labels.The method treats a positive step as both correct and beneficial rather than relying directly on a positive MC score.
4. Experiments
GenPRM outperforms prior process reward models on ProcessBench and benefits substantially from test-time compute scaling. It also improves mathematical answer verification and policy refinement, with CoT reasoning and code verification contributing to performance.
- 4.2 ProcessBench Results: GenPRM-7B surpasses all prior PRMs below 72B, while GenPRM-1.5B substantially outperforms Skywork-PRM-1.5B using only 23K MATH examples.
- 4.2 ProcessBench Results: GenPRM-1.5B surpasses GPT-4 and GenPRM-7B exceeds Qwen2.5-Math-PRM-72B through majority-voting test-time scaling on ProcessBench.Scaling helps more on harder problems than easier questions.
- 4.3 GenPRM as a Verifier: GenPRM outperforms baselines across MATH, AMC23, AIME24, and Minerva Math, with larger advantages when scaling verifier and generator test-time compute.The advantage generalizes from Qwen2.5-Math-7B-Instruct to Gemma-3-12b-it as the generation model.
- 4.3 GenPRM as a Critic: GenPRM significantly improves policy-model performance through critique refinement, and performance continues increasing with additional refinement turns.
- 4.4 Ablation Studies: RPE and its variant outperform hard-label estimation across evaluated thresholds, while CoT reasoning provides the main gain and code execution further improves verification.
5. Related Work
Prior work establishes process reward models as effective step-wise evaluators but highlights the cost of process supervision, while test-time scaling and generative reward modeling extend verifier capabilities. Existing approaches use automated labeling, search strategies, and CoT-based reward generation to improve reasoning evaluation.
- Process Reward Models: Process reward models provide effective step-wise scores and outperform outcome reward models on mathematical reasoning tasks, but process-supervision annotation remains costly.Prior work addresses annotation costs with Monte Carlo estimation and binary search for automated label generation.
- Large Language Model Test-Time Scaling: Test-time computation improves inference performance through external verifiers such as ORMs and PRMs or strategies including beam search and MCTS.The paper situates its approach within test-time scaling of generative process reward models.
- Enhancing the Generative Abilities of Reward Models: Generative reward models have been enhanced with Chain-of-Thought reasoning, critique generation, reward prediction, SFT, preference learning, and hidden-state regularization.Examples include CLoud reward models, GenRM-CoT, GenRM, CTRL, and GRM.
6. Conclusion
GenPRM performs explicit generative reasoning with code verification for process supervision, outperforms prior PRMs, scales with test-time compute, and functions effectively as a critic model. Its added inference computation and current focus on mathematical reasoning motivate future work on dynamic pruning and broader task applications.
- Conclusion: GenPRM combines explicit reasoning and code verification for process supervision, outperforms prior PRMs, improves through test-time scaling, and serves effectively as a critic model.These results are reported on ProcessBench and several mathematical datasets.
- Limitations: GenPRM’s generative reasoning introduces additional inference computation, motivating future work on dynamically pruning the reasoning process.The authors also identify coding and general reasoning tasks as future application areas beyond mathematics.
A. Experimental Details · A.1. Scoring and Voting Methods
The experimental details define three scoring rules for converting step-level process rewards into a final response score: the last-step reward, the mean across steps, or the minimum reward.
- A.1. Scoring and Voting Methods: PRM-Last assigns the entire response a final score equal to the process reward of its last reasoning step.Its rule is score = 𝑟T.
- A. Experimental Details: The experimental-details block contrasts terminal-step scoring with aggregation across the full reasoning process.PRM-Last uses the final step, whereas PRM-Avg and PRM-Min use rewards from all steps.
- A. Experimental Details: Together, the listed methods provide three final-score choices based on the last reward, the average reward, or the weakest step reward.Each choice corresponds to one of the defined PRM scoring rules.
- A.1. Scoring and Voting Methods: PRM-Avg assigns the final response score using the mean process reward across all reasoning steps.The passage specifies PRM-Avg as averaging process rewards across steps.
- A.1. Scoring and Voting Methods: PRM-Avg and PRM-Min both incorporate all process steps, but they summarize them with different statistics: a mean versus a minimum.The distinction follows directly from their scoring definitions.
- A.1. Scoring and Voting Methods: PRM-Min assigns the final response score using the minimum process reward observed across all reasoning steps.Its rule is score = min𝑟{𝑟𝑡}T.
A.2. Implementation Details
GenPRM uses structured prompts to generate stepwise critiques with explicit reasoning and executable Python verification. Separate evaluation prompts require sequential paragraph judgments and standardized correctness outputs.
- A.2. Implementation Details: The rationale-generation prompt asks a math teacher to critique each solution paragraph step by step, separating analysis from Python-based verification.Each paragraph receives an <analyze> section explaining correctness and a <verify> section containing executable Python code.
- A.2. Implementation Details: The verification procedure requires code outputs to be analyzed, permits packages such as sympy and scipy, and enforces a precise executable-code format.The prompt instructs the system to print critic results and analyze the resulting [Code Output].
- A.2. Implementation Details: The ProcessBench evaluation prompt reviews solution paragraphs sequentially, provides detailed analyses and a Correct/Incorrect conclusion, and stops after the earliest detected error.When an error is found, the prompt requests its cause and correction while skipping dependent subsequent paragraphs.
- A.2. Implementation Details: A separate critique prompt instructs the model not to solve the problem, critique each paragraph only, and end every verification with a boxed Yes-or-No judgment.The prompt supplies the problem and solution in tagged paragraph format and requests paragraph-level correctness judgments.
B. Additional Results
Additional results examine GenPRM across benchmark outcomes, model sizes, training-data volumes, and inference-token statistics. Scaling from 1.5B to 7B yields substantial gains, while larger models and datasets show diminishing returns.
- Model Size: 57.3 →75.2 and 63.4 →80.5 performance gains result from scaling GenPRM from 1.5B to 7B parameters, while 32B adds only marginal improvements.The 7B variant offers the best balance between efficiency and effectiveness.
- Data Size: Pass@1 F1 scores improve rapidly with initial increases from 25% to 100% of ProcessBench training data, but growth slows with additional data.
- Inference Tokens: Table 9 provides statistics for reasoning tokens per step and per response.
- Table 6 reports the full results of critic models and PRMs on ProcessBench.
C. Cases
The case studies show that GenPRM can use code-execution feedback to correct CoT mistakes and strengthen process supervision, while also providing accurate supervision through CoT reasoning alone.
- Case 1: Code-execution feedback corrects mistakes in GenPRM’s CoT and enhances its process-supervision abilities.This finding is illustrated by the first case in Figure 5.
- Case 2: GenPRM provides accurate process supervision using CoT reasoning only.This finding is illustrated by the second case in Figure 6.