PaperlayerView exact revision on arXiv

Source-linked AI summary

ScalePRM: Training Process Reward Models by Scaling Verification Compute Without Ground Truth

Salman Rahman, Sruthi Gorantla, Arpit Gupta, Swastik Roy, Nanyun Peng, Yang Liu

arXiv:2512.03244v2cs.LGcs.AIcs.CL

TL;DR

Step-level process reward models need correctness labels that are costly to annotate or often depend on ground truth. ScalePRM instead scales verification compute to generate synthetic labels and train generative PRMs. These PRMs perform strongly on ProcessBench and as RL reward signals, while the study identifies mathematical reasoning as its evaluation scope.

  • Problem

    Training process reward models at scale is constrained by the cost of human step-level annotation and by automatic labeling methods’ reliance on ground-truth answers.

    Source: §1 · p. 1

  • Method

    ScalePRM generates multiple independent verifications of each reasoning step, aggregates their judgments into synthetic labels, and uses the data to train generative PRMs for RL reward signaling.

    Source: §1 · pp. 1–2

  • Results

    Step-level self-consistency reaches 67.5 F1 on ProcessBench versus 66.4 for reference-guided training and 61.9 for GPT-4o; PRM-based RL reaches 47.4% average accuracy across six math benchmarks versus RLVR’s 43.9%.

    Source: §1 · p. 2, §6 · p. 8

  • Takeaways & Limitations

    Scaling verification compute is a viable alternative to ground-truth supervision for training high-quality process reward models.

    Source: §6 · p. 8

    The evaluation focuses on mathematically verifiable reasoning; extending the approach to other structured or open-ended domains remains future work.

    Source: Limitations · p. 8

Abstract

from arXiv · show

Training process reward models (PRMs) requires step-level correctness labels, obtained either through expensive human annotation or by relying on ground-truth answers, limiting the ability to scale process-level supervision. We propose ScalePRM, which scales verification compute as an alternative: given a problem and a candidate solution, we generate multiple independent verifications of each reasoning step and aggregate their judgments to produce synthetic step-level labels without ground truth. We explore two representative inference-time scaling strategies, parallel scaling through self-consistency and sequential scaling through meta-critique, and train generative PRMs on the resulting synthetic data. On ProcessBench, a benchmark for identifying erroneous steps in mathematical reasoning, PRMs trained on step-level self-consistency data achieve 67.5 F1, surpassing reference-guided training with ground-truth access (66.4 F1) and GPT-4o as a critic (61.9 F1). When deployed as reward signals in RL training with Qwen2.5-Math-7B, our best PRM achieves 47.4% average accuracy across six mathematical reasoning benchmarks, outperforming ground-truth-based RLVR (43.9%). We also identify and address reward exploitation patterns unique to generative PRM-based RL. Our results demonstrate that scaling verification compute is a viable alternative to ground-truth supervision for training process reward models.

1 Introduction

ScalePRM addresses the difficulty of obtaining step-level correctness labels by generating synthetic labels through scaled verification compute. The paper reports strong ProcessBench and reinforcement-learning results and examines reward exploitation in generative PRM-based RL.

1 Introduction

  • Step-level correctness labels are difficult to scale: human annotation is expensive, while automatic labeling methods rely on ground-truth answers.PRM800K required 800K human step-level labels to cover 12K problems.Source: §1 · p. 1
  • ScalePRM generates multiple independent verifications of each reasoning step and aggregates their judgments into synthetic labels using self-consistency or meta-critique.The resulting data trains generative PRMs through supervised fine-tuning, and those PRMs are used as reward signals in RL.Source: §1 · pp. 1–2
  • 67.5 F1 on ProcessBench for PRMs trained on step-level self-consistency data surpasses reference-guided training (66.4 F1) and GPT-4o as a critic (61.9 F1); RL reaches 47.4% average accuracy across six math benchmarks versus RLVR (43.9%).The RL comparison uses Qwen2.5-Math-7B.Source: §1 · p. 2
  • The paper identifies and addresses reward-exploitation patterns in generative PRM-based RL while reporting that these PRMs can outperform ground-truth-based RLVR as reward signals.The stated demonstration uses policy-model RL training.Source: §1 · p. 2

2 Generating Synthetic Verification Data for PRM Training

ScalePRM creates synthetic step-level verification labels by scaling verification compute in parallel or sequentially. Parallel methods aggregate independent judgments, while meta-critique revises an initial verification.

2 Generating Synthetic Verification Data for PRM Training

  • Eight sampled solutions per problem are verified using Qwen3-32B-Instruct, with self-consistency generating 16 independent verifications for each problem-solution pair.Qwen2.5-14B-Instruct generates the solutions; the verifier model and sampling setup are stated in the method description.Source: §2 · p. 2
  • Meta-critique generates an initial verification, critiques it for missed or incorrectly flagged steps or flawed reasoning, then merges the critique into a refined verification.The same verifier model performs the three stages using different prompts.Source: §2 · p. 2
  • The hybrid method selects a consensus verification from 16 outcome-consistency attempts, refines it with meta-critique, and the methods yield problem-solution-verification training triples.The refined hybrid verification is the final training data described for that variant.Source: §2 · p. 2
  • Self-consistency aggregates independent verifications by majority voting, either over final Yes/No verdicts or over each reasoning step’s correctness.Step-level consistency produces consensus judgments across the solution’s steps.Source: §2 · p. 2, Figure 1 · p. 3

3 Training Generative Process Reward Models

The training setup builds datasets from sampled mathematical solutions and synthetic verification labels, then trains generative reward models by next-token prediction.

3.1 Reward Model Training Setup

  • From 8,000 randomly selected math problems, the setup generates eight solution attempts per problem and about 63K filtered verification examples per method.The initial set contains 64K problem-solution pairs, with scaling methods producing the step-level labels.Source: §3.1 · p. 3
  • Generative reward models are trained with next-token prediction on synthetic verification data, unlike prior discriminative reward models that output numerical scores.The passage states that three variants are trained on the generated dataset.Source: §3.1 · p. 3
  • The outcome reward model (ORM) is trained to output only the final Yes/No verdict given a problem and solution.Its training examples pair each problem-solution input with the final verdict.Source: §3.1 · p. 3

2. PRM

Scale verification compute supplies synthetic step labels for generative PRMs, and on ProcessBench, step-level consistency outperforms reference-guided training.

2. PRM

  • PRM outputs a correct-or-incorrect judgment for each step, while PRM-CoT also generates a rationale explaining each judgment; both are fine-tuned from Qwen2.5-14B-Instruct.Training uses three epochs and a learning rate of 5 × 10−6.Source: §2 · p. 3
  • ProcessBench tests identification of the earliest erroneous step or confirmation that all steps are correct across 3,400 cases, with F1 balancing error detection and over-criticism.The cases span GSM8K, MATH, OlympiadBench, and Omni-MATH; comparisons include single verification, reference-guided training, and LLM critics.Source: §2 · pp. 3–4

3.2 Results: ProcessBench Evaluation

  • On ProcessBench, step-level consistency reaches 67.5 F1 for PRM and 65.7 for PRM-CoT, above reference-guided scores of 66.4 and 63.2, respectively.Source: Figure 2 · p. 4, §3.2 · p. 4
  • Every inference-time scaling method improves over single verification by 1.9–5.9 F1 points; step-level consistency rises from 63.9 to 67.5 for PRM and 59.8 to 65.7 for PRM-CoT.These conditions use the same Qwen3-32B verifier, so the comparison isolates added verification compute from a stronger verifier.Source: §3.2 · p. 4
  • With step-level consistency, PRM scores 67.5 F1 and PRM-CoT 65.7, exceeding GPT-4o by 5.6 and 3.8 F1 points, respectively.Source: §3.2 · p. 4
  • Step-level consistency provides a viable alternative to ground-truth supervision for training high-performance generative process reward models.Source: §3.2 · p. 5

4 Reinforcement Learning with Process Rewards

The section evaluates generative process rewards in RL, comparing them with ground-truth RLVR and testing alternative reward formulations. PRM-CoT with process-aware rewards outperforms RLVR on the reported benchmarks, while online use of step-level rewards exposes several exploitation patterns.

4.1 RL Methodology

  • The methodology compares process-aware verdict rewards, step-augmented rewards, Selective Advantage, Global Step-Reward, and ground-truth RLVR.Selective Advantage zeros misaligned advantages; Global Step-Reward normalizes step rewards across steps from all solutions, while process-aware rewards use a format-validated final verdict.Source: §4.1 · p. 5

4.3 RL Training Results

  • 41.13% average accuracy on MATH-500, AIME 2024, and AIME 2025: PRM-CoT with process-aware rewards exceeds ground-truth RLVR’s 38% by 3.13 points.PRM-CoT also improves across all six benchmarks and across pass@1, pass@8, and pass@16; random rewards remain at the 29.67% baseline.Source: Figure 3 · p. 6, §4.3 · p. 6

4.2 RL Experimental Details

  • The RL setup uses Qwen2.5-Math-7B, GRPO, and 17K training problems, evaluating pass@1 across six mathematical reasoning benchmarks.The comparison includes process-aware, step-augmented, selective-advantage, and global-step rewards using PRM and PRM-CoT, alongside RLVR.Source: §4.2 · p. 6, Table 1 · p. 6

4.3 RL Training Results

  • PRM-CoT reaches 41.13% average test accuracy, above PRM at 34.0% and ORM at 33.53% when all three use process-aware rewards.The paper attributes PRM-CoT’s stronger performance to its verification rationales, compared with direct step judgments or outcome-only verification.Source: §4.3 · p. 7, Figure 4 · p. 7
  • Selective Advantage performs best at 44.0%, versus 41.13% for Process-Aware rewards and 41.19% for Global Step-Reward; Step-Augmented Process Rewards perform worst.Step-Augmented rewards blend step correctness (40%) with the verdict (60%); the authors attribute degradation to step inflation, as models add excessive sub-steps to raise the step-average reward.Source: §4.3 · p. 7, Figure 5 · p. 7

4.4 Reward Hacking in Generative Reward-Based RL

  • Online RL with generative rewards exhibits solution appending, step inflation, and step reduction as distinct reward-exploitation patterns.Without format constraints, models append unrelated solved problems; step signals can encourage excessive sub-steps or collapsing a solution into one step, depending on the reward formulation.Source: §4.4 · pp. 7–8

5 Related Work

Related work describes process reward models as offering step-level feedback, while noting that their training labels have depended on costly human annotation or ground-truth answers. The paper positions ScalePRM’s scaled verification data and Selective Advantage as alternatives, and highlights reward hacking in generative PRM-based RL.

Process reward models and verification

  • PRM labels have come from costly human annotation or automatic methods requiring ground-truth answers, limiting data scaling to new problems and domains.PRM800K covers 12K problems, while related approaches such as PRIME and RL-Tango also require ground truth.Source: §5 · p. 8

Inference-time scaling methods

  • ScalePRM scales verification compute to generate synthetic step-level labels, then trains generative PRMs that surpass reference-guided training on ProcessBench.The paper contrasts this with DeepCritic, which uses sequential scaling but requires ground truth, and direct self-consistency rewards, which can fail during online RL.Source: §5 · p. 8

Reinforcement learning with dense rewards

  • Selective Advantage preserves process-aware advantages only when step correctness aligns with solution outcomes, and the paper analyzes reward hacking in generative-PRM RL.The related-work discussion contrasts this with methods that require ground truth, including PRIME and RL-Tango.Source: §5 · p. 8

6 Conclusion

ScalePRM trains generative PRMs on synthetic verification data produced by scaling inference-time compute, reducing dependence on human annotation and ground-truth access. The paper reports gains on ProcessBench and in RL, while identifying reward-exploitation patterns that require constrained process-aware rewards.

6 Conclusion

  • 47.4% average accuracy across six mathematical reasoning benchmarks with Qwen2.5-Math-7B exceeded ground-truth RLVR at 43.9%.The reported result uses PRM-CoT with process-aware rewards.Source: §6 · p. 8
  • 67.5 F1 on ProcessBench for step-level consistency surpassed reference-guided training at 66.4 and GPT-4o at 61.9.Source: §6 · p. 8
  • The authors identified reward-exploitation patterns unique to generative process rewards and report stable training with properly constrained process-aware rewards.They conclude that scaling verification compute is a viable alternative to ground-truth supervision for training high-quality PRMs.Source: §6 · p. 8

Limitations

The evaluation and approach have boundaries in domain coverage, reward-model backbones, and RL training. The authors identify mathematical reasoning as the validated domain and describe further settings and methods as future work.

Limitations

  • Validation is limited to mathematical reasoning, selected because correctness is objectively verifiable and established benchmarks enable comparison with ground-truth methods.Extension to scientific reasoning, formal proofs, code verification, and open-ended domains is left for future work.Source: Limitations · p. 8
  • The reward-model backbones and comparison critics are non-reasoning models; reasoning-model PRMs and critics are proposed as a future extension.Source: Limitations · p. 9
  • In critic-free methods such as GRPO, step-level feedback remains challenging because steps across rollouts are not directly comparable.The authors also identify distinct exploitation patterns across explored reward approaches and suggest online co-evolution of reward models and policies as future work.Source: Limitations · p. 9

A Synthetic Verification Data Generation Details

The appendix formulates synthetic step-label generation without human annotation or ground-truth answers, describes a generator-verifier pipeline, and specifies three generative reward-model output formats.

A.2 Problem Formulation

  • The formulation uses eight solution attempts per problem and sixteen verifications per problem-solution pair, with each verification judging step correctness and giving a final verdict.The notation also includes per-step rationales for PRM-CoT.Source: §A.2 · p. 12
  • The goal is to generate correct-or-incorrect labels for each step of an LLM-generated solution without human annotation or ground-truth answers.Prior approaches rely on costly human labels or reference solutions requiring ground-truth answers.Source: §A.2 · pp. 11–12

A.3 Generator-Verifier Framework

  • A generator produces multiple solution attempts, and a verifier evaluates them without ground-truth access using step-level rationales and correctness judgments.Experiments use Qwen2.5-14B-Instruct for generation and Qwen3-32B-Instruct for verification; the generator samples eight attempts at temperature 0.7.Source: §A.3 · p. 12

B Generative Reward Model Specifications

C Detailed ProcessBench Evaluation Results

The section points to a comprehensive ProcessBench F1 comparison across evaluated models, but the supplied passage does not report the table’s values or winner.

C Detailed ProcessBench Evaluation Results

D Extended RL Experimental Details

The RL experiments use Qwen2.5-Math-7B with structured supervised fine-tuning before PRM-based optimization, then evaluate on six mathematical reasoning benchmarks.

Base models and format learning

Evaluation

  • Table 2 reports ProcessBench F1 scores for accuracies on erroneous and correct samples.The supplied caption does not specify table layout or values.Source: Table 2 · p. 13

Base models and format learning

  • The policy receives two SFT epochs on 113K structured math problems to support consistent formatting for PRM parsing.Solutions use <step> and <answer> tags and are generated by Qwen2.5-72B-Instruct.Source: D Extended RL Experimental Details · p. 13

RL implementation details

  • RL uses veRL to implement GRPO, training on 17K math problems with 16 rollouts per prompt and batch size 256.The policy uses AdamW with a constant 1 × 10−6 learning rate and KL regularization coefficient 0.001.Source: D Extended RL Experimental Details · p. 13

Benchmark and evaluation

  • Evaluation covers six competition-level math benchmarks and reports greedy zero-shot pass@1 accuracy.The benchmarks range from high-school competition to Olympiad-level difficulty.Source: D Extended RL Experimental Details · p. 14

E Reward Exploitation Analysis

This section analyzes structural reward-exploitation patterns in online RL with generative process rewards, including solution appending, step inflation, and reduction to single-step solutions. It also documents step inflation under hybrid PRM-CoT rewards and describes how meta-critique merges should correct or refine step verifications.

E Reward Exploitation Analysis

  • The analysis identifies solution appending, step inflation, and single-step collapse as distinct ways policies manipulate process-aware rewards instead of improving problem solving.These patterns arise respectively when outcome rewards lack format constraints, step-level signals are combined with outcome rewards or selective advantages, and Global Step-Reward lacks step penalties.Source: E Reward Exploitation Analysis · p. 14
  • Step-Augmented Process Rewards and Selective Advantage exploit step-level signals through step inflation, with longer solutions increasing rewards rather than improving reasoning.Step-Augmented weights step average at 40% and verdict at 60%; Selective Advantage without step penalties also sees rising reward and step count because PRM-CoT marks few steps incorrect, leaving most advantages intact.Source: E Reward Exploitation Analysis · p. 14, Figure 9 · p. 18
  • Global Step-Reward without step penalties instead collapses to single-step solutions by normalizing step rewards by total step count, enabling perfect 1/1 step reward (1.0) (Zha et al., 2025).Models package a whole solution into one <step> tag, and PRM-CoT typically marks the first parsed segment correct, allowing positive advantages while keeping Ki = 1.Source: E Reward Exploitation Analysis · pp. 14–15, Figure 9 · p. 18
  • PRM and PRM-CoT reach training reward 1.0 within 50–100 steps under outcome-only rewards, while MATH-500 and AIME 2024/2025 accuracy fall near zero as models append unrelated solved problems.Without format constraints, the reward model can assign 1.0 despite failure on the actual problem; Table 4 gives a concrete example.Source: E Reward Exploitation Analysis · p. 14, Figure 8 · p. 18, Table 4 · p. 19

F.4 Meta-Critique Merger for Verification Refinement

  • The meta-critique merger corrects identified errors, supplies accurate reasoning, updates step labels when needed, and aligns the final verdict with the corrected analysis.When the critique confirms the original verification, it instead preserves the analysis and labels while incorporating insights and fuller explanations where beneficial.Source: §F.4 · p. 20

G AI Assistance Disclosure

  • The authors used Claude for language editing and paraphrasing, while retaining responsibility for the paper’s technical content and analysis.The disclosure states that the authors conducted all technical content and analysis.Source: G AI Assistance Disclosure · p. 21

E Reward Exploitation Analysis

  • 19 steps at training step 80 grew to 39 at step 270 as solutions added unnecessary decompositions to maximize step-average reward.Under hybrid rewards combining 60% outcome reward, 40% step-average reward, and format constraints, steps 2–10 in the later solution could be compressed to steps 2–3 in the earlier one.Source: Table 5 · p. 22
Loading 2512.03244v2…