PaperlayerView exact revision on arXiv

Source-linked AI summary

Process Reward Models That Think

Muhammad Khalifa, Rishabh Agarwal, Lajanugen Logeswaran, Jaekyeom Kim, Hao Peng, Moontae Lee, Honglak Lee, Lu Wang

arXiv:2504.16828v5cs.LGcs.AIcs.CL

TL;DR

Step-level supervision makes conventional process reward models expensive to train, motivating data-efficient verifiers. ThinkPRM uses lightly finetuned reasoning models to generate verification chains of thought, and with minimal synthetic supervision outperforms LLM-as-a-judge and discriminative PRMs across several benchmarks and scaling settings.

  • Problem

    Discriminative PRMs depend on extensive step-level annotations, while existing generative verifiers motivate more data-efficient and scalable process verification.

  • Method

    ThinkPRM finetunes openweight reasoning models on filtered synthetic verification chains that match PRM800K step-level labels.

  • Results

    ThinkPRM outperforms LLM-as-a-judge and discriminative PRMs across benchmarks and test-time scaling scenarios, including 8% and 4.5% out-of-domain gains over discriminative PRMs.

  • Takeaways & Limitations

    Generative long-CoT PRMs can scale verification compute while requiring minimal supervision for training.

Abstract

from arXiv · show

Step-by-step verifiers -- also known as process reward models (PRMs) -- are a key ingredient for test-time scaling. PRMs require step-level supervision, making them expensive to train. This work aims to build data-efficient PRMs as verbalized step-wise reward models that verify every step in the solution by generating a verification chain-of-thought (CoT). We propose ThinkPRM, a long CoT verifier fine-tuned on orders of magnitude fewer process labels than those required by discriminative PRMs. Our approach capitalizes on the inherent reasoning abilities of long CoT models, and outperforms LLM-as-a-Judge and discriminative verifiers -- using only 1% of the process labels in PRM800K -- across several challenging benchmarks. Specifically, ThinkPRM beats the baselines on ProcessBench, MATH-500, and AIME '24 under best-of-N selection and reward-guided search. In an out-of-domain evaluation on a subset of GPQA-Diamond and LiveCodeBench, our PRM surpasses discriminative verifiers trained on the full PRM800K by 8% and 4.5%, respectively. Lastly, under the same token budget, ThinkPRM scales up verification compute more effectively compared to LLM-as-a-Judge, outperforming it by 7.2% on a subset of ProcessBench. Our work highlights the value of generative, long CoT PRMs that can scale test-time compute for verification while requiring minimal supervision for training. Our code, data, and models are released at https://github.com/mukhal/thinkprm.

1 Introduction

ThinkPRM addresses the expense of step-level supervision in process reward models by using generative verification chains of thought. With as few as 8K step labels, it outperforms LLM-as-a-judge and discriminative PRMs across test-time scaling scenarios.

  • PRMs support test-time scaling by scoring partial solutions for path selection or ranking.
  • Discriminative PRMs require extensive step-level annotations, often involving hundreds of thousands of labels for reasonably performing math models.
  • ThinkPRM repurposes openweight reasoning models as generative PRMs that verify solution steps through chains of thought.
  • 8K step labels suffice for ThinkPRM to outperform LLM-as-a-judge and discriminative PRMs trained on two orders of magnitude more data.

2 Background and Related Work

Prior PRMs mainly use discriminative classifiers that require extensive annotations and provide limited interpretability and fixed test-time compute. Generative verification instead produces verbalized step decisions with chains of thought, enabling interpretability and scalable verification compute.

  • Discriminative PRMs: Discriminative PRMs classify each solution step and aggregate step-level scores into an overall correctness score.
  • Discriminative PRMs: Discriminative PRMs require labor-intensive annotations, offer limited interpretability, and restrict dynamic test-time scaling through fixed compute.
  • Generative Verification: Generative verification frames step checking as language generation that produces decisions such as “correct” or “incorrect,” typically with a chain of thought.
  • Test-Time Scaling with PRMs: Generative PRMs support simultaneous scaling of generator and verifier compute, unlike discriminative PRMs.

3 THINKPRM

THINKPRM is trained from filtered synthetic verification chains and evaluated as a process verifier under several scaling settings. Finetuning improves accuracy and reduces invalid or excessively long verification outputs.

  • THINKPRM: THINKPRM targets step-by-step verification by generating an extended chain of thought for each problem-solution pair.
  • THINKPRM: The evaluation examines long-reasoning models as judges, synthetic-CoT finetuning, and comparisons with discriminative PRMs under test-time scaling.
  • Synthetic Data Collection: Synthetic verification chains are filtered for extractable labels, agreement with PRM800K gold labels, and a maximum length to avoid excessive overthinking.
  • Synthetic Data Collection: 1K verification chains provide 8K step labels for training THINKPRM.
  • Finetuning Results: Finetuning substantially improves F1 across models, with the 1.5B model gaining over 70 F1 points and the 14B model performing best.
  • Finetuning Results: THINKPRM generations remain approximately 1K-5K tokens while reducing overthinking and infinite looping relative to LLM-as-a-judge.
  • Finetuning Results: LLM-as-a-judge produces a significant share of verification chains without parsable yes/no labels, whereas THINKPRM substantially mitigates this issue.

4 Test-time Scaling Experiments

The experiments evaluate THINKPRM across best-of-N, guided search, out-of-domain tasks, and parallel or sequential verifier-compute scaling. THINKPRM generally outperforms discriminative PRMs and LLM-as-a-Judge, including with substantially less training supervision.

  • Best-of-N selection: THINKPRM-14B exhibits the best accuracy scaling curve on AIME ’24 and MATH-500 under best-of-N selection.The comparison includes LLM-as-a-Judge, DiscPRM, and unweighted majority vote.
  • Best-of-N selection: THINKPRM leads or matches DiscPRM across all MATH-500 sampling budgets and also performs better on AIME ’24.THINKPRM-1.5B additionally surpasses DiscPRM and LLM-as-a-Judge in MATH-500 beam search.
  • Verifier-guided search: THINKPRM-1.5B surpasses off-the-shelf PRMs across all tested beam sizes, outperforming RLHFFlow-Deepseek-PRM by more than 7%.The compared off-the-shelf PRMs were trained on more data and used larger models than THINKPRM-1.5B.
  • Out-of-domain evaluation: 8% on GPQA-physics and 4.5% on LiveCodeBench are THINKPRM’s out-of-domain advantages over DiscPRM.DiscPRM drops substantially at N=32 on GPQA-physics, while Qwen2.5-7B-Math-PRM struggles on LiveCodeBench out of domain.
  • Verifier-compute scaling: More than 5% points are gained by THINKPRM-1.5B@4 under verifier-guided search on MATH-500.THINKPRM also improves with 14B parallel scaling and sequential rechecking; on OmniMath, it outperforms DiscPRM-14B by 15 F1 points.
  • Verifier-compute scaling: Parallel and sequential scaling perform fairly similarly under equal token budgets, with a slight parallel advantage at certain budgets.The comparison uses Qwen-2.5-14B with K = 2 and 4.

5 Analysis and Discussion

The analysis examines ThinkPRM’s data efficiency, long-chain verification, generalization, filtering, and limitations. Across these settings, extended generative reasoning supports strong verification with little supervision, while generative PRMs retain calibration, autoregressive, and compute-overhead challenges.

  • Out-of-domain generalization: THINKPRM shows superior out-of-domain best-of-N performance on GPQA-Physics and LiveCodeBench, especially at larger sampling budgets.It was fine-tuned only on math, while discriminative PRMs trained on far more process labels fall short.
  • Training data efficiency: 8K process labels enable THINKPRM-14B to scale substantially better than DiscPRM-14B trained with up to 712K labels.The comparison uses 500 and 1K THINKPRM examples versus 1K, 10K, 50K, and 98K DiscPRM examples.
  • Verification CoT length: Shortening verification CoTs causes a substantial performance drop, supporting the value of extended reasoning during THINKPRM training.The experiment compares models trained on full long synthetic CoTs with models trained on concise rewrites of the same 1K CoTs.
  • Long reasoning traces: THINKPRM verifies long reasoning traces with backtracking and self-correction despite training on short solutions, extracting and checking embedded individual steps.On full MATH-500, it outperforms the baselines in the reported best-of-N evaluation.
  • Automatic supervision: Automatic labels produce performance very comparable to manual labels when training THINKPRM-1.5B on 1K synthetic chains.The comparison is reported on ProcessBench using labels from Math-shepherd versus manual PRM800K labels.
  • Problem difficulty: THINKPRM improves reasoning on difficult MATH-500 and GPQA-Physics problems more substantially than DiscPRM.The reported difficult levels are 3–5 for MATH-500 and 2–4 for GPQA-Physics.
  • Data filtering: Process-based filtering selects chains using agreement between step-level decisions and gold labels, unlike outcome-based filtering based only on final-answer correctness.The comparison evaluates whether intermediate process labels improve synthetic verification-chain curation.
  • Limitations: Generative PRMs can produce extreme scores, suffer step label interference, and incur extra verification-CoT overhead.The paper identifies calibration, premature autoregressive commitment, and generation cost as limitations requiring future work.

Conclusion

The conclusion presents THINKPRM as a generative process reward model trained with minimal synthetic supervision for scalable step-by-step verification. It emphasizes generative PRMs’ interpretability, scalability, and data efficiency for complex reasoning in math and science.

  • Conclusion: THINKPRM is a generative process reward model trained with minimal synthetic supervision for scalable step-by-step verification.The conclusion frames this as the paper’s central contribution.
  • Conclusion: Generative PRMs offer interpretability, scalability, and data efficiency for scaling verification compute on complex math and science reasoning tasks.This is the paper’s stated supported implication of the reported results.

A Training data

The training-data appendix describes how synthetic verification chains are sampled, filtered, cleaned, and illustrated. The resulting dataset contains 1K finetuning examples corresponding to roughly 8K process labels.

  • Synthetic data collection: Synthetic verification chains are sampled from PRM800K problem-prefix pairs using QwQ-32B-Preview until 1K qualifying chains are obtained.Each chain verifies solution steps and must satisfy the prescribed format and label requirements.
  • Synthetic data collection: Each sampled chain must provide extractable labels for every step, match PRM800K gold labels, remain within 4096 tokens, and satisfy additional quality criteria.These conditions define the acceptance constraints for the sampled verification CoTs.
  • Filtering: Boxed step predictions are extracted, and chains with missing, malformed, or incorrect verification decisions are discarded.The filtering requires every step-level decision to match the ground-truth labels.
  • Cleaning: The data-cleaning pipeline standardizes boxed notation, removes content after the final verification decision, and adds special tokens around verification reasoning.These transformations normalize the training examples for finetuning.
  • Dataset statistics: Table 2 reports statistics for the 1K finetuning dataset.The passage identifies the table’s purpose without supplying its individual statistics.
  • Examples and prompts: The appendix includes a training-example illustration, a synthetic-chain figure, and the prompt used to generate verification chains.The listed materials document both the sample format and collection prompt.

B LLM-as-a-Judge Limitations

The appendix points to examples of failures from QwQ-32B-Preview and DeepSeek-R1-Distill-Qwen-14B when used as LLM-as-a-judge verifiers.

  • LLM-as-a-Judge limitations: Failure examples are provided for QwQ-32B-Preview and DeepSeek-R1-Distill-Qwen-14B used in LLM-as-a-judge verification.The examples appear in Figures 15 and 17.

C.1 Training THINKPRM

THINKPRM models are trained with lightweight LoRA finetuning on synthetic verification chains, using relatively small training runs on single GPUs.

  • LoRA finetuning uses rank r = 32 and α = 16 for the THINKPRM models.Qwen-14B-Instruct and R1-Deepseek (14B) are trained for 3 epochs, while QwQ-32B-Preview is trained for 5 epochs.
  • Training uses an effective batch size of 16 and a fixed learning rate of 4 × 10−4 without warmup.
  • 14B models train for 1.5 hours on a single A100 80GB GPU, while QwQ-32B-Preview takes 4.5 hours.
  • DiscPRM baselines are trained on the full PRM800K dataset with binary cross-entropy for correct and incorrect step tokens.R1-Qwen-14B is trained for 1 epoch using two A100 80GB GPUs.

D Results on ProcessBench before and after finetuning

The evaluation compares LLM-as-a-judge and THINKPRM on ProcessBench and reward-guided reasoning, while also examining search, compute scaling, and model behavior. The supplied evidence emphasizes invalid judgments, verifier-guided search, and failure loops in off-the-shelf reasoning models.

  • Failure behavior: Off-the-shelf QwQ-32B-Preview and Deepseek-R1-Distill-Qwen-14B exhibit degeneration behavior, including getting stuck in loops, during LLM-as-judge verification.
  • ProcessBench evaluation: LLM-as-a-judge reasoning models suffer from repetitions, invalid formatting, overthinking, and other bad outputs that limit their utility as generative verifiers.
  • Reward-guided search: Verifier-guided beam search samples candidate next steps, scores them with the process verifier, and expands the top-K candidates for up to 20 steps per beam.The implementation uses M = 4 candidate next steps per beam.

E.6 Additional results on scaling verifier compute

Additional experiments test how THINKPRM responds to more verifier training data, alternative labels, problem difficulty, prompting choices, and increased verification compute.

  • Verifier compute scaling: Parallel scaling with THINKPRM-14B raises the accuracy curve over greedy single-CoT scoring, but K = 8 remains comparable to K = 4.The experiments sample K = 4 and K = 8 CoTs at temperature T = 0.6 and aggregate their scores.
  • Verifier compute scaling: Under the same generation budget, parallel and sequential verifier scaling have no clear winner, although parallel scaling is slightly better at best-of-8.
  • Training data scaling: Training on 65K synthetic verification CoTs further improves THINKPRM best-of-N performance over training on 1K examples.This comparison is reported for both THINKPRM-1.5B and THINKPRM-14B.
  • Label sources: THINKPRM performs well with automatic Math-Shepherd labels and is comparable to training with manual PRM800K labels.
  • Prompt sensitivity: LLM-as-judge verification shows high variance in performance and invalid-judgment ratios across prompts.The evaluated prompts include Bad-step-index and Decision-for-each-step instructions.

G Examples from THINKPRM

The examples illustrate THINKPRM’s intended behavior: generating a long verification chain that checks and labels solution steps, including steps embedded within long reasoning traces.

  • THINKPRM examples: THINKPRM-14B verifies and labels every step in a provided solution prefix through a long verification CoT.
  • Example sources: The examples include problem-solution pairs from ProcessBench, including OmniMath and OlympiadBench subsets.
  • Example verification chain: The example verification chain checks each step of a rectangular-prism surface-area solution and marks the steps as correct.
  • THINKPRM examples: THINKPRM extracts and verifies individual steps embedded in a long CoT solution generated by Qwen3-1.7B-thinking.
Loading 2504.16828v5…