PaperlayerView exact revision on arXiv

Source-linked AI summary

Process Reward Models That Think

Muhammad Khalifa, Rishabh Agarwal, Lajanugen Logeswaran, Jaekyeom Kim, Hao Peng, Moontae Lee, Honglak Lee, Lu Wang

arXiv:2504.16828v5cs.LGcs.AIcs.CL

TL;DR

Process reward models need step-level labels, which are costly to obtain, limiting data-efficient verification. ThinkPRM uses a long chain-of-thought verifier trained with 8K process labels; it significantly outperforms LLM-as-a-Judge and surpasses discriminative PRMs trained on much more data.

  • Problem

    Discriminative process reward models require costly step-level annotations, motivating data-efficient verifiers that can scale with additional test-time compute.

    Source: §1 · p. 2, §2 · p. 3

  • Method

    ThinkPRM is a generative verifier that checks solution steps by producing verification chains of thought, trained with minimal synthetic supervision and scalable through parallel or sequential compute.

    Source: §3.2 · p. 4, §4.1 · p. 6, Conclusion · p. 10

  • Results

    With just 8K process labels, ThinkPRM significantly outperforms LLM-as-a-Judge and surpasses discriminative PRMs trained on orders of magnitude more data.

    Source: Conclusion · p. 10

  • Takeaways & Limitations

    The findings demonstrate the potential of generative PRMs to scale verification compute for complex reasoning tasks in math and science.

    Source: Conclusion · p. 10

    Generative PRM scores can be overconfident and cluster near 0 or 1, motivating more reliable techniques for extracting calibrated scores.

    Source: §5.8 · p. 10

Abstract

from arXiv · show

Step-by-step verifiers -- also known as process reward models (PRMs) -- are a key ingredient for test-time scaling. PRMs require step-level supervision, making them expensive to train. This work aims to build data-efficient PRMs as verbalized step-wise reward models that verify every step in the solution by generating a verification chain-of-thought (CoT). We propose ThinkPRM, a long CoT verifier fine-tuned on orders of magnitude fewer process labels than those required by discriminative PRMs. Our approach capitalizes on the inherent reasoning abilities of long CoT models, and outperforms LLM-as-a-Judge and discriminative verifiers -- using only 1% of the process labels in PRM800K -- across several challenging benchmarks. Specifically, ThinkPRM beats the baselines on ProcessBench, MATH-500, and AIME '24 under best-of-N selection and reward-guided search. In an out-of-domain evaluation on a subset of GPQA-Diamond and LiveCodeBench, our PRM surpasses discriminative verifiers trained on the full PRM800K by 8% and 4.5%, respectively. Lastly, under the same token budget, ThinkPRM scales up verification compute more effectively compared to LLM-as-a-Judge, outperforming it by 7.2% on a subset of ProcessBench. Our work highlights the value of generative, long CoT PRMs that can scale test-time compute for verification while requiring minimal supervision for training. Our code, data, and models are released at https://github.com/mukhal/thinkprm.

1 Introduction

ThinkPRM is a long-CoT process verifier designed to scale verification compute while using far fewer step labels than discriminative PRMs. The paper reports strong results across math, out-of-domain, and long-form reasoning tasks.

  • THINKPRM repurposes openweight reasoning models as generative PRMs through lightweight training on synthetic data, using as few as 8K step labels and outperforming discriminative PRMs trained on two orders of magnitude more data.It verifies each step by generating a verification chain-of-thought.Source: §1 · p. 2
  • THINKPRM-14B outperforms a same-base discriminative PRM in accuracy with far fewer supervision signals, while THINKPRM-1.5B performs strongly on MATH-500 guided beam search using only 8K step labels.The comparison is shown in Fig. 1; the compute-scaling comparison with LLM-as-a-judge is shown in Fig. 2.Source: §1 · p. 2
  • THINKPRM outperforms strong PRM baselines on MATH-500 and AIME 2024 under best-of-N and guided search, and beats same-base LLM-as-a-judge by thinking longer during verification.The paper also reports that THINKPRM outperforms self-consistency at the same compute budget, especially with high sampling.Source: §1 · p. 2
  • THINKPRM generalizes to scientific reasoning and code generation, and to long-form reasoning without explicit step delimiters despite training only on short solutions.These evaluations test two types of domain shift.Source: §1 · p. 2
  • Fine-grained filtering of synthetic data based on step supervision is crucial for training high-quality PRMs.Source: §1 · p. 2

2 Background and Related Work

The background contrasts annotation-heavy, fixed-compute discriminative PRMs with generative verification, motivating long-CoT verifiers that can scale inference-time verification compute.

Discriminative PRMs

  • Discriminative PRMs classify annotated solution steps and aggregate their scores, but require extensive labels and offer limited interpretability and fixed test-time compute (Yuan et al., 2024; Zhang et al., 2024a; Mahan et al., 2024).They use a classification head and are typically optimized with binary cross-entropy.Source: §2 · pp. 2–3

Generative Verification

  • Generative verifiers emit step decisions with CoT rationales under language-model training, but prior short-CoT systems and unreliable LLM-as-a-Judge outputs limit practical scaling (Zhang et al., 2024a; Zheng et al., 2024; Bavaresco et al., 2024).Reasoning models such as QwQ-32B-Preview show promise, but their test-time-scaling utility remains limited without additional training (Team, 2024; Zheng et al., 2024).Source: §2 · p. 3

Test-Time Scaling with PRMs

  • Best-of-N and tree search rely on verifier quality, while generative PRMs permit joint generator/verifier scaling; long-CoT models enable parallel and sequential verifier-compute scaling (Zhang et al., 2024a; Kalra & Tang, 2025).The cited test-time scaling approaches include Best-of-N selection and tree-based search (Yao et al., 2023; Chen et al., 2024c).Source: §2 · p. 3

3 THINKPRM

ThinkPRM is a verbalized process reward model that checks each solution step through an extended chain of thought. The section motivates lightweight fine-tuning with synthetic verification chains to address weaknesses in off-the-shelf LLM-as-a-judge verification.

3.1 LLM-as-a-judge PRMs are suboptimal

  • Off-the-shelf LLM-as-a-judge verification is instruction-sensitive, with slight wording changes shifting F1 by 3–4 points, and can produce invalid labels, overthink, or loop.Accurate chains tend to be under 3K tokens, while inaccurate chains spike around 7K–8K tokens.Source: §3.1 · p. 4

3.2 Finetuning on synthetic data boosts LLM-as-a-judge verification

  • Training data comprises 1K synthetic verification chains sampled from QwQ-32B-Preview, corresponding to 8K PRM800K step labels and filtered for format, label agreement, and length.Filtering uses step-level annotations rather than gold rationales or verification chains.Source: §3.2 · p. 4
  • Step-label-only filtering makes the data pipeline low-overhead; the math-trained verifier also remains robust on science QA and code generation.Without gold step labels, the passage notes that silver labels can be obtained via Monte Carlo rollouts (Wang et al., 2023b; Chen et al., 2024a). Fine-tuning QwQ-32B-Preview takes 4.5 hours on one A100 80GB GPU.Source: §3.2 · p. 4
  • THINKPRM boosts F1 across all tested models, with the 1.5B model gaining over 70 F1 points and the 14B model performing best.Fine-tuning also reduces invalid labels for all models except QwQ, where they increase slightly.Source: §3.2 · p. 5, Figure 5 · p. 5
  • THINKPRM generations remain a reasonable 1K–5K tokens while reducing overthinking and looping and becoming substantially more accurate.The figure contrasts the long, repetitive LLM-as-a-judge chains with THINKPRM outputs.Source: §3.2 · p. 5, Figure 4 · p. 5

4 Test-time Scaling Experiments

THINKPRM generally outperforms discriminative and judge baselines across test-time scaling experiments, while its verification compute can be scaled in parallel or sequentially. The experiments cover math reasoning and out-of-domain scientific reasoning and code generation.

4.2 Results

  • On MATH-500, THINKPRM matched or exceeded DiscPRM across best-of-N budgets, and THINKPRM-14B had the best accuracy scaling curve on MATH-500 and AIME ’24.Under beam search on MATH-500, THINKPRM-1.5B also surpassed DiscPRM and LLM-as-a-Judge.Source: Figure 7 · p. 6, §4.2 · pp. 6–7

4.1 Experimental Setup

  • The evaluation spans MATH-500 and AIME ’24, plus out-of-domain GPQA-Diamond physics and LiveCodeBench; the study uses 1.5B and 14B ThinkPRM verifiers with task-specific generators.The setup evaluates 86 GPQA-physics questions and 200 LiveCodeBench problems, and uses different generator families for MATH-500 best-of-N and beam search.Source: §4.1 · p. 6
  • THINKPRM scales verifier compute by averaging scores from K independent verification CoTs in parallel or by prompting the model to double-check and self-correct sequentially.The study compares it with DiscPRM, trained on 712K PRM800K process labels, unweighted majority voting, and LLM-as-a-Judge.Source: §4 · p. 5, §4.1 · p. 6

4.2 Results

  • On MATH-500, THINKPRM-1.5B outperformed the best tested off-the-shelf PRM, RLHFFlow-Deepseek-PRM, by more than 7% across all beam sizes.The comparison included RLHFFlow-Deepseek-PRM and MATH-Shepherd-PRM, both trained on more data than PRM800K and larger than 1.5B.Source: §4.2 · p. 7, Figure 8 · p. 7
  • On out-of-domain tasks, THINKPRM outperformed DiscPRM by 8% on GPQA-physics and 4.5% on LiveCodeBench.On LiveCodeBench, Qwen2.5-7B-Math-PRM struggled under domain shift, while the authors characterize generative PRMs as more robust than discriminative PRMs.Source: §4.2 · p. 7
  • On MATH-500, THINKPRM-1.5B@4 improved verifier-guided search accuracy by more than 5% points and achieved the best accuracy; sequential verification beat DiscPRM-14B by 15 F1 points.On 200 OmniMath ProcessBench problems, THINKPRM’s verification F1 improved with longer CoTs, while LLM-as-a-Judge performance dropped after 16K tokens.Source: §4.2 · p. 7
  • Under the same token budget, parallel and sequential scaling performed fairly close overall, with a slight parallel-scaling advantage at some budgets.The comparison used Qwen-2.5-14B and K = 2 or 4 for both scaling modes.Source: §4.2 · p. 7

5 Analysis and Discussion

The analysis finds that ThinkPRM benefits from data-efficient training and extended verification reasoning, generalizes to challenging verification settings, and has limitations including overconfidence and added generation cost.

5.1 Training data efficiency

  • ThinkPRM-14B scales substantially better than DiscPRM-14B with two orders of magnitude fewer process labels, and benefits from expanding training from 1K to 65K CoTs.Although trained only on math, ThinkPRM also outperforms baselines on GPQA-Physics and LiveCodeBench, especially at larger sampling budgets.Source: §5.1 · p. 8, Figure 10 · p. 8

5.2 Effect of Verification CoT Length on PRM Quality

  • Training on short, compressed verification CoTs substantially reduces ProcessBench verification F1, supporting the value of extended reasoning during training.The comparison used short rewrites of the same 1K CoTs used for long-CoT training.Source: §5.2 · p. 8, Table 1 · p. 8

5.3 Reasoning traces without clear step boundaries

  • Despite training on short solutions, ThinkPRM verifies long CoTs and outperforms baselines by extracting and checking their embedded steps.The evaluation used Qwen3-1.7B in thinking mode on the full MATH-500 test set.Source: §5.3 · p. 9, Figure 11 · p. 9

5.4 Compute-matched comparison to self-consistency

  • At low sampling budgets, ThinkPRM best-of-N performs comparably to self-consistency, but gains a clear advantage as compute increases.The compute-matched comparison evaluated solution accuracy against sampling FLOPs on MATH-500 and GPQA-Physics.Source: §5.4 · p. 9, Figure 11 · p. 9

5.5 THINKPRM with Monte Carlo step labels

  • Training ThinkPRM-1.5B on automatic Math-Shepherd step labels yields ProcessBench performance very comparable to training on manual labels.This supports the training pipeline’s agnosticism to step-labeling strategy.Source: §5.5 · p. 9

5.6 THINKPRM helps with difficult reasoning problems

  • ThinkPRM improves reasoning more than DiscPRM on difficult problems: MATH-500 levels 3–5 and GPQA-Physics levels 2–4.The comparison used Qwen-2.5-14B for MATH-500 and Qwen-2.5-32B-Instruct for GPQA-Physics.Source: §5.6 · p. 9

5.7 Filtering based on process vs. outcome labels

  • Process-based filtering produces significantly better verification performance than outcome-based filtering despite using 65K CoTs rather than 128K.Process filtering uses agreement between generated step-level decisions and gold process labels; outcome filtering retains chains based only on final-answer correctness.Source: §5.7 · p. 9

5.8 Limitations of Generative PRMs

  • Generative PRMs can produce overconfident, extreme scores and let earlier step judgments interfere with later ones, while verification CoT generation adds overhead.The authors identify calibrated score extraction and the extra generation cost as limitations and future-work directions.Source: §5.8 · p. 10

Conclusion

THINKPRM is a generative process reward model trained with minimal synthetic supervision for scalable step-by-step verification. With 8K process labels, it outperforms LLM-as-a-judge and surpasses discriminative PRMs trained on much more data.

  • With 8K process labels, THINKPRM significantly outperforms LLM-as-a-judge and surpasses discriminative PRMs trained on orders of magnitude more data.Source: Conclusion · p. 10
  • THINKPRM is a generative process reward model trained with minimal synthetic supervision for scalable step-by-step verification.Source: Conclusion · p. 10
  • The results highlight generative PRMs’ interpretability, scalability, and data efficiency, and their potential to scale verification compute for complex reasoning tasks in math and science.Source: Conclusion · p. 10

A Training data · A.1 Sampling · A.2 Preprocessing

The training data consists of 1K synthetic verification CoTs sampled from PRM800K and filtered for step-level label agreement. Preprocessing extracts and validates boxed decisions, then standardizes the chains for training.

A.1 Sampling

  • The final set contains 1K verification CoTs that meet format, gold-label agreement, length, and final-answer-balance criteria.Each chain is at most 4096 tokens long, and the set is roughly balanced by final-answer correctness.Source: §A.1 · p. 15
  • Sampling uses problem-prefix pairs from the PRM800K train split, which is based on MATH (Lightman et al., 2023; Hendrycks et al., 2021).Source: §A.1 · p. 15
  • QwQ-32B-Preview generates four verification chains per prefix at temperature T = 0.1.Source: §A.1 · p. 15
  • About 20% of initial samples met the format and label-match criteria, requiring roughly 5K samples to obtain 1K filtered chains.Source: §A.1 · p. 15

A.2 Preprocessing

  • Preprocessing extracts boxed predictions, such as boxed correct or incorrect, for each reasoning step.Source: §A.2 · p. 15
  • Chains with missing or malformed decisions are discarded, and all step-by-step decisions must match ground-truth labels.Source: §A.2 · p. 15
  • The remaining chains have boxed notation standardized, content after the final decision removed, and special tokens added around verification reasoning.Source: §A.2 · p. 15

B LLM-as-a-Judge Limitations · C Training details

The section reports examples of QwQ-32B-Preview and Deepseek-R1-Distill-Qwen-14B failing as LLM-as-judge verifiers.

B LLM-as-a-Judge Limitations

C.1 Training THINKPRM · C.2 Training Discriminative Verifiers

ThinkPRM is trained on 1K filtered chains with LoRA or full training, using model-specific schedules and limited GPU time. Discriminative R1-Qwen verifiers are trained on all of PRM800K with binary cross-entropy supervision.

C.1 Training THINKPRM

  • LoRA fine-tuning on 1K filtered chains covers Qwen-14B-Instruct and R1-Deepseek (14B) for 3 epochs and QwQ-32B-Preview for 5 epochs.The LoRA rank is 32 and α is 16.Source: §C.1 · p. 17
  • The LoRA models use an effective batch size of 16 and a fixed learning rate of 4 × 10−4 without warmup.Source: §C.1 · p. 17
  • Training the 14B models takes 1.5 hours and QwQ-32B-Preview takes 4.5 hours on a single A100 80GB GPU.Source: §C.1 · p. 17
  • QwQ-32B-Preview is difficult to train with LoRA and still produces a relatively high percentage of invalid judgments afterward.The authors suggest full training would likely resolve the issue but require more compute than available.Source: §C.1 · p. 17
  • R1-Distill-Qwen{1.5B,7B} use full training on four RTX A6000 48GB GPUs with data parallel; the 1.5B model trains for 3 epochs with batch size 32 and learning rate 6 × 10−5.The learning rate is constant, without decay or warmup.Source: §C.1 · p. 17

C.2 Training Discriminative Verifiers

  • Discriminative R1-Qwen-14B trains for 1 epoch on all PRM800K using binary cross-entropy for “+” and “-” step labels, while R1-Qwen-1.5B uses batch size 64 and learning rate 1 × 10−4.R1-Qwen-14B uses two A100 80GB GPUs, batch size 8, and learning rate 6 × 10−5; the 1.5B model uses the same infrastructure and a 10% warmup ratio.Source: §C.2 · p. 17

D Results on ProcessBench before and after finetuning … E.5 MATH-500 test examples

These sections report ProcessBench comparisons and specify the evaluation and test-time scaling procedures, including the sampled MATH-500 subset. The supplied passages describe a performance gain from parallel verification-CoT sampling but do not provide its numerical magnitude.

E Evaluation details

D Results on ProcessBench before and after finetuning

  • Table 3 compares average F1-scores for LLM-as-a-Judge and ThinkPRM finetuned on 1K examples across OlympiadBench and OmniMath ProcessBench subsets.The caption gives random baselines of 39.1% for OlympiadBench and 32.7% for OmniMath; it does not provide the compared methods’ scores.Source: D Results on ProcessBench before and… · p. 17, Table 3 · p. 18

E.1 Predicting verification labels

  • THINKPRM generates verification CoT up to 8192 tokens, then force-decodes “Is the solution correct?” and scores with P(“yes”)/(P(“yes”)+P(“no”)).DiscPRM scores use the last step.Source: §E.1 · p. 18

E.2 Best-of-N selection

  • Best-of-N uses weighted majority voting over verifier scores, except AIME ’24, where verifier scores directly rank solutions.Solutions are sampled at T = 0.8 for Llama-3.2-3B-Instruct and T = 0.4 for Qwen-2.5-14B.Source: §E.2 · p. 21

E.3 Verifier-guided beam search

  • Verifier-guided beam search expands top-K candidates scored from M = 4 next steps per beam, for at most 20 steps, using T = 0.6.Double newlines delimit steps.Source: §E.3 · p. 21

E.4 Sequential scaling of verifier compute

  • Sequential verifier scaling uses up to R = 4 thinking rounds with distinct trigger phrases to prompt revisiting or double-checking earlier verification.The same phrases are used for THINKPRM and LLM-as-a-Judge for a fair comparison.Source: §E.4 · p. 21

E.5 MATH-500 test examples

  • Because evaluating all MATH-500 examples would require substantial compute, experiments use 100 randomly sampled problems covering different difficulty levels.Figure 18 shows the difficulty-level histogram for the subset.Source: §E.5 · p. 22, Figure 18 · p. 22
  • Parallel verification-CoT sampling further boosts the performance curve over scoring with a single CoT.Figure 19 compares THINKPRM-14B with THINKPRM-14B@4 and THINKPRM-14B@8.Source: Figure 19 · p. 22

E.6 Additional results on scaling verifier compute … F More details on LLM-as-judge verification

These additional experiments show how ThinkPRM responds to more verification compute and training data, while examining label sources, problem difficulty, and prompt sensitivity in LLM-as-judge verification. More synthetic training data improves MATH-500 performance but can reduce out-of-domain performance.

E.6 Additional results on scaling verifier compute

  • Parallel scaling lifts THINKPRM-14B accuracy over greedy K = 1, but performance plateaus quickly and K = 8 remains comparable to K = 4.K = 8 is slightly better at smaller sampling budgets; parallel scaling samples and aggregates multiple verification CoTs.Source: §E.6 · p. 22
  • Under the same token budget, parallel and sequential scaling show no clear winner, though parallel scaling appears slightly better at best-of-8.Figure 20 compares independently sampled verification CoTs with repeated self-revision rounds, generating up to 8192 tokens per generation.Source: §E.6 · p. 22, Figure 20 · p. 22

E.7 Scaling training data of THINKPRM

  • Training on 65K rather than 1K synthetic verification CoTs improves ThinkPRM accuracy on MATH-500, but can reduce out-of-domain performance.The comparison covers THINKPRM-1.5B and THINKPRM-14B; the reported out-of-domain drop is attributed to distribution shift.Source: §E.7 · p. 23, Figure 21 · p. 23, Figure 22 · p. 23

E.8 Results with automatic labels

E.9 Verifier performance in terms of problem difficulty

  • THINKPRM helps more than DiscPRM on harder reasoning problems, with GPQA-Physics difficulty estimated by binning Qwen2.5-32B-Instruct pass@1 rates from 32 sampled solutions per problem.MATH problems use their annotated difficulty levels; difficulty was not estimated for them.Source: §E.9 · p. 23, Figure 23 · p. 24

F More details on LLM-as-judge verification

  • LLM-as-judge verification performance and invalid-judgment rates vary substantially by prompt across Single-label, Bad-step-index, and Decision-for-each-step instructions.Bad-step-index asks for the first incorrect step and a yes/no solution label; Decision-for-each-step asks for a critique and boxed label per step.Source: F More details on LLM-as-judge… · p. 24, Table 5 · p. 24

G Examples from THINKPRM

The examples show THINKPRM producing long verification chains that inspect solution steps individually, including steps embedded in a long chain of thought. One math example illustrates how it explains a step’s correctness before labeling it.

Examples from THINKPRM

  • On an OmniMath ProcessBench example, THINKPRM-14B generates a long verification CoT that verifies and labels every step in the provided solution prefix.The problem and solution are shown in Figure 27, and the verification output in Figure 28.Source: G Examples from THINKPRM · p. 24, Figure 27 · p. 27, Figure 28 · p. 29
  • THINKPRM-1.5B extracts and verifies individual steps embedded in a long CoT solution from Qwen3-1.7B-thinking.Figure 29 illustrates this long-CoT verification example.Source: Figure 29 · p. 30

Related papers

Loading 2504.16828v5…