Unsupervised Process Reward Models

• Outcome reward models provide only a single trajectory-level score, which can reinforce incorrect reasoning traces when the final answer is correct.Process reward models provide dense step-wise scores and can guide reasoning more gradually.Source: §1 · p. 1
• uPRM avoids the costly step annotations and ground-truth answer verification that limit supervised PRMs and earlier unsupervised labeling approaches.Earlier approaches infer step labels from ground-truth answers or external verifiers and can be computationally demanding.Source: §1 · pp. 1–2
• The method interleaves reasoning steps with correctness markers and uses the LLM’s next-token probabilities for those markers to score candidate first-error positions.The paper’s premise is that these probabilities implicitly encode judgments about step correctness.Source: §1 · p. 2
• Rather than evaluating trajectories independently, uPRM scores them jointly and uses RL to distill the LLM’s evaluation capability into a dedicated process reward model.Source: §1 · p. 2
• uPRM improves first-error identification accuracy by up to 15% over LLM-as-a-Judge, with the largest gains on challenging datasets.The cited examples are OlympiadBench and Omni-Math [16].Source: §1 · p. 2
• In test-time scaling, uPRM gains up to 6.9% absolute over majority voting and is competitive with supervised PRMs on Best-of-8 selection.The comparison includes verification of 256 Llama-3.2-1B-Instruct generations [17].Source: §1 · p. 2
• For Qwen2.5-Math-1.5B [18], uPRM yields a 4% accuracy gain over training with a verifiable outcome reward and supports more robust RL optimization than a supervised PRM.Reward hacking still occurs with uPRM, but failures are less frequent and less severe, yielding superior final performance across multiple policy models.Source: §1 · p. 2

Process Reward Models from Outcome Labels

• Earlier approaches derive step supervision from outcome labels through automated annotations or implicit rewards, but still require ground-truth outcomes; uPRM requires neither outcome labels nor step annotations.Math-Shepherd [11] labels steps by their potential to lead to a correct answer, while implicit approaches use ORM partial responses as Q-values [13, 14] or other ORM-derived feedback [23] [24] [25]; automated annotation can require costly Monte Carlo rollouts.Source: §2 · p. 2

LLM-as-a-Judge paradigm

• Unlike judges that generate explicit verdicts, uPRM scores solutions with raw next-token probabilities and turns that assessment into direct supervision for PRM training.Prior LLM judges include pairwise-comparison systems such as MT-Bench and Chatbot Arena [26] and AlpacaEval [27], GPTScore’s generation likelihood [28], and G-Eval’s probability-weighted score tokens [29].Source: §2 · pp. 2–3

Test-time Scaling with Process Reward Models

• uPRM performs on par with supervised PRMs in test-time scaling, where reward models select among sampled answers using strategies such as Best-of-N, Beam Search, or DVTS.Selection methods remain below pass@N, which requires only one correct answer; majority voting is a reward-model-free alternative.Source: §2 · p. 3

Reinforcement Learning with Process Reward Models

• uPRM is more robust to reward hacking than a supervised PRM trained on the same dataset, addressing a key risk of using dense process rewards in reinforcement learning.RL frameworks commonly use sparse response-level outcome rewards, while PRMs offer dense intermediate rewards; algorithmic mitigations include min-form credit assignment [10].Source: §2 · p. 3

3.1 Supervised Process Reward Models

• A supervised PRM models each reasoning step’s correctness, and training uses trajectories labeled with the position of the first error or no error.Correctness is defined at the step level; its meaning may be ambiguous after the first error.Source: §3.1 · pp. 3–4

3.2 Large Language Models as Scoring Functions

• LLM next-token probabilities can be combined at selected positions in a template to score candidate answers for plausibility and consistency with pretrained knowledge.The background illustrates this with candidate completions of a template about Albert Einstein.Source: §3.2 · p. 4
• A policy can be trained with such plausibility scores, and the following section applies this principle to PRMs without ground-truth labels.The score is derived from probabilities assigned to specific tokens in a suitably constructed prompt.Source: §3.2 · p. 4

4 Unsupervised Process Reward Models

• The method defines a next-token-probability score for candidate first-error positions and trains uPRM by optimizing it, without curated step labels.The stated goal is to eliminate expert annotations.Source: §4 · p. 4

4.1 Scoring First Erroneous Position with LLMs

• For each candidate position, steps before it are marked correct and the candidate step incorrect; the LLM’s next-token probabilities for the labels define the score.The no-error candidate marks every step correct.Source: §4.1 · p. 4

4.2 Scoring Multiple Trajectories at Once

• Joint scoring conditions each trajectory’s candidate on preceding trajectories and labels, but can spuriously favor shared candidate positions regardless of errors; an Appendix A correction mitigates this.Marked sequences are concatenated before the LLM produces the joint score.Source: §4.2 · p. 5

4.3 Training PRM via Optimizing Joint Score

• The model uses LoRA on the scoring LLM and an MLP correctness head, then optimizes its first-error distribution with entropy regularization to prevent premature convergence.The entropy regularization strength is chosen by monitoring training curves to prevent collapse.Source: §4.3 · p. 5
• On 8 H200 GPUs, uPRM training takes ≈5.5 hours versus ≈4.25 hours for supervised PRM; joint scoring is training-only, so inference is independent.The paper describes the additional training overhead as negligible relative to the expert-labeling effort removed.Source: §4.3 · pp. 5–6

5.1 ProcessBench

• For direct error detection, uPRM is evaluated on first-error positions across four ProcessBench math datasets, with F1 as the primary aggregate metric.The LLM-as-a-Judge baseline uses the same base model and prompt template but scores each trajectory independently; ProcessBench also measures accuracy on erroneous and correct trajectories.Source: §5.1 · p. 6
• uPRM improves ProcessBench F1 by 13% absolute on OlympiadBench and 13% on MATH and Omni-MATH over LLM-as-a-Judge.It outperforms the baseline across all four datasets, with larger gains on the more challenging benchmarks.Source: Table 1 · p. 6, §5.1 · pp. 6–7

5.2 Scaling Test-Time Compute with uPRM

• Test-time scaling uses Best-of-N and DVTS to select among sampled answers, with the candidate budget reaching 256 and last-step reward scores used for selection.The setup evaluates instruction-tuned models on MATH-500, MinervaMath, and OlympiadBench; last-step scoring marginally outperforms the product of step-wise scores.Source: §5.2 · p. 7
• 17.1%: Llama-3.2-1B-Instruct’s average accuracy across three benchmarks rises from 14.6% with one candidate to 31.7% with 256; DVTS gains 6.9% and 4.4% over majority voting for Llama-3.2-1B and Qwen2.5-1.5B.Those DVTS gains are 2.8% and 1.5% over Best-of-N, respectively; DVTS performance degrades for larger policy models, and scaling effects depend on model and sampling strategy.Source: Figure 1 · p. 7, §5.2 · p. 7
• uPRM is competitive with supervised PRMs across three datasets on Best-of-8, despite using no step-level annotations and a generic instruction-tuned base model.The comparison uses Qwen2.5-Math-7B-Instruct to generate eight answers per question and contrasts Best-of-8 with pass@8.Source: Table 2 · p. 8, §5.2 · p. 8

5.3 Reinforcement Learning with uPRM

• For reinforcement learning, uPRM supplies dense PRM rewards through PURE’s min-form credit assignment, with RLOO fine-tuning tested using PRM-only, verifiable-reward-only, or mixed rewards.The mixed condition uses verifiable rewards for 10% of the data; training uses hard level 3–5 MATH problems and Qwen2.5 policy models.Source: §5.3 · p. 8
• 4 points: Qwen2.5-Math-1.5B trained with uPRM gains 4-point average accuracy across three benchmarks over verifiable-reward training, and uPRM training showed no reward hacking.By contrast, sPRM training collapsed in under 50 iterations for Qwen2.5-Math-7B or after several hundred iterations for the 1.5B model.Source: §5.3 · pp. 8–9, Table 3 · p. 9

• uPRM is competitive with expert-trained supervised PRMs in test-time verification and reinforcement learning despite sometimes lagging state-of-the-art on ProcessBench; localization accuracy is an incomplete proxy for downstream utility, consistent with [54].The paper argues that strong downstream utility does not require perfect localization of erroneous steps.Source: §6 · p. 9

Limitations

• Joint scoring constrains base-model choice because it requires sufficient context length and capability to judge concatenated trajectory batches.The constraint can be mitigated by using a capable LLM for scoring and a smaller model as the final PRM; growing open-source context windows and capabilities may also ease it.Source: §6 · p. 9

• The paper identifies a practically important robustness phenomenon whose precise origin remains uncharacterized and is a direction for future work.The authors describe this robustness as a contribution beyond introducing the first fully unsupervised PRM training method.Source: §6 · p. 9

A Score Correction to Mitigate Degenerate Solutions

• The correction excludes two degenerate configurations: every j_n = 1, or every j_n = T_n + 1.These correspond respectively to labeling the first step erroneous in every trajectory and predicting no erroneous steps.Source: A Score Correction to Mitigate… · p. 13
• The correction weights each trajectory by W_n = 1 + log(√T_n + 1) and sets the allowed surprise budget to B = (1 − ρ)S_max.ρ controls how many non-corner values of j_n are allowed in the batch.Source: A Score Correction to Mitigate… · p. 13
• With ρ = 0.25, regularization activates only when more than 75% of batch predictions collapse to corner categories.The authors describe this default as a weak safeguard against severe collapse, not a strong bias against general corner predictions.Source: A Score Correction to Mitigate… · p. 13
• Changing ρ produced no significant differences in pθ(·|τ) entropy or model performance.Figure A1 reports the ablation of the non-corner budget.Source: A Score Correction to Mitigate… · p. 13, Figure A1 · p. 14

B.1 Gradient Estimator

• Because the joint score factorizes autoregressively, the estimator derives an unbiased gradient for its first term while entropy gradients use standard automatic differentiation.The estimator addresses the score term summed across trajectories.Source: §B.1 · p. 14
• Custom FlexAttention masks allow b_imm_m values for all m to be computed in a single LLM forward pass.The estimator is inspired by the actor-critic framework and has a REINFORCE form with a variance-reduction baseline.Source: §B.1 · p. 14
• The actor-critic estimator uses the exact immediate baseline b_imm_m and a prefix-conditioned critic V_ϕ(j_<m) to estimate future returns without bias.The critic is trained to approximate those returns to further reduce estimator variance.Source: §B.1 · p. 14

B.2 Critic Architecture

• The design avoids additional LLM forward passes by reusing already-computed hidden states and exploits future trajectories to estimate returns.Future trajectories are privileged information used to facilitate return estimation.Source: §B.2 · p. 15
• The critic reuses joint-score hidden states at marked-sequence ends and PRM hidden states at trajectory-end special tokens, with the latter available only during training.The joint-score states encode trajectory and sampled-position history; an initial state is taken at the end of the system prompt.Source: §B.2 · p. 15
• Cross-attention combines history-conditioned representations with contextualized future trajectories to predict future returns.The attention context weights trajectory information by relevance to the current history; future-trajectory embeddings are privileged training-only information.Source: §B.2 · p. 15
• In practice, the critic uses 8-head cross-attention, projects hidden states to 1024 dimensions, and applies dropout, layer normalization, and a two-layer GELU MLP.Gradients are not backpropagated through the hidden states.Source: §B.2 · p. 15

C.1 Prompt Templates

• The prompt uses Qwen2.5-14B-Instruct as a strict mathematical reasoning judge, evaluating one step at a time and returning only + or -.Once a step is judged incorrect, subsequent steps for that problem are also marked incorrect.Source: §C.1 · pp. 15–16
• The joint-score prompt concatenates trajectories, ending each at a candidate first-error step; PRM processing includes all steps and handles trajectories independently.Joint-score prompts use +/- markers, while PRM prompts use the special token [*].Source: §C.1 · p. 16, Figure C1 · p. 16

C.2 Hyperparameters for Training PRM

• The PRM uses LoRA adapters on all transformer linear layers and trains for 1,000 AdamW updates at learning rate 10^-5 with effective batch size 64.LoRA rank is 64 with scaling factor α = 32; trainable parameters also include the [*] embedding and a two-layer MLP.Source: §C.2 · p. 16
• Training packs trajectories to a fixed 80 reasoning steps per batch, stabilizing gradient signal-to-noise by keeping the number of step-level predictions consistent.Batches contain about 13 trajectories on average; a final trajectory may be truncated because removing later steps preserves the first-error objective.Source: §C.2 · pp. 16–17

C.3 Test-time Scaling with PRM

• In Best-of-8 test-time scaling, last aggregation marginally outperforms product aggregation for uPRM, so the authors use last in Section 5.2.The experiments use the search-and-learn codebase; aggregation choice can affect performance differently across PRMs.Source: §C.3 · p. 17, Table C1 · p. 17

C.4 Reinforcement Learning with PRM

• RL experiments integrate PRMs into PURE on VeRL, run three seeds per reward-policy combination, and evaluate with greedy decoding; PRM and VR coefficients are both 1.Curriculum learning is disabled; Qwen2.5-Math uses generation length 4096, and Qwen2.5-7B with sPRM + VR is reported over two seeds after one trial failed before its first checkpoint.Source: §C.4 · p. 17, Table C2 · p. 18

D.1 Ablation of Entropy Regularization Strength

• γ = 3 balances exploration and exploitation, avoiding the rapid entropy collapse at γ = 1 and near-uniform predictions at γ = 9; the authors use it in all main-paper experiments.At γ = 1, entropy falls near zero by step 400, while γ = 3 decreases entropy gradually and maintains exploration.Source: §D.1 · p. 18, Figure D1 · p. 18

D.2 Complete Results on ProcessBench

• Optimizing the joint score distills the LLM’s evaluation capability into a more effective process reward model.The authors present this as the explanation for uPRM’s gains over LLM-as-a-Judge.Source: §D.2 · p. 19
• Error detection gains reach +15% on Omni-MATH and +13% on OlympiadBench, the most challenging datasets for identifying erroneous trajectories.Detecting errors is generally harder than recognizing correct solutions, and LLM-as-a-Judge accuracy degrades on harder benchmarks.Source: §D.2 · p. 19
• Across ProcessBench datasets and metrics, uPRM consistently outperforms LLM-as-a-Judge, with F1 gains from +8% on GSM8K to +14% on OlympiadBench.Table D1 provides the full benchmark breakdown.Source: Table D1 · p. 19, §D.2 · p. 19
• Table D1 reports accuracy on erroneous trajectories (Err.), accuracy on correct trajectories (Corr.), and their harmonic mean (F1).The appendix also reports PRM800K results as a sanity check, since uPRM is expected to outperform LLM-as-a-Judge on those training trajectories.Source: §D.2 · pp. 18–19, Table D1 · p. 19

D.3 Reinforcement Learning with PRM

• D.3 provides additional analysis and results from reinforcement-learning experiments using a PRM as the reward source.Source: §D.3 · p. 19

Reward Hacking Analysis

• sPRM induces Case 3 reward hacking with empty or nonsensical outputs, whereas uPRM eventually induces Case 2 with one reasoning step before stopping [10].The uPRM output remains sensible, while the sPRM policy neglects the prompt after collapsing.Source: §D.3.1 · p. 19
• SFT on uPRM-generated labels removes trivial reward hacking, but the policy still hacks reward with one long solution step, suggesting hacking type primarily follows labeling patterns [10].This ablation preserves the sPRM training procedure while replacing its labels with per-step uPRM labels; Figure D3 shows the single-step solution behavior.Source: §D.3.1 · pp. 19–20, Figure D3 · p. 20
• uPRM-trained Qwen2.5-7B experiences reward hacking later, stays closer to the reference policy, and produces lengthier responses than sPRM-trained Qwen2.5-7B.Figure D2 reports average response length and KL divergence to the reference policy.Source: §D.3.1 · p. 19, Figure D2 · p. 20

• The plots track accumulated PRM reward, response length, and KL to the initialization reference; accumulated reward excludes VR and is not comparable across uPRM and sPRM.Accumulated PRM reward is the PURE return for the first response step, approximating the minimum of per-step PRM rewards.Source: §D.3.2 · p. 21, Figure D4 · p. 23, Figure D5 · p. 24
• For Qwen2.5-7B, sPRM reward hacking was trivial Case 3 (0-step), while uPRM hacking was Case 2 (1-step): a stable, single correct reasoning step receiving high PRM reward [10].Table D2 gives examples of uPRM generations that begin a correct derivation but do not complete it.Source: §D.3.2 · p. 21, Table D2 · p. 22
• Later sPRM reward-hacking transitions did not change the degenerate behavior: models switched from empty-like responses to trivially continuing the prompt.One example was a drop in KL around iteration 300 for sPRM-trained Qwen2.5-7B.Source: §D.3.2 · p. 21, Figure D4 · p. 23
• Adding VR neither necessarily improved final performance nor delayed reward hacking; for Qwen2.5-Math-1.5B, sPRM + VR reward-hacked earlier than sPRM.The authors suggest that changing reward-term weights or increasing the share of data with VR could affect the outcome.Source: §D.3.2 · p. 21, Figure D4 · p. 23
• Across three Qwen2.5 training runs, uPRM training succeeded with or without verifiable reward, while sPRM (including sPRM + VR) reward-hacked into empty or extremely short responses rewarded at ~0.65.The plotted metrics show sharp increases in accumulated reward and KL alongside drops in response length when reward hacking occurs.Source: §D.3.2 · p. 21, Figure D4 · p. 23, Figure D5 · p. 24