PaperlayerView exact revision on arXiv

Source-linked AI summary

Unsupervised Process Reward Models

Artyom Gadetsky, Maxim Kodryan, Siba Smarak Panigrahi, Hang Guo, Maria Brbic

arXiv:2605.10158v1cs.LG

TL;DR

PRMs provide fine-grained supervision for steering LLM reasoning, but expert step annotations and verification requirements make them costly and computationally demanding. This paper trains fully unsupervised PRMs using LLM next-token probabilities and joint trajectory scoring, achieving competitive downstream performance with supervised PRMs across error localization, test-time scaling, and reinforcement learning.

  • Problem

    PRMs require costly expert step-level annotations and ground-truth verification, limiting their scalability and general applicability.

  • Method

    uPRM uses LLM next-token probabilities for correctness markers to jointly score candidate first-error positions across multiple trajectories, then optimizes a PRM using this score.

  • Results

    uPRM is competitive with supervised PRMs across error localization, test-time verification, and reinforcement learning, while improving over majority voting by up to 6.9% in test-time scaling.

  • Takeaways & Limitations

    Strong downstream utility does not require perfect erroneous-step localization, suggesting unsupervised PRMs can reduce the marginal cost of step-wise guidance for new domains and pipelines.

  • Takeaways & Limitations

    Joint scoring requires a sufficiently capable LLM with enough context length to process concatenated trajectory batches, constraining base-model choice.

Abstract

from arXiv · show

Process Reward Models (PRMs) are a powerful mechanism for steering large language model reasoning by providing fine-grained, step-level supervision. However, this effectiveness comes at a significant cost: PRMs require expert annotations for every reasoning step, making them costly and difficult to scale. Here, we propose a method for training unsupervised PRMs (uPRM) that requires no human supervision, neither at the level of step-by-step annotations nor through ground-truth verification of final answers. The key idea behind our approach is to define a scoring function, derived from LLM next-token probabilities, that jointly assesses candidate positions of first erroneous steps across a batch of reasoning trajectories. We demonstrate the effectiveness of uPRM across diverse scenarios: (i) uPRM achieves up to 15% absolute accuracy improvements over the LLM-as-a-Judge in identifying first erroneous steps on the ProcessBench dataset; (ii) as a verifier for test-time scaling, uPRM performs comparably to supervised PRMs and outperforms the majority voting baseline by up to 6.9%, and (iii) when used as a reward signal in reinforcement learning, uPRM enables more robust policy optimization throughout training compared to a supervised PRM trained using ground-truth labels. Overall, our results open a path toward scalable reward modeling for complex reasoning tasks.

1 Introduction

PRMs provide denser reasoning supervision than outcome-only rewards but require costly step-level labels. This work introduces fully unsupervised PRMs using LLM-derived joint scores and evaluates them across error localization, test-time scaling, and reinforcement learning.

  • Outcome Reward Models provide sparse trajectory-level feedback that can reinforce incorrect reasoning traces despite correct final answers.This limitation is especially pronounced for long chains of thought.
  • Process Reward Models improve control by assigning dense step-wise scores, but require meticulously annotated reasoning trajectories.Existing approaches also infer labels from final answers or external verifiers, often at substantial computational cost.
  • uPRM requires neither step-level annotations nor ground-truth verification of final answers.Its scoring function uses LLM next-token probabilities to assess candidate first-error positions jointly across trajectories.
  • Up to 15% absolute accuracy improvements over the LLM-as-a-Judge baseline were achieved for identifying first erroneous steps.The largest gains occurred on challenging datasets including OlympiadBench and Omni-Math.
  • In test-time scaling, uPRM outperformed majority voting by up to 6.9% and remained competitive with supervised PRMs on Best-of-8 selection.It also supported more robust reinforcement-learning policy optimization than a supervised PRM trained with ground-truth labels.

2 Related Work

Prior work reduces PRM annotation demands through outcome labels, implicit rewards, or LLM judges, but these approaches retain supervision or selection limitations. The proposed approach instead removes outcome-label dependence and uses probabilistic judging for optimization.

  • Automated process-annotation methods infer step labels from final outcomes but still complement rather than replace outcome-based supervision.These methods include assigning labels according to each step’s potential to lead to a correct final answer.
  • Implicit process reward methods derive step-wise rewards from outcome rewards, but depend on ground-truth outcome labels for rollouts or ORM training.The proposed approach eliminates both step-level annotations and outcome labels.
  • LLM-as-a-Judge systems commonly generate explicit verdicts or scores that are parsed into labels or weighted evaluations.Examples use pairwise comparisons, generation likelihoods, or token probabilities over score tokens.
  • This method uses raw next-token probabilities to score solution plausibility rather than sampling a judgment, then converts that assessment into an optimization objective.This extends LLM judging from offline evaluation and model selection to reward-model training.
  • In test-time scaling, uPRM performs on par with supervised PRMs despite being fully unsupervised.The result addresses the challenge of using PRM rewards to select final responses.
  • Process rewards are also used in reinforcement learning because dense intermediate feedback is preferable to sparse outcome rewards.Reward hacking remains a central challenge when policies exploit spurious reward-model patterns.

3 Background

A PRM models step correctness and the position of the first erroneous step in a reasoning trajectory. The paper motivates replacing expert labels with scoring functions derived from LLM next-token probabilities and optimized by reinforcement learning.

  • A trajectory contains a problem and reasoning steps, while a PRM assigns a distribution over binary step-correctness labels.A correct label indicates that the corresponding reasoning step is correct.
  • PRM training ordinarily requires each trajectory to be paired with the ground-truth position of its first erroneous step.The standard objective is maximum likelihood over these labeled positions.
  • This paper defines PRM correctness at the step level rather than as the probability that a partial trajectory eventually yields a correct final answer.Correctness after the first error is treated as ambiguous, motivating the first-error definition.
  • The first-error position ranges from the first step through T + 1, where T + 1 denotes that the trajectory contains no error.This position variable is used in the PRM’s likelihood objective.
  • LLM next-token probabilities can score candidate solutions by measuring the plausibility of selected tokens in a constructed template.Probabilities at arbitrary template positions can be combined into task-specific scoring functions.
  • Given such a score, reinforcement learning can train a policy to produce more plausible answers.The paper applies this principle to PRM training without ground-truth first-error labels.

4 Unsupervised Process Reward Models

uPRM constructs correctness-marked trajectories, scores candidate first-error positions with LLM next-token probabilities, and jointly evaluates batches to obtain a training signal. It then trains a PRM by reinforcement learning with entropy regularization, while independent inference preserves ordinary PRM usage.

  • 4 Unsupervised Process Reward Models: uPRM replaces curated first-error labels with an LLM-derived score for candidate first-error positions.The score is optimized to train the unsupervised PRM without expert annotations.
  • 4.1 Scoring a Single Trajectory: For a candidate error position, uPRM marks preceding steps correct and the candidate step incorrect; the no-error case marks every step correct.The constructed sequence interleaves reasoning steps with “+” and “-” correctness markers.
  • 4.1 Scoring a Single Trajectory: The scoring function extracts and renormalizes the LLM’s next-token probabilities for the correctness markers after each reasoning step.These probabilities determine how plausible each candidate first-error position is.
  • 4.2 Scoring Multiple Trajectories at Once: Joint scoring concatenates marked trajectories so later trajectory scores are conditioned on earlier trajectories and their candidate labels.This uses preceding trajectories as in-context examples rather than scoring each trajectory independently.
  • 4.2 Scoring Multiple Trajectories at Once: Joint scoring can spuriously favor configurations where all trajectories share the same candidate label, so the paper introduces a correction for this failure mode.The issue arises from in-context learning effects in the LLM scorer.
  • 4.3 Training PRM via Optimizing Joint Score: The PRM uses special step markers and a classification head to convert hidden states into step-level correctness probabilities.The first-error distribution is then derived from these step probabilities.
  • 4.3 Training PRM via Optimizing Joint Score: Training optimizes an entropy-regularized objective, with entropy preventing premature convergence and γ controlling regularization strength.The regularizer is selected to prevent collapse of the PRM during training.
  • 4.4 Efficient Optimization: uPRM training takes approximately 5.5 hours versus 4.25 hours for supervised PRM training on the same setup, while joint scoring is used only during training.At test time, the trained uPRM processes trajectories independently without additional context-length requirements.

5 Experiments

Across error localization, test-time scaling, and reinforcement learning, uPRM provides useful process rewards without step-level labels and remains competitive with supervised alternatives.

  • Experimental scope: uPRM is evaluated for step-level error detection, test-time scaling, and reinforcement-learning reward guidance.The experiments cover ProcessBench, verifier-based inference, and policy optimization.
  • ProcessBench: uPRM consistently outperforms LLM-as-a-Judge on ProcessBench, with especially large gains on OlympiadBench and Omni-MATH.The reported absolute improvements are 13% on each of these challenging benchmarks.
  • Test-time scaling: 17.1% absolute accuracy improvement is observed for Llama-3.2-1B-Instruct as test-time candidates increase from 1 to 256.Average accuracy across MATH-500, MinervaMath, and OlympiadBench rises from 14.6% to 31.7%; gains depend on model size and sampling strategy.
  • Test-time scaling: 6.9% improvement over majority voting is achieved by DVTS with uPRM for Llama-3.2-1B-Instruct.The corresponding gain over Best-of-N is 2.8%, while larger policy models can experience degraded DVTS performance.
  • Supervised comparisons: uPRM is competitive with supervised PRMs on Best-of-8 verification despite generic instruction-tuned initialization and no step-level supervision.The comparison includes supervised PRMs trained from expert or outcome-derived annotations.
  • Reinforcement learning: 4-point average accuracy gain is achieved by Qwen2.5-Math-1.5B with uPRM versus the same model trained with ground-truth verifiable rewards.uPRM training completed without observed reward hacking, whereas sPRM training collapsed early for the Qwen2.5-Math models.

6 Conclusion and Limitations

The paper presents fully unsupervised PRM training as a way to reduce the cost of obtaining step-wise guidance while retaining downstream utility. It also identifies robustness in reinforcement learning, although the source of that robustness remains unresolved.

  • Conclusion: uPRM requires neither step-level annotations nor ground-truth verification of final answers.The approach is intended to reduce the marginal cost of step-wise guidance for new domains, model families, and inference or training pipelines.
  • Conclusion: Strong downstream utility does not require perfect localization of erroneous steps.The paper reports utility in test-time scaling and reinforcement learning even when uPRM may lag state-of-the-art supervised PRMs on ProcessBench.
  • Open direction: The paper identifies a practically important robustness phenomenon in reinforcement learning whose precise origin remains uncharacterized.The authors describe fully characterizing this origin as an important next step.

A Score Correction to Mitigate Degenerate Solutions

The joint score can favor degenerate corner predictions, so the method adds a correction term that permits non-corner positions while penalizing severe collapse. The correction is designed as a weak safeguard, and varying its budget produces no significant changes in entropy or performance.

  • Motivation: The joint score can encourage degenerate solutions in which every trajectory’s first error is assigned to the first step or no error is assigned.These correspond to j_n = 1 and j_n = T_n + 1, respectively.
  • Correction design: The correction term uses an allowed surprise budget B derived from S_max to control how many non-corner positions occur across the batch.ρ determines the amount of non-corner values permitted among the candidate positions.
  • Correction design: ρ = 0.25 makes the correction inactive unless more than 75% of batch predictions collapse to corner categories.This setting is a weak safeguard against severe collapse rather than a strong bias against corner predictions.
  • Ablation: Ablations over ρ show no significant differences in pθ(·|τ) entropy or model performance.The experiments directly assess the robustness of the correction term.
  • Optimization: The training objective uses an entropy term and an unbiased gradient estimator for the autoregressively factorized joint score.The estimator is implemented in REINFORCE form with a critic baseline for variance reduction.

B.2 Critic Architecture

The critic estimates future returns from sampled-position history and privileged representations of future trajectories. Cross-attention contextualizes future-trajectory information before an MLP produces the critic value.

  • Representations: The critic uses hidden states from the joint-score computation and PRM forward pass as two complementary representation types.History states encode sampled positions, while PRM states provide trajectory embeddings available only during training.
  • Cross-attention: Cross-attention aggregates future-trajectory embeddings according to their relevance to the current sampled-position history.The history matrix H contains prior states, and G contains trajectory embeddings.
  • Value prediction: The critic concatenates the current history representation with the contextualized future representation and maps them to a scalar value with a two-layer MLP.The MLP uses GELU activation, while the implementation applies multi-head cross-attention with eight heads.

C Experimental and Implementation Details

The experiments specify prompt formats, optimization settings, batching, reward aggregation, and evaluation procedures for uPRM training, test-time scaling, and reinforcement learning. Ablations examine aggregation choices and entropy regularization to stabilize training.

  • C.1 Prompt Templates: The judge is instructed to evaluate one reasoning step at a time and mark subsequent steps incorrect after the first detected error.The allowed outputs are only + and -.
  • C.1 Prompt Templates: The joint-score prompt concatenates multiple trajectories with +/- correctness markers, whereas the PRM prompt uses [*] markers and processes each trajectory independently.For the joint score, a trajectory ends at its candidate first-error position or continues with + markers when no error is assigned.
  • C.2 Hyperparameters for Training PRM: Training uses LoRA adapters, a 10^-5 AdamW learning rate, 1000 updates, and an effective batch size of 64.Trainable parameters also include the [*] embedding and the MLP producing step-level probabilities.
  • C.2 Hyperparameters for Training PRM: Batches fix the total number of reasoning steps at 80 rather than the number of trajectories to stabilize the gradient signal-to-noise ratio.Truncating a final trajectory remains valid because only later steps are removed after the first-error position objective is defined.
  • C.3 Test-time Scaling with PRM: For test-time scaling, last aggregation marginally outperforms product aggregation with Best-of-8, so last is used in the reported experiments.The aggregation function converts step-wise PRM rewards into a score for each candidate answer.
  • C.4 Reinforcement Learning with PRM: RL experiments use the PURE implementation with equal PRM and verifiable-reward coefficients, disabled curriculum learning, and greedy evaluation.Qwen2.5-Math models use a 4096-token generation length because of limited context.
  • D.1 Ablation of Entropy Regularization Strength: Entropy regularization is evaluated through training curves for γ values spanning 30, 31, and 32.The curves track average entropy and average joint score.
  • D.1 Ablation of Entropy Regularization Strength: γ = 3 balances exploration and exploitation, while γ = 1 causes rapid entropy collapse and γ = 9 prevents sufficient exploitation of the joint-score signal.The intermediate setting is used for all main-paper experiments.

D.2 Complete Results on ProcessBench

On ProcessBench, the unsupervised PRM consistently outperforms the LLM-as-a-Judge baseline, with especially large gains on erroneous-trajectory detection and harder datasets.

  • The ProcessBench breakdown reports erroneous-trajectory accuracy, correct-trajectory accuracy, and their harmonic mean F1.
  • The PRM800K results serve as a sanity check because the unsupervised PRM was trained on those trajectories.
  • The unsupervised PRM consistently outperforms LLM-as-a-Judge across datasets and metrics, improving F1 by +8% on GSM8K to +14% on OlympiadBench.
  • +15% on Omni-MATH and +13% on OlympiadBench are the largest gains for detecting errors in erroneous trajectories.

D.3.1 Reward Hacking Analysis

Reward hacking differs substantially between supervised and unsupervised PRM training: uPRM delays hacking and preserves more sensible, longer responses, whereas sPRM can collapse to empty outputs.

  • uPRM-trained Qwen2.5-7B experiences reward hacking substantially later, stays closer to the reference policy, and produces lengthier responses than sPRM-trained models.
  • sPRM reward hacking is associated with empty or nonsensical responses that receive high PRM rewards, whereas uPRM produces a single reasoning step before stopping.
  • uPRM-SFT removes trivial reward hacking but leads the policy to generate one long reasoning step containing the full solution.
  • The uPRM-SFT example uses a multi-step-looking completion that introduces the solution through a single generated sequence.

D.3.2 Further Analysis and More Results

Across additional RL runs, uPRM supports successful training with or without verifiable rewards and produces milder reward hacking than sPRM, while combining rewards does not reliably improve outcomes.

  • The experiments compare uPRM and sPRM with verifiable reward alone, PRM rewards alone, or PRM rewards plus verifiable reward on 10% of data.
  • Accumulated PRM reward approximates the minimum per-step PRM reward for a response and excludes verifiable reward; uPRM and sPRM values are not formally comparable.
  • uPRM successfully trains Qwen2.5-Math models with and without verifiable reward, while sPRM causes reward hacking across all considered models.
  • sPRM reward hacking converges to empty or extremely short responses, whereas uPRM hacking is milder and outputs a single correct reasoning step.
  • sPRM can later switch from empty-like outputs to trivial prompt continuations without fundamentally changing the hacking type.
  • Combining PRM rewards with verifiable rewards neither necessarily improves final performance nor delays reward hacking.
Loading 2605.10158v1…