PaperlayerView exact revision on arXiv

Source-linked AI summary

Process Rewards with Learned Reliability

Jinyuan Li, Langlin Huang, Chengsong Huang, Shaoyang Xu, Donghong Cai, Yuyi Yang, Wenxuan Zhang, Jiaxin Huang

arXiv:2605.15529v1cs.CLcs.AIcs.LG

TL;DR

Existing PRMs provide single step scores without indicating when those imperfect predictions should be trusted. BetaPRM learns a Beta belief from Monte Carlo continuation counts, predicting both success probability and reliability, and uses that signal in ACA. Across four backbones and four benchmarks, BetaPRM improves Best-of-N selection, while ACA reduces tokens by up to 33.57% and improves final-answer accuracy.

  • Problem

    Existing PRMs expose a single step-level score, so downstream methods lack an indication of whether an imperfect prediction should be trusted.

  • Method

    BetaPRM predicts a Beta belief over prefix success probability and trains it with a Beta-Binomial likelihood from Monte Carlo continuation counts.

  • Results

    Across four backbones and four benchmarks, BetaPRM improves PRM-guided Best-of-N selection while preserving step-level error detection; ACA reduces inference tokens by up to 33.57% and improves final-answer accuracy.

  • Takeaways & Limitations

    Reliability-aware process rewards support adaptive computation that stops for confident selections and spends more computation on uncertain prefixes.

  • Takeaways & Limitations

    Experiments focus on multimodal PRMs because VisualPRM400K-v1.1 is the only publicly available training dataset reporting successful continuation counts for each prefix.

Abstract

from arXiv · show

Process Reward Models (PRMs) provide step-level feedback for reasoning, but current PRMs usually output only a single reward score for each step. Downstream methods must therefore treat imperfect step-level reward predictions as reliable decision signals, with no indication of when these predictions should be trusted. We propose BetaPRM, a distributional PRM that predicts both a step-level success probability and the reliability of that prediction. Given step-success supervision from Monte Carlo continuations, BetaPRM learns a Beta belief that explains the observed number of successful continuations through a Beta-Binomial likelihood, rather than regressing to the finite-sample success ratio as a point target. This learned reliability signal indicates when a step reward should be trusted, enabling downstream applications to distinguish reliable rewards from uncertain ones. As one application, we introduce Adaptive Computation Allocation (ACA) for PRM-guided Best-of-N reasoning. ACA uses the learned reliability signal to stop when a high-reward solution is reliable and to spend additional computation on uncertain candidate prefixes. Experiments across four backbones and four reasoning benchmarks show that BetaPRM improves PRM-guided Best-of-N selection while preserving standard step-level error detection. Built on this signal, ACA improves the accuracy--token tradeoff over fixed-budget Best-of-16, reducing token usage by up to 33.57% while improving final-answer accuracy.

1 Introduction

Existing PRMs expose a single step score without indicating whether it should be trusted, despite noisy Monte Carlo supervision. BetaPRM models both prefix success and prediction reliability, enabling adaptive use of computation.

  • Motivation: Existing PRMs provide step-level feedback but expose only a single point estimate, leaving downstream methods without a signal for whether the score is trustworthy.These scores guide candidate selection, policy optimization, test-time scaling, and reinforcement learning.
  • Motivation: A scalar reward cannot represent uncertainty about whether a seemingly correct prefix will lead to a correct final answer.Causal PRMs evaluate prefixes before observing future continuations.
  • Motivation: Monte Carlo success ratios are noisy finite-sample estimates, but standard PRM training regresses to K/N as a point label.Repeated continuation sampling from the same prefix can produce different success counts.
  • BetaPRM: BetaPRM predicts a Beta belief whose mean µ is the process reward and whose concentration κ measures the reliability of that prediction.The belief is trained to explain Monte Carlo observations rather than treating the observed ratio as exact.
  • Adaptive Computation Allocation: Adaptive Computation Allocation stops when a high-reward answer is reliably ahead and continues computation from uncertain prefixes otherwise.ACA applies the learned reliability signal to progressive-batch Best-of-N reasoning.
  • Results: +3.37 points on average on InternVL2.5-8B accompanies improved PRM-guided Best-of-N selection across four backbones and four benchmarks.BetaPRM preserves standard step-level error detection, while ACA reduces token usage by up to 33.57% and raises final-answer accuracy.

2 Related Work

Prior work uses PRMs for step judging, prefix-success estimation, and fixed-budget test-time scaling. BetaPRM instead makes reliability part of the PRM output for downstream decisions.

  • Process Reward Models: PRMs provide step-level feedback, unlike outcome reward models that score only final answers.Prior approaches include local error detection and Q-value-style estimates of whether a prefix can be completed correctly.
  • Process Reward Models: Existing methods often collapse finite-sample evidence from Monte Carlo continuations into a single point label.BetaPRM retains reliability as part of the PRM output.
  • Test-Time Scaling: Best-of-N samples multiple candidate solutions and selects one with a verifier or reward model, but most methods allocate a fixed budget to every problem.Recent methods calibrate PRM success estimates to choose instance-specific sampling budgets.

3 Preliminaries

PRMs evaluate each reasoning prefix causally, before future steps are observed, by estimating the probability that its continuation yields a correct final answer. Monte Carlo continuation counts provide noisy supervision for this latent quantity.

  • Prefix-Conditioned Process Rewards: A PRM inserts a process marker after each reasoning step and produces a score at every marker position.The input sequence interleaves reasoning steps with <prm> markers.
  • Prefix-Conditioned Process Rewards: The score at step t uses the prefix c_t=(x,s≤t) without access to future steps.This matches online evaluation during generation or search.
  • Prefix-Conditioned Process Rewards: Process-reward quality is defined as the latent prefix success probability q_t=Pr(final answer is correct | x,s≤t).This quantity describes whether the current partial reasoning state can lead to a correct answer.
  • Monte Carlo Step Supervision: Sampling N continuations and counting K_t successful completions yields the Monte Carlo estimate q̂_t=K_t/N.The underlying prefix success probability is unobserved.
  • Monte Carlo Step Supervision: Standard PRM objectives optimize cross-entropy against q̂_t, treating the empirical ratio as the latent probability itself.The predicted step score is denoted p_t.
  • Monte Carlo Step Supervision: Because repeated sampling can change K_t, count-based supervision is preferred over fitting a single noisy point estimate.The finite continuation sample can produce different observed ratios for the same prefix.

4 BETAPRM

BETAPRM models each reasoning prefix with a Beta belief whose mean represents success probability and whose concentration represents reliability. It trains this belief on finite continuation counts using a Beta-Binomial likelihood, with auxiliary regularization to calibrate concentration.

  • 4.1 Beta-Binomial Count Model: BETAPRM models prefix success probability with a Beta belief parameterized by mean µt and concentration κt.µt is the expected success probability and standard PRM score, while κt controls how sharply the belief concentrates around that mean.
  • 4.1 Beta-Binomial Count Model: The Beta-Binomial formulation assigns likelihood to observed continuation counts rather than regressing to a finite-sample point target.A concentrated belief near the observed ratio is favored, while lower concentration allows a wider range of finite-sample observations.
  • 4.2 BETAPRM Parameterization: BETAPRM preserves the standard PRM reward interpretation by deriving µt from the Yes/No reward-token logits and predicting κt with a separate head.This separates the scalar reward channel from the reliability channel.
  • 4.3 Beta-Binomial Training Objective: The auxiliary regularizer discourages high κt when µt disagrees with the observed Monte Carlo ratio Kt/N.Stop-gradient prevents the regularizer from pulling µt toward the noisy ratio and instead focuses calibration on concentration.
  • 4.3 Beta-Binomial Training Objective: The overall training objective combines the Beta-Binomial loss with the auxiliary reliability regularizer.The objective is L = LBeta-Binomial + Lreg.

5 Reliability-Aware Inference: Adaptive Computation Allocation

Adaptive Computation Allocation uses BETAPRM reward and reliability estimates to allocate Best-of-N computation progressively. It stops when the leading candidate is reliably ahead and otherwise repairs uncertain competitive prefixes.

  • Risk-Adjusted Candidate Score: ACA ranks complete candidates with risk-adjusted scores that subtract an uncertainty penalty from predicted process quality.The step uncertainty is the Beta standard deviation, and larger κt produces smaller uncertainty.
  • Progressive Batch Generation and Early Stopping: ACA begins with a small candidate pool, scores it with BETAPRM, and progressively allocates batches up to the maximum budget N.This replaces standard Best-of-N generation of all N candidates in one shot.
  • Progressive Batch Generation and Early Stopping: At each stage, ACA selects the highest-scoring candidate and constructs lower and upper confidence bounds for the stopping test.The bounds use cstop to scale their width.
  • Progressive Batch Generation and Early Stopping: ACA stops and returns the winner when its pessimistic score exceeds the optimistic score of every competitor.Under this criterion, further continuations are unlikely to change the PRM-guided selection.
  • Uncertainty-Guided Prefix Repair: If stopping fails, ACA allocates the next batch to the non-winning candidate with the highest UCB and selects an early low-quality step for prefix repair.This targets computation where additional sampling is most likely to change the current decision.

6 Experiments

Experiments show that BETAPRM improves PRM-guided candidate selection while preserving step-level error detection, and that its learned reliability supports more efficient adaptive computation allocation.

  • BETAPRM Evaluation: BETAPRM improves Best-of-N selection across four backbones and four benchmarks, with average gains of +1.29, +1.46, +3.37, and +2.66 points.The gains correspond to InternVL3-14B, InternVL3-8B, InternVL2.5-8B, and Qwen2.5-VL-7B, respectively.
  • BETAPRM Evaluation: BETAPRM preserves standard step-level error-detection ability, with overall micro-F1 comparable to standard PRM across evaluated backbones.It matches PRM on InternVL3-14B, improves slightly on InternVL3-8B and Qwen2.5-VL-7B, and is slightly lower on InternVL2.5-8B.
  • BETAPRM Evaluation: The auxiliary evidence regularizer improves all four Best-of-16 benchmarks by an average of +1.02 points.With stop-gradient, the regularizer calibrates concentration rather than turning into another soft-label regression objective.
  • BETAPRM Evaluation: During training, concentration κt first decreases and later recovers, assigning higher confidence to prefixes whose reward estimates are better supported.The 90th percentile recovers more strongly than the mean, indicating an upper tail of substantially higher-confidence prefixes.
  • ACA Improves the Inference-Time Accuracy-Token Tradeoff: ACA improves the accuracy–token tradeoff across both evaluated backbones, saving 16.76%–33.57% tokens on InternVL2.5-8B and 19.39%–33.00% on Qwen2.5-VL-7B.It combines uncertainty-guided expansion with confidence-based stopping under the same maximum budget of N = 16 candidate generations.
  • ACA Improves the Inference-Time Accuracy-Token Tradeoff: Learned uncertainty gives ACA the best accuracy–token tradeoff, whereas reward-only allocation often uses fewer tokens but incurs a clear accuracy cost.The µ-only variant cannot identify prefixes whose rewards are high but uncertain enough that additional continuations could change the selected answer.

7 Conclusion

The paper introduces BETAPRM, which makes process rewards reliability-aware by predicting a Beta belief over prefix success from Monte Carlo observations. Its reliability signal improves Best-of-N selection and enables ACA to increase final-answer accuracy while reducing inference tokens.

  • BETAPRM represents each reasoning prefix with a Beta belief over success probability and trains it from Monte Carlo observations using a Beta-Binomial objective.
  • BETAPRM predicts both prefix success probability and a learned reliability estimate, turning scalar process rewards into reliability-aware signals.
  • BETAPRM improves PRM-guided Best-of-N selection without sacrificing step-level error detection.
  • 33.57%: Adaptive Computation Allocation reduces inference tokens while improving final-answer accuracy using BETAPRM's reliability signal.

A.1 Training Data and Backbones

The experiments train PRMs on Monte Carlo continuation counts and compare BetaPRM with a standard PRM across four multimodal backbones. The dataset provides raw count supervision and broad visual-reasoning coverage.

  • Training data: VisualPRM400K-v1.1 records K successful continuations among N = 16 samples for each supervised reasoning prefix.BetaPRM trains on the count pair (K, N), whereas the standard PRM uses K/N as a soft label.
  • Training data: After filtering, the training split contains 565,096 rollouts and 3,174,394 annotated steps.The dataset spans 38 subsets covering diagram, chart, document, visual-question-answering, science, mathematical, and geometry reasoning.
  • Backbones: BetaPRM and the standard PRM are instantiated with InternVL2.5-8B, InternVL3-8B, InternVL3-14B, and Qwen2.5-VL-7B backbones.Both models supervise marker positions after reasoning steps; BetaPRM adds a linear head to predict concentration κt.
  • Training objectives: The standard PRM uses cross-entropy over Yes/No reward tokens, while BetaPRM uses a Beta-Binomial loss and adds the concentration head.Shared backbones use the same optimization recipe, with native image preprocessing for Qwen2.5-VL-7B.

A.3 Best-of-N Evaluation Protocol

The Best-of-N protocol compares PRM-guided candidate selection using shared candidate pools. Standard PRM ranks candidates by mean reward, while BetaPRM additionally uses predicted concentration-derived uncertainty and a risk-budget selector.

  • Evaluation setup: All PRM selectors use the same candidate pools, isolating differences due to the reward model and selection rule.
  • Candidate scoring: Each candidate is formatted with a <prm> marker after every reasoning step.The input consists of the question, process steps, and marker positions used for scoring.
  • Candidate scoring: The standard PRM ranks candidates by average normalized Yes probability, while BetaPRM also extracts concentration κt to compute Beta uncertainty.
  • Selection rule: BetaPRM candidates are ranked with the risk-budget selector used in the main experiments.The penalty weight λ and uncertainty threshold τ are selected from fixed grids shared across reported runs.

A.4 VisualProcessBench Evaluation Protocol

The protocol evaluates both step-level error detection and adaptive Best-of-16 computation. VisualProcessBench uses risk-adjusted step scores, while ACA expands computation when its stopping criterion is not satisfied.

  • VisualProcessBench: VisualProcessBench converts one marker score per rationale step into correct-versus-erroneous predictions, ignoring neutral labels.The evaluation uses the same question-and-rationale input format as PRM training.
  • VisualProcessBench: BETAPRM computes σt = sqrt(µt(1 − µt)/(κt + 1)) and evaluates the risk-adjusted score st = µt − λσt with λ = 0.5.This scores uncertain positive-looking steps more conservatively.
  • VisualProcessBench: A single global threshold τcls is selected per model by maximizing overall validation F1, with overall and per-source macro-F1 reported.
  • Adaptive computation allocation: ACA starts with n0 = 4 candidates and adds batches of m = 4 up to N = 16 under the same maximum budget as fixed-budget Best-of-16.New candidates use InternVL2.5-8B and the fixed-budget baseline’s decoding parameters.
  • Adaptive computation allocation: ACA uses confidence bounds with cstop = 0.3, expands the highest-UCB non-winner competitor, and applies prefix repair below pbad = 0.3.Prefix repair uses ccut = 1.0 in the conservative score µt − ccutσt.
  • Ablation: Main ACA selection uses the risk-budget selector SRB, while the Table 5 ablation uses shared linear score Slin and sets σt = 0 for Standard PRM.This keeps the ablation focused on the source of uncertainty.

B Limitations

BetaPRM experiments require supervision preserving Monte Carlo success counts, and the evaluated availability constraint leads the experiments to focus on multimodal PRMs. Applications beyond evaluated benchmarks require additional oversight and domain-specific checks because learned reliability is not a guarantee of correctness.

  • BETAPRM requires supervision that preserves Monte Carlo continuation counts rather than only binarized step labels.
  • Experiments focus on multimodal PRMs because VisualPRM400K-v1.1 is described as the only publicly available dataset reporting successful-continuation counts for each prefix.The Beta-Binomial formulation itself is not tied to multimodal inputs.
  • Beyond evaluated benchmarks, learned reliability is an additional signal rather than a guarantee of correctness, so high-stakes applications need human oversight, calibration checks, and domain-specific evaluation.
Loading 2605.15529v1…