PaperlayerView exact revision on arXiv

Source-linked AI summary

Process Reward Modeling with Reliability- and Conservatism-Aware Bayesian Belief

Zhongjian Qiao, Zhixiang Zhou, Jiafei Lyu, Peisong Wang, Qiaosheng Zhang, Siyang Gao, Shuang Qiu

arXiv:2610.23223v1cs.LG

TL;DR

PRMs can misestimate rewards because supervision may be insufficient or the supervision policy may differ from the test-time policy, and prior work has not jointly addressed both uncertainties. BayesianPRM represents reward hypotheses with a belief and updates it for reliability and conservatism; experiments report more accurate reward estimates and improved test-time scaling performance.

  • Problem

    PRMs can give inaccurate process rewards, while prior approaches have not comprehensively addressed uncertainty from both limited supervision and policy mismatch.

    Source: §1 · pp. 1–2

  • Method

    BayesianPRM discretizes reward hypotheses with an ensemble and updates a Bayesian belief through reliability-aware updating and conservatism-aware calibration.

    Source: §1 · p. 2, §4.1 · p. 4

  • Results

    Experiments across multimodal reasoning benchmarks indicate that BayesianPRM consistently improves process reward prediction and test-time scaling performance.

    Source: §1 · p. 2, §6 · p. 10

  • Takeaways & Limitations

    The paper's results support jointly addressing PRM and policy uncertainty in process reward modeling for test-time scaling.

    Source: §1 · p. 2, §6 · p. 10

    Calibration requires manually specified hyperparameters, and its one-sided correction is most effective when prediction error is dominated by overestimation.

    Source: H LIMITATIONS · p. 31

Abstract

from arXiv · show

Process Reward Models (PRMs) play a central role in test-time scaling for large language model (LLM) reasoning. However, PRMs may provide inaccurate process rewards for test-time scaling, which we identify as arising from two sources of uncertainty: (1) \textit{PRM uncertainty} that arises from insufficient supervision data and imperfect PRM training; and (2) \textit{policy uncertainty} that results from the mismatch between the policy to collect supervision data and the policy for test-time scaling, leading to systematic reward overestimation. Previous studies fail to jointly consider both uncertainties. To bridge this gap, we propose BayesianPRM, which treats the process reward as a random variable within a reward hypothesis space and maintains a Bayesian belief over it. Since exact Bayesian inference over the continuous hypothesis space is generally intractable, we discretize the reward hypothesis space with a reward ensemble and introduce two complementary belief updates, each targeting one source of uncertainty: (1) to address PRM uncertainty, we introduce \textit{reliability-aware belief updating} that measures how well each ensemble member explains the supervision data and updates the belief with an amortized belief network optimized by the evidence lower bound (ELBO); and (2) to tackle policy uncertainty, we propose \textit{conservatism-aware belief calibration} which further reweights the belief toward lower rewards to suppress overestimation. These two updates yield a reliability- and conservatism-aware Bayesian belief, whose expectation determines the process reward. Experiments on multimodal reasoning benchmarks indicate that BayesianPRM consistently yields more accurate process reward estimates while enhancing test-time scaling performance.

1 INTRODUCTION

PRMs support test-time scaling but can misestimate rewards, and prior work has not jointly addressed the uncertainty arising from limited supervision and policy mismatch. BayesianPRM combines two belief updates for these sources, and experiments report more accurate reward estimates and stronger test-time scaling performance.

1 INTRODUCTION

  • PRM uncertainty comes from insufficient supervision or imperfect training, while policy uncertainty arises when supervision and test-time policies differ.A stronger policy may generate supervision labels than the weaker policy used at test time, contributing to systematic reward overestimation.Source: §1 · pp. 1–2
  • Earlier calibration methods target policy uncertainty, whereas BetaPRM models PRM uncertainty, leaving the two sources unaddressed together.The passage identifies QR-PRM and OT-PRM as addressing policy uncertainty, and BetaPRM as leaving policy uncertainty unaddressed.Source: §1 · p. 2
  • BayesianPRM treats process reward as a random variable over hypotheses and uses reliability-aware updating followed by conservatism-aware calibration.An ensemble discretizes the reward hypothesis space; an amortized belief network is trained with an evidence lower bound, and calibration reweights belief toward lower rewards.Source: §1 · p. 2
  • Experiments across multimodal reasoning benchmarks indicate more accurate process reward estimates and improved test-time scaling performance.The introduction summarizes these as consistent experimental outcomes.Source: §1 · p. 2

2 PRELIMINARIES

A PRM estimates the probability that a reasoning prefix will eventually lead to a correct answer, typically learning from Monte Carlo supervision. Bayesian learning offers a framework for representing uncertainty over hypotheses and marginalizing predictions.

2 PRELIMINARIES

  • A PRM predicts a process reward for a reasoning prefix to approximate the probability that the policy will eventually complete it correctly.The prefix comprises the input problem and reasoning steps up to the current step.Source: §2 · p. 2
  • PRMs learn from Monte Carlo estimates formed by sampling continuations and counting how many reach a correct answer.For each prefix, the training data records the number of successful continuations among the sampled continuations.Source: §2 · p. 2
  • Bayesian learning maintains a belief over model hypotheses and can obtain a prediction by marginalizing across plausible hypotheses.The framework begins with a prior and updates to a posterior after observing data.Source: §2 · pp. 2–3
  • Figure 1 defines prediction error as predicted reward minus empirical success probability and separates PRM-uncertainty and policy-uncertainty examples.Its panels cover MathV360K and Geometry3K for PRM uncertainty, and UniGeo and MathVision for policy uncertainty.Source: Figure 1 · p. 3

3 ANALYSIS OF TWO SOURCES OF UNCERTAINTY

The analysis separates PRM prediction error under the supervision policy from changes in success probability caused by a policy mismatch. Experiments show error variance and larger OOD errors for PRM uncertainty, while policy uncertainty appears as positive bias, especially on OOD MathVision.

3 ANALYSIS OF TWO SOURCES OF UNCERTAINTY

  • PRM uncertainty is prediction error under the supervision policy; policy uncertainty is the change in true continuation success probability when the test-time policy differs.The former can reflect insufficient supervision or imperfect training, while the latter reflects mismatch between rollout policies.Source: §3 · p. 3
  • PRM prediction errors are centered near zero on ID MathV360K but substantially larger on held-out OOD Geometry3K.The evaluation removes policy uncertainty, so the observed errors illustrate PRM uncertainty, more pronounced on unseen problems.Source: §3 · p. 3
  • Prediction error has positive bias under both ID UniGeo and OOD MathVision, with more pronounced reward overestimation on MathVision.The study estimates continuation success using a weak rollout policy and evaluates prefixes under ID and OOD settings.Source: §3 · pp. 3–4

4 METHODOLOGY

BayesianPRM represents process reward as a belief over reward hypotheses and combines supervision-based reliability with conservative reweighting toward lower rewards. Its finite reward ensemble makes the belief-weighted prediction tractable, while the method also gives a continuous-space interpretation and a staged implementation.

4.1 PROCESS REWARD MODELING AS BAYESIAN LEARNING

  • The posterior process reward is the belief-weighted average of hypothesis-specific success probabilities, with a finite reward ensemble replacing intractable continuous inference by a tractable weighted sum.Each hypothesis predicts the probability that a continuation from a reasoning prefix reaches a correct answer.Source: §4.1 · p. 4

4.2 RELIABILITY- AND CONSERVATISM-AWARE BAYESIAN BELIEF

  • A belief network estimates each ensemble member’s reliability from validation supervision likelihoods, with an ELBO-trained amortized update approximating a Bayesian posterior.The likelihood measures how well each member’s predicted success probability explains the supervision data.Source: §4.2 · pp. 4–5
  • Conservatism-aware calibration uses the reliability-aware belief as its prior and shifts belief mass toward lower-reward hypotheses to mitigate overestimation.The resulting belief weights ensemble predictions to produce the final process reward.Source: §4.2 · pp. 5–6

4.3 CONNECTION WITH CONTINUOUS HYPOTHESIS SPACE

  • The finite-ensemble formulation can be interpreted as an implicit Bayesian update that reshapes a prior over continuous hypotheses into a continuous posterior.The posterior is a weighted mixture of continuous components concentrated around particular quantiles.Source: §4.3 · p. 6

4.4 PRACTICAL IMPLEMENTATION

  • The implementation shares an LLM backbone across reward heads and a belief network, trains the belief network on validation data, then combines its belief with ensemble predictions at inference.The backbone and reward heads are frozen while the belief network is trained with the ELBO.Source: §4.4 · p. 7

5 EXPERIMENTS

BayesianPRM improves process-reward calibration and test-time scaling across multimodal benchmarks. Ablations find that both reliability-aware updating and conservatism-aware calibration contribute to these outcomes.

5.2 TEST-TIME SCALING PERFORMANCE

  • With Best-of-N (N = 16), BayesianPRM achieves 56.50 average accuracy; with IAS (Nmax = 16), it achieves 56.07, leading the compared methods in both settings.It is best on MathVision, MathVista, and MMStar under Best-of-N, and on MathVision, MathVerse-VO, and MMStar under IAS.Source: Table 3 · p. 8, §5.2 · p. 9

5 EXPERIMENTS

  • Evaluation uses VisualPRM400K-v1.1 for training and four multimodal benchmarks, assessing reward calibration and PRM-guided test-time scaling.BayesianPRM uses an 80/20 training-validation split; baselines train on the full dataset.Source: §5 · p. 7, §5.1 · p. 8, §5.2 · p. 9

5.1 PROCESS REWARD ACCURACY EVALUATION

  • BayesianPRM reduces average Brier Score, Positive Brier Score, ECE, and AverageCE by 17.2%, 24.1%, 27.7%, and 19.2%, respectively, versus Standard PRM.It also improves calibration on MathVista, where Standard PRM already has relatively small prediction errors.Source: §5.1 · p. 8

5.3 ABLATION STUDY

  • Removing reliability-aware belief updating worsens all four calibration metrics and consistently lowers Best-of-16 performance across four datasets.Figure 2 compares MathVision calibration with and without the update; Table 4 reports Best-of-16 results.Source: §5.3 · p. 9, Figure 2 · p. 9, Table 4 · p. 9
  • Removing conservatism-aware belief calibration degrades both MathVision calibration and test-time scaling across four benchmarks.Figure 3 reports calibration results, while Table 5 reports Best-of-16 results.Source: §5.3 · p. 9, Figure 3 · p. 9, Table 5 · p. 9

6 CONCLUSION

The conclusion identifies PRM uncertainty and policy uncertainty as sources of process-reward uncertainty and presents BayesianPRM as addressing both.

6 CONCLUSION

  • BayesianPRM combines reliability-aware belief updating and conservatism-aware belief calibration, and experiments indicate improved reward prediction and test-time scaling on multimodal benchmarks.Source: §6 · p. 10

7 AI USE STATEMENT

The authors describe their use of LLMs for manuscript presentation and parts of the experimental code.

7 AI USE STATEMENT

  • LLMs assisted with grammar, presentation, and parts of the experimental code; the authors state they were not involved in core research contributions and that assisted content was reviewed.Source: §7 · p. 10

8 ETHICS STATEMENT

The authors state that they believe the study raises no ethical concerns.

8 ETHICS STATEMENT

  • The study does not involve human or animal experiments, sensitive datasets, or applications the authors believe may pose ethical risks.Source: §8 · p. 10

9 REPRODUCIBILITY STATEMENT

BayesianPRM is presented as a distributional PRM that updates beliefs over reward hypotheses, unlike prior ensemble methods that reweight members or use disagreement. The proof shows that a reliability-aware belief update recovers the standard Bayesian posterior when β1 = 1.

9 REPRODUCIBILITY STATEMENT

  • The authors state that they provide BayesianPRM training scripts and detailed experimental and implementation setups in the appendices.Source: §9 · p. 10

A RELATED WORK

  • BayesianPRM uses Bayesian belief updating at training and test time, whereas HyRe reweights ensembles at test time and ActivePRM uses ensemble disagreement for active learning.The related-work section also characterizes BayesianPRM as a distributional PRM whose posterior belief represents probabilities over reward hypotheses.Source: A RELATED WORK · p. 16

B.1 PROOF OF PROPOSITION 4.1

  • For β1 > 0, the reliability-aware belief objective has a unique global optimum; setting β1 = 1 recovers the standard Bayesian posterior update.The proof attributes strict concavity to the KL term.Source: §B.1 · p. 17

B.2 PROOF OF PROPOSITION 4.2

Proposition 4.2 constructs a posterior belief whose expected reward matches a weighted expectation over reward hypotheses ranked by their scores. The derivation uses order statistics to express that posterior as a reweighted prior.

B.2 PROOF OF PROPOSITION 4.2

  • Order statistics give the prior density of a hypothesis occupying rank m, reweighted by the probability that its score falls at that rank.The score distribution is induced by sampling hypotheses from the prior.Source: §B.2 · p. 17
  • The coefficients ᾱm(θ; ct) represent the expected belief weight for hypothesis θ when its score ranks m, matching a weighted expectation over prior-sampled hypotheses.Source: §B.2 · p. 18
  • Comparing the weighted-expectation and posterior integral forms yields a likelihood ratio that reweights the prior into the proposed posterior belief.The proof verifies that the reweighted belief integrates to one and is therefore a valid probability distribution.Source: §B.2 · pp. 18–19

C.1 BACKBONES AND DATASETS

The appendix describes the shared multimodal backbone, training and evaluation data, metrics, and comparison methods. It specifies both BayesianPRM-specific supervision requirements and the calibration procedures used for baselines.

C.1 BACKBONES AND DATASETS

  • All methods use InternVL3-8B, with BayesianPRM adding lightweight linear reward heads; its vision encoder is frozen while the language model and added layers are fine-tuned.A complete fine-tuning run takes approximately 20 hours on 4×H100 GPUs.Source: §C.1 · p. 19
  • VisualPRM400K-v1.1 supplies empirical success rates and success counts from 16 rollouts, which BayesianPRM needs to compute its binomial likelihood.The reward ensemble trains on 80% of the data and the belief network on the held-out 20%; other baselines use the full dataset.Source: §C.1 · p. 19
  • Evaluation uses prefixes from MathVision, MathVerse, MathVista, and MMStar, with 1,000 sampled prefixes per benchmark and 16 judged rollouts per prefix.Source: §C.1 · pp. 19–20

C.2 EVALUATION METRICS

  • PRM accuracy is assessed with Brier Score, Positive Brier Score, ECE, and AverageCE; lower values indicate better performance.Positive Brier specifically penalizes overestimation, while AverageCE weights each non-empty prediction bin equally.Source: §C.2 · p. 20

C.3 BASELINES

  • BetaPRM models rewards with a Beta distribution and trains with a Beta-Binomial loss; DreamPRM uses bi-level optimization to reweight training domains.Source: §C.3 · pp. 20–21
  • QR-PRM adapts to a target policy through calibration-set quantile regression, while IAS allocates larger sampling budgets to more difficult questions.The described IAS setup uses a conservative lower quantile, τ = 0.1.Source: §C.3 · pp. 21–22
  • Temperature scaling adjusts probability sharpness while preserving prediction ranking; isotonic regression learns a non-decreasing mapping from predictions to empirical success rates.Source: §C.3 · p. 22
  • Histogram binning replaces each prediction with the observed empirical success rate in its prediction bin, while self-consistency selects the majority-vote answer without an external reward model.Source: §C.3 · pp. 22–23

D IMPLEMENTATION DETAILS

BayesianPRM combines an ensemble with a learned reliability belief and a conservative inference-time adjustment. Additional experiments report performance across a different rollout backbone, a step-level benchmark, and calibration-set comparisons.

D IMPLEMENTATION DETAILS

  • The implementation uses an InternVL3-8B reward ensemble with fixed randomized prior networks to encourage diversity, and a belief network that estimates prefix-dependent reliability over reward hypotheses.The belief network takes the prefix representation and ensemble predictions as input; its training freezes the backbone and reward heads.Source: D IMPLEMENTATION DETAILS · p. 23
  • The reward ensemble is trained with cross-entropy, then the frozen ensemble and backbone feed a belief network trained on held-out data with the ELBO objective.Source: D IMPLEMENTATION DETAILS · p. 23
  • At inference, a conservatism-aware adjustment produces a hybrid belief whose weighted ensemble reward is averaged across trajectory steps for test-time scaling.Source: D IMPLEMENTATION DETAILS · pp. 23–24

E.1.1 PROCESS REWARD ACCURACY EVALUATION

  • With Qwen3-VL-8B-Instruct as rollout policy, BayesianPRM achieves the best PRM accuracy across four benchmarks and all metrics; average Brier Score falls from 0.2491 to 0.2205 versus Standard PRM.Average ECE falls from 0.2258 to 0.1582, and the method also outperforms BetaPRM and DreamPRM across metrics.Source: Table 7 · p. 25, §E.1.1 · p. 26

E.1.2 TEST-TIME SCALING PERFORMANCE

  • Under Best-of-N with N = 16, BayesianPRM reaches average accuracy 53.05, beating Standard PRM by 0.81 points; under IAS with Nmax = 16, its 52.60 average is best across all four benchmarks.The first result is best on three benchmarks and second on MathVision; the IAS result is best on all four.Source: Table 8 · p. 26, §E.1.2 · p. 26

E.2 EVALUATION ON VISUALPROCESSBENCH

  • BayesianPRM scores 62.23 overall on VisualProcessBench, 2.44 points above Standard PRM and 1.40 above BetaPRM.It performs best on four of five subsets; on DynaMath it is 0.21 points below BetaPRM.Source: Table 9 · p. 27, §E.2 · p. 27

E.3 EVALUATION WITH CALIBRATION SETS

  • With calibration sets, BayesianPRM achieves the best results on every benchmark across four accuracy metrics; versus QR-PRM, average Brier Score drops from 0.1668 to 0.1231.Average ECE drops from 0.1869 to 0.1264; post-hoc calibration methods generally underperform QR-PRM and BayesianPRM across most metrics.Source: §E.3 · p. 27, Table 10 · p. 28
  • With calibration sets, BayesianPRM achieves average accuracy 56.75 with Best-of-N (N = 16), exceeding QR-PRM by 0.74 points, and 56.14 under IAS (Nmax = 16), by 0.64 points.Source: §E.3 · p. 28

E.4 EXTENDED ABLATION STUDY

The extended ablations find that BayesianPRM outperforms a vanilla ensemble PRM, while performance depends on the belief-update and ensemble settings. The paper also reports training costs, illustrates step-wise rewards, and identifies limits of manual, one-sided calibration.

E.4 EXTENDED ABLATION STUDY

  • BayesianPRM consistently outperforms a vanilla ensemble PRM on test-time scaling, supporting the value of belief updating and calibration.Table 12 compares Best-of-N (N = 16) on MathVision, MathVerse-VO, MathVista, and MMStar.Source: §E.4 · p. 28, Table 12 · p. 29
  • β1 values that are too small or large reduce performance; small values concentrate belief on Head 2, while β1 = 0.1 leaves it too evenly distributed, motivating β1 = 0.05.The study varies β1 over {0.001, 0.01, 0.05, 0.1}; Figure 4 visualizes belief evolution with five reward heads.Source: §E.4 · pp. 28–29, Figure 4 · p. 29, Table 13 · p. 30
  • Both overly strong and weak conservative reweighting degrade performance; larger β2 values are more favorable on MMStar, so the optimal value can vary by dataset.The experiments use β2 = 0.05 unless otherwise specified, rather than tuning it for each benchmark.Source: §E.4 · pp. 29–30, Table 14 · p. 30
  • Performance starts to fluctuate when M > 20, while λprior values that are too small or large degrade performance; the chosen settings are M = 20 and λprior = 10.M controls the number of reward heads; λprior controls the random prior network's contribution to each head.Source: §E.4 · p. 30, Table 15 · p. 30, Table 16 · p. 30

F COMPUTATIONAL OVERHEAD

  • Less than 20 hours are required for BayesianPRM’s full training pipeline, versus approximately 24 hours for one Standard PRM epoch on 4 × H100 GPUs.The pipeline trains the reward ensemble on 80% of the data and the lightweight belief network on the remaining 20% with the backbone frozen; BayesianPRM has 0.48% parameter overhead.Source: F COMPUTATIONAL OVERHEAD · p. 31

G CASE STUDY

  • At the first erroneous step, Step 8, the reward drops from 0.977 to 0.041; the correct trajectory receives consistently high rewards across its steps.The case study presents the correct and incorrect trajectories in Figure 5.Source: G CASE STUDY · p. 31, Figure 5 · p. 32

H LIMITATIONS

  • Calibration strength must be specified manually, and its one-sided shift toward lower rewards is most effective when prediction error is dominated by overestimation.The degree of overestimation and optimal hyperparameter values can vary across datasets and test-time rollout policies.Source: H LIMITATIONS · p. 31
Loading 2610.23223v1…