Source-linked AI summary
Process Reward Model with Q-Value Rankings
Wendi Li, Yixuan Li
TL;DR
Classification-based process reward models evaluate intermediate steps independently, leaving their dependencies and relative contributions unaccounted for. PQM frames process reward modeling as an MDP-grounded Q-value ranking problem trained with a comparative loss. It outperforms classification-based PRMs in reported evaluations, including on MATH500.
Problem
Method
PQM frames PRM as an MDP-grounded Q-value ranking problem and trains models to approximate optimal rankings with a comparative loss.
Source: §1 · pp. 1–2Results
PQM outperforms classification-based PRMs across reported settings, improving Llama-3-70B-Instruct MATH500 verification accuracy from 39.8% (BCE) to 51.4%.
Source: §1 · p. 2, §6 · p. 10Takeaways & Limitations
The framework gives intermediate-step rewards a Q-value interpretation and includes prior classification-based PRMs as a special case under certain conditions.
Source: §1 · p. 2Math-Shepherd data quality is restricted by noise from automatic annotation.
Source: F LIMITATIONS & FUTURE WORKS · p. 17
Abstract
from arXiv · showhide
Process Reward Modeling (PRM) is critical for complex reasoning and decision-making tasks where the accuracy of intermediate steps significantly influences the overall outcome. Existing PRM approaches, primarily framed as classification problems, employ cross-entropy loss to independently evaluate each step's correctness. This method can lead to suboptimal reward distribution and does not adequately address the interdependencies among steps. To address these limitations, we introduce the Process Q-value Model (PQM), a novel framework that redefines PRM in the context of a Markov Decision Process. PQM optimizes Q-value rankings based on a novel comparative loss function, enhancing the model's ability to capture the intricate dynamics among sequential decisions. This approach provides a more granular and theoretically grounded methodology for process rewards. Our extensive empirical evaluations across various sampling policies, language model backbones, and multi-step reasoning benchmarks show that PQM outperforms classification-based PRMs. The effectiveness of the comparative loss function is highlighted in our comprehensive ablation studies, confirming PQM's practical efficacy and theoretical advantage.
1 INTRODUCTION
Classification-based PRMs evaluate steps independently, while PQM frames process reward modeling as Q-value ranking to capture dependencies among steps. Experiments report higher verification accuracy than classification-based PRMs across varied settings.
1 INTRODUCTION
- Prior PRM research typically classifies each intermediate state as correct or incorrect using cross-entropy, treating states independently (Wang et al., 2023a; Shao et al., 2024; Lightman et al., 2024; Luo et al., 2024).The authors identify this approach as lacking a theoretical grounding for how it approximates the desired reward function.Source: §1 · p. 1
- PQM frames process reward modeling as a Q-value ranking problem grounded in a Markov Decision Process.It defines each state-action Q-value as the probability of reaching the correct final answer and trains PRMs to approximate optimal rankings with a comparative loss.Source: §1 · pp. 1–2
- Under the framework, Q-values rise as correct steps continue and fall as wrong steps proceed, with a prominent gap between them.The paper also casts classification-based PRMs as a special case under certain conditions.Source: §1 · p. 2
- On MATH500, PQM raises Llama-3-70B-Instruct verification accuracy from 39.8% (BCE) to 51.4%, an 11.6% improvement.The authors report consistent results across datasets, sampling policies, and LLM backbones.Source: §1 · p. 2
- The paper reports theoretical analysis, evaluations across sampling policies, LLM backbones, and test sets, and ablations of the comparative training objective.The ablations examine variations of the objective and their impact on performance and design.Source: §1 · p. 2
2 PRELIMINARIES
LLM reasoning generates intermediate steps autoregressively, and process reward models assess those steps rather than only the final answer. The section explains how classification-based PRMs use binary cross-entropy and motivates a Q-value ranking formulation to account for dependencies among steps.
2 PRELIMINARIES
- LLMs generate reasoning steps autoregressively from the instruction and previously generated steps, before producing the final answer.The final answer can itself be considered the last reasoning step.Source: §2 · pp. 2–3
- Outcome reward models score only the final result, whereas process reward models provide feedback on intermediate-step quality in multi-step tasks.The section describes process rewards as a more granular signal for the reasoning path.Source: §2 · p. 3
- Classification-based PRMs use binary cross-entropy to predict each reasoning state's correctness, but this treats states independently and lacks theoretical support for the PRM formulation (Wang et al., 2023a; Shao et al., 2024; Lightman et al., 2024; Luo et al., 2024).The section motivates Q-value ranking as a way to evaluate relative step quality and capture dependencies among steps.Source: §2 · p. 3
3 PQM: PROCESS REWARD MODEL WITH Q-VALUE RANKINGS
PQM reframes process reward modeling as Q-value ranking in a deterministic MDP for LLM text generation. It defines Q-values by final-answer success, derives their ordering, and trains them with a margin-aware comparative loss.
3.1 DETERMINISTIC MDP FOR LLMS
- LLM generation is modeled as a deterministic MDP: the state contains the instruction and prior text, the action is the next text piece, and the reward scores that action given its history.Source: §3.1 · p. 4
3.2 DEFINING Q-FUNCTION IMPLICITLY DEFINES A REWARD FUNCTION
- Q-values represent the probability of reaching the correct final answer, and under a deterministic MDP the optimal-policy advantage can serve as an equivalent reward.Source: §3.2 · p. 4
3.3 OPTIMAL Q-VALUE RANKING
- Under Assumption 3.1, optimal Q-values rise across correct steps and fall across wrong steps, with a gap between the first correct and first incorrect steps.Source: §3.2 · p. 4, §3.3 · p. 5
3.4 COMPARATIVE LOSS FUNCTION FOR OPTIMIZING Q-VALUE RANKINGS
- The comparative loss emphasizes large Q-value gaps between correct and incorrect steps; its practical variant handles corpora labeling every post-error step wrong and ignores ordering among wrong steps.Source: §3.4 · p. 6
3.5 CLASSIFICATION-BASED PRM IS A SPECIAL CASE OF Q-VALUE APPROXIMATORS
- Under extreme conditions on next-step correctness, classification-based PRMs are a special case, and their cross-entropy loss estimates Q-values without bias.Source: §3.5 · p. 6
4 EXPERIMENTS
The experiments evaluate PQM across policies, backbones, datasets, and ablations, reporting stronger verification than prior baselines and examining factors that affect its performance.
4.1 EXPERIMENTAL SETTINGS
- The experiments use BON@n best-of-n verification on 128 solutions per question from GSM-Plus and MATH500, sampled from three math policy models.Training uses Math-Shepherd; the reward-model comparisons include ORM, BCE, and MSE variants.Source: §4.1 · p. 7
4.2 MAIN RESULTS
- PQM improves accuracy from 39.8% (BCE) to 51.4%, an 11.6% improvement on MATH500 for solutions sampled from Llama-3-70B-Instruct.PQM also consistently outperforms existing methods across MATH500 and GSM-Plus and three policy-model scales.Source: §4.2 · pp. 7–8
- Combining PQM with self-consistency boosts verification performance, with the reported PQM versus SC+PQM gap increasing for larger policy models.The comparison covers three policy models on MATH500 with ζ = 2 and ζ = 4.Source: §4.2 · p. 8, Figure 2 · p. 8
- PQM remains effective across six LLM backbones, and stronger backbones generally yield better performance under the same sampling policy.Qwen2-Math-1.5B-Instruct also performs impressively among the six backbones, indicating effective verification with a small math-specialized backbone.Source: §4.2 · p. 8, Table 2 · p. 8
4.3 FURTHER STUDIES
- ζ values of 2, 4, and 8 perform best, while 0 and 16 perform less effectively; the loss curves flatten after about 200 steps across settings.The authors attribute weaker results at large margins to overemphasizing correct–wrong discrepancies, and at small margins to insufficiently capturing Q-value differences.Source: §4.3 · pp. 8–9
- Because training automatically marks all steps after the first incorrect one as negative, the authors compared their loss with the theoretical version in Eq. 9 and found a marked performance decline, revealing noise in negative annotations.Source: §4.3 · p. 9
- Empirical validation finds correct states more likely to lead to a correct next step or trajectory, while approximated Q-values rise across correct steps and are generally lower for later wrong steps.The reported Q-values are generally over 0.5 for correct steps and below 0.15 for incorrect steps.Source: §4.3 · p. 9, Figure 3 · p. 9, Figure 4 · p. 9
5 RELATED WORKS
Prior PRM work largely treats step correctness as classification, while related approaches use automatic annotations or online search; MDP-based LLM methods still rely partly on outcome rewards.
Process Reward Models
- Most prior PRMs classify each step as correct or incorrect, and the paper frames this approach as a special case within its framework.Automatic annotation approaches can mark all steps after the first error as incorrect; MCTS-based estimates use different Q-value designs and require dense online search.Source: §5 · p. 10
- The authors characterize their comparative loss as easy to use and theoretically unbiased, contrasting it with the cost and potential estimation bias of MCTS-based methods.They attribute potential MCTS bias to distribution shift between the sampling policy and optimal π*.Source: §5 · p. 10
MDP RL for LLMs
- Recent MDP-based RL algorithms for LLMs still rely at least partially on ORMs, while step-level DPO methods are presented as a related, orthogonal direction.The authors discuss possible integration of those preference-optimization methods in an appendix.Source: §5 · p. 10
6 CONCLUSION
The conclusion presents PQM as Q-value ranking for process rewards, using a margin-based comparative objective to capture dependencies among steps.
6 CONCLUSION
- PQM uses Q-value ranking and a margin-based comparative objective, and the authors report an 11.6% accuracy improvement on Llama-3-70B-Instruct solutions from MATH500.They also report robust results across backbone scales, policy models, and datasets.Source: §6 · p. 10
A RELATED WORKS
The appendix outlines an automatic annotation pipeline that samples completions for reasoning steps and labels steps by whether a completion reaches the correct final answer.
A RELATED WORKS
- For questions sampled from ρ, an LLM samples trajectories whose reasoning steps are then annotated for correctness.The pipeline is described as a fundamental version from Wang et al. (2023a).Source: A RELATED WORKS · p. 14
- To annotate a step, the pipeline samples n completions conditioned on the question and preceding steps, then checks each completion’s final answer by string matching.The method marks a step correct if any completion yields the correct answer; otherwise it marks that step and all subsequent steps incorrect.Source: A RELATED WORKS · p. 14
- Other efficiency strategies include training an additional confidence module or using binary search to locate the first error.The appendix attributes these approaches to Lu et al. (2024a) and Luo et al. (2024), respectively.Source: A RELATED WORKS · p. 14
B IMPLEMENTATION DETAILS
The implementation details specify the compute setup and training procedure, alongside materials documenting the ablation settings and results.
B IMPLEMENTATION DETAILS
- Training used eight NVIDIA A100-SXM4-80GB GPUs with ZeRO-3 and bfloat16; grouping sequences by length reduced main-experiment training time from about eleven hours to six.The length groups were under 512, 512–1024, and over 1024 tokens, with batch sizes 64, 24, and 8, respectively.Source: B IMPLEMENTATION DETAILS · p. 14
- Best-of-n trajectories were generated with VLLM at temperature 1, top-p 1, and maximum length 2048.The MCTS baseline used Qwen2-math-7B-Instruct and a training set matched in scale to half the Math-Shepherd corpus.Source: B IMPLEMENTATION DETAILS · p. 14
- Ablation documentation includes experimental settings in Table 5, results in Table 6, and loss curves for Table 3.The Table 6 caption lists MATH500 BON@1 values of 24.4 for MetaMath-Mistral-7B and 37.4 for Llama3-70B-Instruct.Source: Table 5 · p. 14, Table 6 · p. 15, Figure 5 · p. 15
C ADDITIONAL EXPERIMENTS
Additional experiments examine dataset size, verification ceilings, loss variants, ranking behavior, and PRM-guided search. Across these settings, the reported findings support PQM’s performance while identifying remaining gaps and annotation concerns.
Loss variation
- Focusing the ranking loss on the first incorrect step, which is automatically verified, left performance relatively stable, suggesting limited utility for later negative steps.The loss variant is described as emphasizing only the first incorrect step.Source: C ADDITIONAL EXPERIMENTS · p. 15
Comparison with ceiling performance
- On MATH500, current PRMs, including PQM, remain below the best-achievable verification performance shown for three policy models.Figure 6 compares Pass@N with the best achievable verification performance, indicating room to close the gap.Source: Figure 6 · p. 15, C ADDITIONAL EXPERIMENTS · p. 15
Empirical validation for Assumption 3.1 and Theorem 3.5
- For sampled trajectories with more than six steps, estimated Qσ generally rose through correct steps and was higher for correct than incorrect steps.The study sampled 256 trajectories, balanced between correct and incorrect, and used 32 completions per reasoning state; after step five, Qσ was generally over 0.5 for correct steps and below 0.15 for incorrect steps.Source: C ADDITIONAL EXPERIMENTS · pp. 15–16
PRM-guided beam search
- PQM more effectively guides reasoning than BCE-based PRMs in MATH500 beam search with beam size 8 and temperature 0.7.The comparison used Llama-3-8B-Instruct and Eurus-7b-sft policy models; the results are reported in Table 7.Source: C ADDITIONAL EXPERIMENTS · p. 16, Table 7 · p. 16
- In PQM-guided beam-search ablations with Eurus-7b-sft, a sufficiently large ζ range supported strong performance, with optimal values typically in the middle of the range.The ablation results are reported in Table 8.Source: C ADDITIONAL EXPERIMENTS · p. 16, Table 8 · p. 16
Sample-efficiency of PQM
- PQM generally outperforms BCE across 25%, 50%, and 75% dataset sizes and is reported as more sample efficient.Table 9 compares Best-of-n performance for PRMs trained with BCE and PQM losses.Source: C ADDITIONAL EXPERIMENTS · p. 16, Table 9 · p. 16
Comparison of ranking behaviors between PQM and BCE
- BCE scores decreased across the example’s correct steps, while PQM’s Qσ dropped from 0.482 to 0.004 between the last correct and first incorrect steps.The BCE probabilities were 0.916, 0.882, and 0.848 for steps 1–3, then 0.628 for step 4.Source: C ADDITIONAL EXPERIMENTS · pp. 16–17
- Only 29.18% of test solutions shared step rankings between BCE and PQM, and no test questions had identical cross-solution rankings.On 2,048 sampled training cases, the methods produced different rankings in 62.79%.Source: C ADDITIONAL EXPERIMENTS · p. 17
D INTER-SOLUTION COMPARISON
The section extends Q-value ranking from steps within one trajectory to comparisons between trajectories that share a correct prior. It presents rankings for a single divergent step and for successive step pairs.
D INTER-SOLUTION COMPARISON
- Inter-solution comparison considers trajectories that diverge at step t after a shared correct prior, contrasting the correct and wrong next steps.The paper distinguishes this from intra-solution comparison, where compared steps belong to one trajectory.Source: D INTER-SOLUTION COMPARISON · p. 17
- The general corollary treats successive correct–wrong step pairs conditioned on preceding correct steps, extending Q-value rankings across a trajectory.The formulation covers H successive pairs.Source: D INTER-SOLUTION COMPARISON · p. 17
- The discussion says existing step-level methods often compare pairs independently, neglecting integral Q-value rankings within the tree; the corollaries also encompass concurrent offline step-level DPO derivations.The paper notes that some methods use only the wrong-step value and discard intermediate ranking relationships.Source: D INTER-SOLUTION COMPARISON · p. 17
E CASE STUDIES
The case studies report a slight predicted-value increase across correct steps and a sharp decrease upon encountering a wrong step.
E CASE STUDIES
- Predicted Q* values generally rise slightly across correct steps, then drop sharply when a wrong step appears.The examples report the value transition across reasoning steps.Source: E CASE STUDIES · p. 17
F LIMITATIONS & FUTURE WORKS
The discussion identifies dataset quality, remaining performance headroom, and limited study of online RL with PRMs as areas for further work.
F LIMITATIONS & FUTURE WORKS
- Math-Shepherd's automatic-annotation noise restricts its data quality, and the authors suggest that more advanced datasets could improve PRM verification.The authors also note a gap between current PRM performance and the ceiling performance.Source: F LIMITATIONS & FUTURE WORKS · p. 17
- Systematic studies of online RL algorithms that leverage PRMs are lacking, leaving their potential use largely unexplored.Source: F LIMITATIONS & FUTURE WORKS · p. 17