PaperlayerView exact revision on arXiv

Source-linked AI summary

Free Process Rewards without Process Labels

Lifan Yuan, Wendi Li, Huayu Chen, Ganqu Cui, Ning Ding, Kaiyan Zhang, Bowen Zhou, Zhiyuan Liu, Hao Peng

arXiv:2412.01981v1cs.LGcs.CL

TL;DR

PRM training requires costly intermediate-step labels, limiting the accessibility of dense process feedback. The paper shows that parameterizing outcome rewards as policy–reference log-likelihood ratios allows an implicit PRM to be learned during ordinary ORM training from response-level data. On MATH, these implicit PRMs outperform baselines with lower development overhead, while their scope includes assumptions and unresolved opportunities for improved annotation and training algorithms.

  • Problem

    Training PRMs requires labels for every intermediate reasoning step, making data collection substantially more difficult and expensive than response-level ORM labeling.

  • Method

    The paper parameterizes outcome rewards as policy–reference log-likelihood ratios so an ORM can intrinsically provide process rewards during training, across multiple objectives.

  • Results

    Implicit PRMs with varied objectives outperform competitive baselines on MATH with better accuracy–development-overhead trade-offs, with CE especially effective and majority voting providing further gains.

  • Takeaways & Limitations

    PRMs can be trained from cheaper response-level outcome data without dedicated step-label collection, making process-reward training more accessible within the studied setting.

  • Takeaways & Limitations

    The approach may not use the optimal algorithm, and more advanced methods could still integrate noisy stepwise labels into implicit PRMs.

Abstract

from arXiv · show

Different from its counterpart outcome reward models (ORMs), which evaluate the entire responses, a process reward model (PRM) scores a reasoning trajectory step by step, providing denser and more fine grained rewards. However, training a PRM requires labels annotated at every intermediate step, presenting significant challenges for both manual and automatic data collection. This paper aims to address this challenge. Both theoretically and empirically, we show that an \textit{implicit PRM} can be obtained at no additional cost, by simply training an ORM on the cheaper response-level labels. The only assumption is to parameterize the outcome reward as the log-likelihood ratios of the policy and reference models, which can be optimized regardless of the specific choice of loss objectives. In experiments, we instantiate our implicit PRMs with various objectives and evaluate their performance on MATH. We show that our implicit PRM outperforms a strong MCTS-based baseline \textit{á la} Math-Shepherd using less than $1/38$ of the training data. Its performance can be further improved with majority voting. We further find that scaling up instructions and responses benefits our implicit PRM, and the latter brings a larger gain. Particularly, we find that our implicit PRM, when instantiated with the cross-entropy (CE) loss, is more data-efficient and can keep improving generation models even when trained with only one response per instruction, the setup that suffers from extreme data scarcity and imbalance. Further, instructions should be relevant to downstream tasks while the diversity of responses does not bring gains. Surprisingly, training on extra Math-Shepherd step labels brings no further improvements to our implicit PRM trained on only outcome data. We hope that our work will encourage a rethinking of PRM training approaches and contribute to making training PRMs more accessible.

1 INTRODUCTION

Outcome reward models evaluate complete responses but provide sparse feedback, while process reward models offer denser step-level feedback at much higher annotation cost. This paper proposes learning an implicit PRM during ORM training from response-level data and reports stronger performance with lower overhead.

  • Outcome reward models evaluate full responses, but sparse outcome rewards can hurt inference reranking and destabilize reinforcement-learning training.
  • Process reward models require intermediate-step annotations, making their training substantially harder than ORM training.Automatic annotation can require massive look-ahead sampling or verifier training.
  • A log-likelihood-ratio reward parameterization lets an ORM intrinsically learn a process reward without additional training or step labels.
  • Implicit PRMs can be instantiated with multiple objectives, including DPO, KTO, NCA, and cross-entropy, while using only response-level data.
  • Across MATH evaluations, implicit PRMs outperform competitive PRM and ORM baselines with substantially better accuracy–development-overhead trade-offs.The experiments use best-of-N sampling across three generation models and report strong results for all four objective variants.
  • The KTO-based implicit PRM is the only reported variant that improves policy performance, and omitting the reference model can improve inference efficiency without hurting accuracy.

2 ORMS VS. PRMS: DILEMMA OF PERFORMANCE AND EXPENSE

PRMs provide denser feedback than ORMs but require substantially more expensive and potentially noisy step-level data collection. MCTS-based annotation can also misestimate process quality, motivating cheaper alternatives.

  • PRMs assign rewards to intermediate steps, whereas ORMs provide a sparse reward only after the complete response is generated.
  • Process rewards can be defined as advantages, namely differences between Q values, rather than only step correctness or raw Q values.
  • Dense step-level PRM rewards support more stable reinforcement-learning training and better response reranking than sparse ORM rewards.
  • 80 trajectories are required for each instruction under a 10-step, 8-rollout MCTS setup, versus one response-level collection process for ORM data.The paper presents this as 80 times more trajectory generation than ORM collection.
  • MCTS step labels may be noisy because hard estimation uses the maximum rollout outcome, while soft estimation uses the proportion of correct rollouts.
  • Hard estimation can overestimate Q because it uses the maximum outcome reward rather than its expectation.
  • Limited policy capability can make correct solutions difficult to sample for hard instructions, producing false negatives and underestimating Q values.

3 IMPLICIT PRMS FOR FREE THROUGH REWARD PARAMETERIZATION

The paper shows that an ORM can implicitly provide process rewards when its outcome reward is parameterized as a policy–reference log-likelihood ratio, eliminating dedicated step-label collection. The resulting process reward represents an expectation of outcome reward and can match or improve upon MCTS-derived estimates.

  • Reward parameterization: An ORM with a policy–reference log-likelihood-ratio reward parameterization can implicitly learn a process reward without dedicated PRM training.The approach applies across ORM loss objectives, including DPO and related variants that use the same parameterization.
  • Motivation: The method avoids the expensive step labels required by existing sequence-classifier and generative-logit PRMs.Those conventional parameterizations still require step-level supervision, motivating response-level ORM training instead.
  • Reward parameterization: The implicit process reward is calculated on partial responses, extending the same reward representation from complete outcomes to intermediate steps.The conclusion also holds when the intermediate unit is a token rather than a reasoning step.
  • Theoretical properties: The implicit process reward lies between soft- and hard-estimated Q values from MCTS-based approaches.The theoretical bounds are attained as β approaches infinity and zero, respectively, and suggest improved accuracy and robustness to noise.
  • Practical considerations: The reward construction incurs reference-model inference cost, although excluding the reference model can preserve accuracy for strong preference-trained models.The reference model doubles reward-model serving relative to a vanilla PRM, but generation-model inference often dominates best-of-N overhead.

4 EXPERIMENTS

The experiments instantiate implicit PRMs with several objectives and evaluate them through best-of-N sampling on MATH using multiple generation models. All variants improve generation-model accuracy while requiring substantially less data-collection and training computation than key baselines.

  • Results: All four implicit PRM variants consistently improve the accuracies of three generation models on MATH best-of-N evaluation.The variants use DPO, KTO, NCA, or cross-entropy objectives, with PRMs scoring each step and selecting responses by their lowest step reward.
  • Results: DPO reaches 50.4 averaged accuracy and NCA reaches 49.4 across the evaluated generation models.CE reaches 48.4 averaged accuracy despite training on unpaired and imbalanced rollouts.
  • Results: CE exceeds the implemented Math-Shepherd and AutoPSV baselines by 0.6 and 2.7 averaged-accuracy points, respectively.CE also outperforms the other open-source reward models except RLHFlow-8B-Mistral-Data and RLHFlow-8B-DS-Data, each reported at 49.1.
  • Efficiency: Math-Shepherd generally costs 38.8× more FLOPs than the CE implicit PRM.Relative to implicit PRM with DPO, the overhead ratios are 146.5×, 49.9×, and 21.3× across different response counts per instruction.
  • Scaling and setup: Implicit PRMs achieve better performance with much less data-collection and training overhead as the consumed-token budget scales.The training setup uses eight sampled rollouts per instruction from 33K math instructions, with correctness assessed against ground truth.

5 ANALYSIS

The analysis examines majority voting, data scaling, additional training factors, policy transfer, and inference overhead for implicit PRMs. Performance generally benefits from more instructions and responses, while several proposed augmentations provide no consistent gains.

  • 5.1 INCORPORATING MAJORITY VOTING: Implicit PRM scores can be aggregated across responses yielding the same answer, and the answer with the highest aggregated score is selected.This weighted best-of-N procedure differs from selecting the response with the highest individual score.
  • 5.1 INCORPORATING MAJORITY VOTING: Implicit PRMs outperform using either implicit PRM scoring or majority voting alone when the two methods are combined.KTO and CE gain the most, with CE becoming the most effective variant after majority voting.
  • 5.2 SCALING UP INSTRUCTIONS AND RESPONSES CAN IMPROVE IMPLICIT PRMS: Scaling instructions and responses consistently improves implicit PRM performance, with response scaling producing larger performance variations than instruction scaling.The trend is clearest for Mistral-7B-Instruct-v0.2 and Llama-3.1-8B-Instruct, with some outliers for Llama-3.1-70B-Instruct.
  • 5.2 SCALING UP INSTRUCTIONS AND RESPONSES CAN IMPROVE IMPLICIT PRMS: CE performs better than DPO with insufficient data and can improve generation models when trained with one response per instruction.DPO is under-trained with two responses per instruction because many instructions may not provide usable response pairs.
  • 5.3 ARE THERE ANY OTHER FACTORS CAN IMPROVE IMPLICIT PRM PERFORMANCE?: Adding task-irrelevant instructions, increasing response diversity, or adding step labels does not consistently improve implicit PRMs.Task-irrelevant instructions hurt performance, deduplication also hurts, and extra step-level KTO labels provide no gains; the authors caution that annotation quality and algorithm choice may matter.
  • 5.4 PRM ABILITY DOES NOT TRANSLATE INTO POLICY PERFORMANCE: PRM ability does not necessarily improve policy performance, and the two abilities can exhibit an unexpected trade-off.Only KTO improves MATH500 performance relative to Llama-3.1-8B-Instruct, despite KTO being the weakest implicit PRM in Table 1.
  • 5.5 CAN WE REDUCE THE INFERENCE OVERHEAD OF THE REFERENCE MODEL?: The reference model adds an inference concern, but it does not double overall inference cost in practice and can sometimes be removed without performance degradation.The reference model’s impact is smaller when the generation model is much larger than the reward model.

6 RELATED WORK

The paper situates implicit reward modeling within prior work on complex reasoning and preference learning, distinguishing its focus from existing uses of implicit rewards.

  • Complex reasoning remains challenging for state-of-the-art large language models, motivating techniques across pre-training, post-training, and inference.
  • Outcome reward models evaluate full responses and support reinforcement learning and inference, but their rewards are sparse.
  • Prior preference-learning work primarily applies implicit rewards to align models after supervised fine-tuning.
  • Recent studies use implicit rewards from resulting models as outcome rewards or Q functions, but do not specifically target improving or deriving process rewards.

7 CONCLUSION

The paper shows that log-likelihood-ratio reward parameterization enables intrinsic process-reward learning during outcome-reward training, and evaluates its empirical scope and practical behavior.

  • Various implicit reward-modeling objectives outperform baselines on MATH with better accuracy–development-overhead trade-offs, particularly cross-entropy.
  • Majority voting further improves implicit PRM performance.
  • Scaling instructions and responses benefits implicit PRMs, with responses having the larger effect; relevant instructions help, whereas response diversity does not.
  • Adding Math-Shepherd step labels provides no further improvement over an implicit PRM trained only on outcome data.

A PROOF OF PROPOSITION

This appendix proves the proposition for an outcome reward parameterized by the log-likelihood ratio of two causal language models, using autoregressive factorization and induction.

  • The proposition considers an outcome reward defined as the log-likelihood ratio of two causal language models.
  • The proof uses mathematical induction.
  • The response is assumed to contain T tokens.
  • Autoregressive factorization supplies the basis for the proof’s conclusion.
Loading 2412.01981v1…