PaperlayerView exact revision on arXiv

Source-linked AI summary

Process Reinforcement through Implicit Rewards

Ganqu Cui, Lifan Yuan, Zefan Wang, Hanbin Wang, Yuchen Zhang, Jiacheng Chen, Wendi Li, Bingxiang He, Yuchen Fan, Tianyu Yu, Qixin Xu, Weize Chen, Jiarui Yuan, Huayu Chen, Kaiyan Zhang, Xingtai Lv, Shuo Wang, Yuan Yao, Xu Han, Hao Peng, Yu Cheng, Zhiyuan Liu, Maosong Sun, Bowen Zhou, Ning Ding

arXiv:2502.01456v2cs.LGcs.AIcs.CL

TL;DR

Dense rewards could improve LLM reinforcement learning, but online process-reward training is hindered by costly process labels, reward hacking, and explicit modeling overhead. PRIME uses implicit process rewards learned from policy rollouts and outcome labels, combines them with outcome rewards across advantage functions, and reports a 15.1% average gain over SFT while surpassing Qwen2.5-Math-7B-Instruct on seven reasoning benchmarks using 10% of its training data.

  • Problem

    Online dense-reward reinforcement learning for LLMs is limited by expensive process labels, vulnerability to reward hacking, and costly process reward-model training.

  • Method

    PRIME updates an implicit process reward model online from policy rollouts and outcome labels, combines dense and outcome rewards, and avoids a dedicated reward-modeling stage.

  • Results

    15.1% average improvement was achieved across key reasoning benchmarks over the SFT model, and Eurus-2-7B-PRIME surpassed Qwen2.5-Math-7B-Instruct on seven reasoning benchmarks using 10% of its training data.

  • Takeaways & Limitations

    PRIME provides a scalable dense-reward approach that benefits sample efficiency and policy performance across mathematical reasoning and coding experiments.

  • Takeaways & Limitations

    Explicit process reward modeling remains constrained by the extensive annotation and broad data coverage needed for adaptability and generalization under distribution shifts.

Abstract

from arXiv · show

Dense process rewards have proven a more effective alternative to the sparse outcome-level rewards in the inference-time scaling of large language models (LLMs), particularly in tasks requiring complex multi-step reasoning. While dense rewards also offer an appealing choice for the reinforcement learning (RL) of LLMs since their fine-grained rewards have the potential to address some inherent issues of outcome rewards, such as training efficiency and credit assignment, this potential remains largely unrealized. This can be primarily attributed to the challenges of training process reward models (PRMs) online, where collecting high-quality process labels is prohibitively expensive, making them particularly vulnerable to reward hacking. To address these challenges, we propose PRIME (Process Reinforcement through IMplicit rEwards), which enables online PRM updates using only policy rollouts and outcome labels through implict process rewards. PRIME combines well with various advantage functions and forgoes the dedicated reward model training phrase that existing approaches require, substantially reducing the development overhead. We demonstrate PRIME's effectiveness on competitional math and coding. Starting from Qwen2.5-Math-7B-Base, PRIME achieves a 15.1% average improvement across several key reasoning benchmarks over the SFT model. Notably, our resulting model, Eurus-2-7B-PRIME, surpasses Qwen2.5-Math-7B-Instruct on seven reasoning benchmarks with 10% of its training data.

1 INTRODUCTION

PRIME addresses the difficulty of using dense rewards in online LLM reinforcement learning by updating process rewards from outcome labels, while avoiding a separate reward-modeling stage. Experiments report improved efficiency and reasoning performance across mathematical tasks and multiple RL algorithms.

  • Motivation: Dense rewards can improve training efficiency and credit assignment over sparse outcome rewards, but scalable online use remains underexplored.The main obstacles are expensive process labeling and the difficulty of updating process reward models online without reward hacking.
  • PRIME: PRIME trains implicit process reward models online using policy rollouts and outcome-level labels.This approach avoids requiring high-quality step-level process annotations during online reinforcement learning.
  • PRIME: PRIME combines dense token-level rewards with sparse outcome rewards and supports diverse reinforcement-learning algorithms.The framework separately calculates their returns before combining them, enabling compatibility with multiple advantage functions.
  • PRIME: PRIME eliminates the dedicated reward-modeling stage by initializing from the SFT model or base model.This reduces development overhead compared with approaches requiring explicit reward-model training.
  • Results: 16.7% average improvement was achieved on key mathematical reasoning benchmarks over the SFT model, with over 20% improvement on AMC&AIME competitions.The experiments used Qwen2.5-Math-7B-Base after a lightweight SFT warmup stage.
  • Results: PRIME achieved a 2.5× sample-efficiency gain and a 6.9% performance improvement over outcome-reward-only reinforcement learning on challenging math problems.The resulting model also surpassed Qwen2.5-Math-7B-Instruct on five key mathematical benchmarks using 10% of Qwen-Math’s data.
  • Analysis: Online PRM updating was identified as key to PRIME’s success, and PRIME improved RLOO, REINFORCE, PPO, and GRPO.The analysis also found that implicit PRMs work better as reward models than as value models.

2 REINFORCEMENT LEARNING FOR LLMS AND THE CHALLENGES OF INCOPORATING DENSE REWARDS

LLM reinforcement learning typically relies on sparse outcome rewards, which limit feedback to the end of generation. Dense process rewards could address this limitation, but their online use is constrained by labeling ambiguity, scalability concerns, and explicit reward-modeling costs.

  • RL preliminaries: In autoregressive language modeling, the state is the prompt plus the current response prefix, and the action is the next token or step.RL learns a policy that maximizes expected cumulative discounted reward while interacting with its environment.
  • RL preliminaries: The advantage function measures how much better an action is than alternatives in a given state.In practice, it is estimated from cumulative discounted rewards minus a baseline.
  • RL preliminaries: γ ∈[0, 1] discounts future rewards, while r(y_s) denotes the reward supplied at step s.Different baselines include value estimates, group-average rewards, and leave-one-out average rewards.
  • RL preliminaries: Value models reduce variance in advantage estimation by predicting expected accumulated rewards from a state.GAE uses temporal-difference errors and λ to control the bias-variance tradeoff; PPO trains a value model with the policy.
  • Reward sparsity: Sparse outcome rewards assign meaningful feedback only to the final token, which can encourage incorrect processes that produce correct answers and reduce sample efficiency.This setting also makes credit assignment more difficult because feedback arrives only after the entire generation.
  • Challenges: Dense reward models score responses at each token or step, but incorporating them into online RL is usually infeasible because of three implementation challenges.The challenges concern defining process rewards, updating PRMs at scale, and the cost of explicit reward modeling.
  • Challenges: Intermediate process labels are costly and can be ambiguous because some incorrect steps may still help reach a final answer by pruning search branches.This makes defining absolute correctness for dense intermediate rewards difficult.
  • Challenges: Online PRM training requires extensive nuanced step-level annotation, creating scalability and generalization concerns for dense-reward RL.Static reward models also risk overoptimization or reward hacking when policy distributions shift.

3 PRIME

PRIME is a scalable online RL framework that uses implicit process rewards to update reward models from policy rollouts and outcome supervision, then incorporates token-level dense rewards into advantage estimation. It also initializes PRMs from the SFT or base model and uses online prompt filtering to support stable, efficient training.

  • Advantage estimation: PRIME combines implicit process rewards with outcome rewards in Monte Carlo advantage estimates and remains compatible with alternative advantage functions.The paper evaluates REINFORCE, RLOO, PPO, and GRPO, and instantiates PRIME with a leave-one-out baseline.
  • Implicit process rewards: Implicit PRM rewards are derived from an outcome-trained reward model using the policy and reference model likelihood ratio.At inference, the model is repurposed to provide process rewards at each step.
  • PRIME framework: PRIME updates an Implicit PRM online using on-policy rollouts and outcome supervision, then calculates token-level dense rewards for advantage estimation.This avoids requiring annotated step labels for new policy rollouts.
  • Policy update: PRIME uses PPO’s clipped surrogate loss to make policy updates more stable by limiting deviation from the original policy distribution.The clipping parameter constrains the updated policy for importance sampling.
  • Other techniques: The starting SFT or base model can initialize the PRM, bypassing dedicated PRM training and outperforming a separately trained PRM in the reported experiment.Online prompt filtering retains prompts in a median-level difficulty range and balances the Implicit PRM training distribution.
  • Other techniques: Online prompt filtering largely lowers the variance of RL training in the reported outcome-reward-only ablation.The filter selects prompts within a specified accuracy range.

4 EXPERIMENTS

Experiments evaluate PRIME on competition-level mathematics and programming tasks, comparing dense-reward training with outcome-reward baselines and testing PRM initialization and training efficiency. PRIME improves downstream reasoning performance while requiring more time per step but fewer steps overall.

  • Experimental Setup: Experiments use Qwen2.5-Math-7B-Base after supervised fine-tuning, with evaluation spanning seven mathematics and programming benchmarks.The benchmarks include AIME 2024, AMC, MATH-500, Minerva Math, OlympiadBench, LeetCode, and LiveCodeBench (v2).
  • Main Results: 15.1% average improvement over the SFT model is achieved across key reasoning benchmarks, with over 20% improvement on AMC and AIME.Eurus-2-7B-PRIME achieves these gains over the SFT version of the model.
  • Dense Rewards vs. Sparse Rewards: 2.5× sample efficiency and 6.9% performance improvement distinguish PRIME from RLOO with outcome verifier rewards.PRIME also substantially outperforms RLOO on downstream tasks.
  • PRM Ablations: Online PRM initialization from the SFT model outperforms initialization from a PRM trained on extra rollouts.The authors attribute this to reduced distribution shift when the policy and PRM start from the same model.
  • Main Results: 26.7% pass@1 on AIME 2024 surpasses GPT-4o, Llama-3.1-70B-Instruct, and Qwen2.5-Math-7B-Instruct.This result is reported as a benchmark outcome for Eurus-2-7B-PRIME.
  • Training Efficiency: 24% more time per training step is offset by PRIME reaching the same training rewards in 40% as many steps, yielding an estimated 2× time-efficiency advantage.The single-controller design requires no extra GPU memory because other components are offloaded to CPU during PRM updates.

5 ANALYSIS

PRIME’s analysis finds that online PRM updates, shared SFT initialization, increased computation, and compatibility with multiple RL algorithms support stable and efficient training. Using implicit PRM rewards for returns outperforms value-model alternatives.

  • 5.1 DESIGN CHOICES FOR THE IMPLICIT PRM: The Implicit PRM is a key PRIME component whose initialization model and update mechanism substantially affect RL.
  • 5.1 DESIGN CHOICES FOR THE IMPLICIT PRM: Initializing the PRM from the SFT model greatly outperforms initializing it from a specially trained PRM.The specially trained EurusPRM used an additional 500K generated samples.
  • 5.1 DESIGN CHOICES FOR THE IMPLICIT PRM: Online PRM updates mitigate overoptimization and reward hacking, whereas offline PRM accuracy gradually drops during RL because of distribution shift.Online PRMs trained on policy rollouts show the reverse accuracy trend.
  • 5.2 SCALING PRIME WITH MORE COMPUTE: 3.7% improvement: PRIME consistently outperforms outcome-reward RLOO during extended training, while 4.4% improvement appears with 16 responses per prompt.The extended experiment used 800 rollout steps and 3200 gradient steps; the larger-rollout comparison increased samples from 4 to 16 responses per prompt.
  • 5.3 PRIME WITH OTHER RL ALGORITHMS: PRIME boosts REINFORCE, GRPO, and PPO in both efficiency and performance, indicating compatibility across policy-update methods.The implementation modifies advantage estimation functions while leaving the clip surrogate loss unchanged.
  • 5.4 VALUE OR REWARD, HOW TO USE THE IMPLICIT PRM?: Implicit PRM rewards for calculating returns greatly outperform baselines, while using PRMs as value models falls behind reward-model use.An additional value model does not benefit policy performance when comparing PPO and REINFORCE.

6 RELATED WORK

Related work situates PRIME within the shift toward RL for reasoning LLMs and prior research on implicit rewards. It addresses the use of dense rewards for LLM RL within these lines of work.

  • RL for LLM Reasoning: RL has been widely used for aligning human preferences, while open-source reasoning efforts initially relied mostly on imitation learning.
  • RL for LLM Reasoning: Recent work reports scaling effects from RL with outcome rewards, while the role of dense rewards in LLM reasoning remains part of the emerging paradigm.
  • Implicit Rewards: Prior studies use implicit rewards in LLM alignment, including work showing that DPO learns a Q function implicitly and that dense implicit rewards can work in PPO.

7 CONCLUSION

The paper develops PRIME to produce and use dense rewards in online RL for LLM reasoning. Experiments validate benefits for efficiency and policy performance, low cost, and broad algorithmic applicability.

  • 7 CONCLUSION: PRIME produces and leverages dense rewards in online RL for LLM reasoning.
  • 7 CONCLUSION: PRIME benefits sample efficiency and policy performance, is easy to use with minimum cost, and works with broad RL algorithms.

ETHICS STATEMENT

The paper states that it has potential societal consequences but does not identify any that require specific highlighting.

  • ETHICS STATEMENT: The authors report potential societal consequences but do not identify any requiring specific emphasis.

A LIMITATIONS

The reported limitations and design comparisons constrain interpretation of PRIME’s scalability and implementation choices. Reference-model variants have similar rewards, while double-forward improves PRM accuracy without clearly improving training rewards.

  • Experiments were limited to models up to 32B, with fewer steps for some ablations.
  • Reference model choice: The policy-reference and SFT-reference strategies achieve similar rewards.The policy can naturally serve as the reference, whereas KL calculation requires retaining the initial SFT model.
  • Single-forward vs. double-forward: Double-forward increases PRM accuracy, but training rewards remain close to single-forward.Double-forward updates the PRM before recalculating process rewards; single-forward uses rewards from older PRMs.

B.3 RESULTS OF DIFFERENT RL ALGORITHMS

PRIME is evaluated across several RL algorithms, model scales, reward-model capacities, and efficiency comparisons. The results generally favor PRIME, while zero-shot RL can saturate early and some comparisons have constrained scope.

  • RL algorithms: PRIME contributes consistently across policy-update methods, supporting its use as a generic algorithm.The PPO variant provides no performance gain, making simpler REINFORCE-like advantage estimators a practical alternative to critic-based computation.
  • Zero experiments: RL from base models converges faster than the SFT-RL setting, and 32B models gain more than 7B models.The reported observations come from experiments with Qwen2.5-Math-7B-Base and Qwen2.5-32B-Base without SFT.
  • Zero experiments: PRIME-Zero saturates at about 50 steps, which may hinder further improvement and is possibly related to reduced response diversity.
  • VinePPO comparison: 11x efficiency: PRIME requires 1.22 hours versus VinePPO’s 13.94 hours and consistently achieves higher validation accuracy.
  • DeepScaleR comparison: PRIME reaches comparable training accuracy to DeepScaleR in 330 steps, versus 1040 steps for DeepScaleR’s first stage.PRIME improves test performance by 3.1 points and consumes 446.7 A800 GPU hours, while DeepScaleR’s first stage roughly requires 600 GPU hours.
  • Loss function: DPO and CE produce similar test accuracy for PRM updates, so CE is selected for memory efficiency.The training logs for this comparison were broken, and the training curve was estimated from a figure in the blog.

D SFT DATA & TRAINING DETAILS

The paper prepares RL with supervised fine-tuning, constructs separate mathematics and coding datasets, and applies filtering, reformatting, and validation to improve data quality.

  • SFT preparation: RL preparation begins with supervised fine-tuning focused on mathematical and coding problems, using Qwen2.5-Math-7B-Base.
  • SFT preparation: The action-centric chain-of-thought framework has the model choose one of seven actions at each reasoning step.Responses begin with the ASSESS action and use multi-step reasoning.
  • SFT data: The SFT dataset combines open-source reasoning instructions while reserving many ground-truth datasets for later RL to diversify exploration.
  • SFT training: SFT uses full-parameter fine-tuning with a 1e-05 learning rate, batch size 96, random seed 42, and three epochs on 230K datasets.
  • RL data: The RL dataset covers mathematics and coding problems with outcome verifiers, sourced from NuminaMath-CoT, APPS, CodeContests, and TACO.
  • Data processing: Mathematical data are filtered, reformatted, and validated for solvability, difficulty, correctness, and standardized LaTeX answers.Multiple-choice conversion uses rule-based and LLM-based filtering, followed by question and solution reformatting and repeated consistency checks.

E.5 PRM DATA

The paper reports dataset statistics for training EurusPRM.

  • Dataset statistics for training EurusPRM are reported in Table 11.
Loading 2502.01456v2…