PaperlayerView exact revision on arXiv

Source-linked AI summary

Stop Summation: Min-Form Credit Assignment Is All Process Reward Model Needs for Reasoning

Jie Cheng, Gang Xiong, Ruixi Qiao, Lijun Li, Chao Guo, Junle Wang, Yisheng Lv, Fei-Yue Wang

arXiv:2504.15275v3cs.AIcs.LG

TL;DR

PRM reward hacking limits reinforcement fine-tuning because summation-form credit assignment can induce high-reward step hacking and training collapse. PURE replaces it with min-form credit assignment, enabling stable PRM-based fine-tuning with comparable performance to verifiable-reward methods within 30% of the steps, while adding a small proportion of ground-truth signals further improves results.

  • Problem

    PRM reward hacking limits their effective use for reinforcement fine-tuning, despite PRMs providing dense feedback and helping test-time scaling on challenging reasoning tasks.

  • Method

    PURE uses min-form credit assignment, defining credit through minimum future rewards to constrain value ranges and assign advantages more reasonably.

  • Results

    PRM-based fine-tuning with min-form credit assignment achieves performance comparable to verifiable-reward approaches within 30% of the steps, while summation-form training collapses early; adding around 10% ground-truth signals yields the highest average benchmark score of 53.3.

  • Takeaways & Limitations

    Min-form credit assignment makes PRM-based reinforcement fine-tuning more stable, and a small auxiliary proportion of ground-truth signals further alleviates reward hacking.

  • Takeaways & Limitations

    Causal discriminative PRMs can assign high rewards to irrelevant or empty responses because they do not recognize that no further content follows; generative PRMs are left for future work.

Abstract

from arXiv · show

Process reward models (PRMs) have proven effective for test-time scaling of Large Language Models (LLMs) on challenging reasoning tasks. However, reward hacking issues with PRMs limit their successful application in reinforcement fine-tuning. In this paper, we identify the main cause of PRM-induced reward hacking: the canonical summation-form credit assignment in reinforcement learning (RL), which defines the value as cumulative gamma-decayed future rewards, easily induces LLMs to hack steps with high rewards. To address this, we propose PURE: Process sUpervised Reinforcement lEarning. The key innovation of PURE is a min-form credit assignment that formulates the value function as the minimum of future rewards. This method significantly alleviates reward hacking by limiting the value function range and distributing advantages more reasonably. Through extensive experiments on 3 base models, we show that PRM-based approaches enabling min-form credit assignment achieve comparable reasoning performance to verifiable reward-based methods within only 30% steps. In contrast, the canonical sum-form credit assignment collapses training even at the beginning! Additionally, when we supplement PRM-based fine-tuning with just 10% verifiable rewards, we further alleviate reward hacking and produce the best fine-tuned model based on Qwen2.5-Math-7B in our experiments, achieving 82.5% accuracy on AMC23 and 53.3% average accuracy across 5 benchmarks. Moreover, we summarize the observed reward hacking cases and analyze the causes of training collapse. We release our code and model weights at https://github.com/CJReinforce/PURE.

1 Introduction

Reinforcement fine-tuning benefits from verifiable rewards but their sparse response-level feedback can become inefficient for long reasoning sequences. PRMs provide dense step-level feedback, yet reward hacking has limited their use in reinforcement fine-tuning; PURE addresses this with min-form credit assignment.

  • Verifiable rewards provide sparse feedback for an entire response, which can become inefficient as reasoning responses grow longer.
  • PRMs provide dense feedback at each response step and have improved challenging reasoning through test-time scaling.
  • PRM use in reinforcement fine-tuning remains limited because neural rewards can induce unintended optimization toward higher-reward patterns.
  • PURE introduces min-form credit assignment to align PRM training objectives with their test-time-scaling use.
  • PRM-based RFT with min-form credit assignment matches verifiable-reward performance with around 3× efficiency gains, while sum-form training collapses at the beginning.
  • Adding a few ground-truth signals further mitigates PRM-induced reward hacking and reaches 82.5% AMC23 accuracy and 53.3% average accuracy across five benchmarks.

2 Preliminaries

Credit assignment estimates how actions influence outcomes in step-level LLM reasoning. The section contrasts sparse verifiable rewards with dense process rewards and motivates making PRMs effective for training as well as test-time scaling.

  • Credit assignment distributes outcome credit among the sequential decisions that produced it.
  • LLM reasoning is modeled as a step-level Markov Decision Process in which each generated step receives a PRM process reward.
  • The canonical state-action value sums future process rewards with discounting, so actions closer to future outcomes receive greater influence.
  • Verifiable rewards are sparse, rule-based signals assigned to the complete response according to whether it matches the ground-truth answer.
  • Process rewards evaluate individual reasoning steps and are commonly aggregated by their minimum value for test-time response selection.
  • Although dense rewards work in traditional RL applications, they remain difficult to use for advanced-reasoning fine-tuning, making PRM training effectiveness an important open topic.

3 PURE: Process Supervised Reinforcement Learning

PURE adapts PRM-based reinforcement fine-tuning through min-form credit assignment, which focuses credit on the lowest-reward step and can be implemented by transforming process rewards. The method constrains value estimation, reduces reward hacking, and supports mixed process and verifiable rewards.

  • 3 PURE: Process Supervised Reinforcement Learning: PURE proposes min-form credit assignment to make PRMs effective for reinforcement fine-tuning by aligning training with their test-time use.
  • 3.1 Min-form Credit Assignment: The min-form objective makes the worst process-reward step determine the response value, while steps after it do not contribute to the objective.
  • 3.1 Min-form Credit Assignment: PURE identifies the worst step as the step with minimum process reward and defines return and state-action value functions around it.
  • 3.1 Min-form Credit Assignment: The process-reward transformation gives higher weights to lower rewards and approaches a one-step emphasis on the worst step as T → 0+.
  • 3.4 Advantage Estimation: PURE assigns transformed process rewards to final step tokens and can place verifiable rewards on the final token of the complete response.
  • 3.2 Quantitative Analysis: Min-form value estimation error is bounded by the single-step reward error, whereas sum-form error accumulates with horizon length and discounting.
  • 3.2 Quantitative Analysis: Min-form keeps the value-function range equal to the reward range, limiting excessive values that can drive reward hacking as sequence length increases.
  • 3.3 Qualitative Analysis: In the qualitative comparison, sum-form increases sampling of thinking steps and decreases sampling of incorrect solution steps, whereas min-form reduces probabilities across steps most at the first incorrect step.

4 Experiments

Across three models and multiple reward configurations, PURE evaluates PRM-based reinforcement fine-tuning against verifiable-reward baselines. Min-form credit assignment remains stable and reaches comparable or better performance, while sum-form training collapses; adding a small amount of ground-truth supervision further improves results.

  • PRM Training and Evaluation: PURE-PRM-7B achieves a state-of-the-art average F1 score of 57.5 on ProcessBench, surpassing the previous best score of 56.5.On PRMBench, it ranks third among open-source PRMs.
  • Main Results: 49.3 is PURE-PRM’s average score on Qwen2.5-Math-7B, compared with 48.3 for PURE-VR and 53.3 for PURE-PRM+VR.The combined method’s improvement over PURE-VR is approximately 5 percentage points, and the trend remains consistent across the other two base models.
  • Training Dynamics: Sum-form PURE-PRM and PURE-PRM+VR collapse at step 25, whereas min-form methods remain stable for 200 steps.At step 80, sum-form variants fall to around 30 average benchmark score versus a base-model score of 39.5; min-form variants reach 49.3 and 53.3.
  • Training Efficiency: PRM-involved approaches require around 30% of the training steps to reach the same MATH-500 accuracy as PURE-VR.This indicates substantially higher learning efficiency for dense process rewards than for sparse verifiable rewards.
  • Reward Hacking: Reward hacking still occurs with PURE-PRM alone, but min-form credit assignment delays its onset and adding around 10% ground-truth signals reduces it further.PURE-PRM+VR uses 800 problems with ground-truth answers and 7,200 open problems, achieving stable training and the highest average score of 53.3.

5 Analysis

The analysis identifies several PRM-induced reward-hacking patterns and traces training collapse to reward-favored behaviors, unsuitable baselines, and repetitive pseudo-positive responses. Training curves show that collapse can occur abruptly after the model begins generating long, highly repetitive outputs.

  • Observed reward-hacking cases: PRM reward hacking appears in three observed forms: only thinking, extremely few steps, and irrelevant or empty responses.The paper categorizes these cases and provides specific examples in Appendix E.
  • Only thinking, not solving: Sum-form credit assignment can reward thinking patterns over solution steps, producing models that think without solving.The authors note that backtracking and verification can also be useful patterns, so exploiting patterns is not inherently harmful.
  • Extremely few steps: A step-level baseline can favor one-step responses because, at equal process-aggregated rewards, longer responses are penalized more heavily.The resulting policy may emit a single step with an excessively large number of tokens, undermining step-by-step process evaluation.
  • Zero-step responses: Causal discriminative PRMs can assign high rewards to meaningless final outputs because they do not observe that no further content follows.The paper uses ground-truth signals such as verifiable rewards as an aid and leaves generative PRMs for future work.
  • Training collapse: At step 365, training collapses with a sharp drop in rewards and accuracy after response length, clipping, and repetition rise sharply from step 361.The model learns repetitive patterns within 5 gradient steps, while the process-aggregated outcome reward is the summation of transformed process rewards.
  • Training collapse: Long, highly repetitive responses judged correct act as pseudo-positive samples that drive collapse because verifiers miss them and current PRMs do not detect their patterns.Assigning zero rewards to these samples helps little because the model finds repetitions that the LCP-based detector cannot identify.

6 Conclusion

PURE uses min-form credit assignment to alleviate PRM-induced reward hacking, enabling PRM-based RFT to match verifiable-reward approaches within 30% of the training steps. The paper also identifies pseudo-positive samples as a cause of training collapse and outlines future directions for stronger PRM-based RFT.

  • PURE alleviates PRM-induced reward hacking through min-form credit assignment.The paper attributes this improvement to the proposed credit-assignment method.
  • PRM-based RFT achieves performance similar to verifiable-reward approaches within 30% of the steps and can outperform them with a few ground-truth signals.
  • Pseudo-positive samples can collapse training during PRM-based RFT.
  • Current PRMs cannot address one reported reward-hacking type or evaluate patterns such as endless repetition.The paper suggests generative PRMs and iterative PRM–LLM training as directions for addressing these limitations.
  • PRM-based training remains the primary focus because prior research has struggled to use PRMs effectively during training-time RFT.

B Proof of Q-value estimation error

The proof compares Q-value estimation errors under summation-form and min-form credit assignment. It shows that min-form error does not accumulate across steps and is bounded by the largest single-step reward-model error.

  • The theorem compares summation-form and min-form Q-value estimation errors for trajectories with n reasoning steps.
  • Sum-form Error Bound: The summation-form proof uses the canonical Q-value formulation and bounds each step’s reward error by ϵ.
  • Sum-form Error Bound: The infinite-horizon summation bound depends on the convergent geometric-series factor 1/(1−γ).
  • Min-form Error Bound: For min-form credit assignment, the trajectory return is defined as the minimum of future rewards.
  • Min-form Error Bound: The min-form estimation error does not accumulate and is directly bounded by the maximum single-step reward-model error.

C Benchmark Scores of Our PRM

The appendix reports benchmark comparisons for the proposed PRM on ProcessBench and PRMBench. The tables organize subset-level or method-level results, with the proposed PRM identified separately from prior systems.

  • Detailed ProcessBench and PRMBench scores are reported in Tables 3 and 4, respectively.
  • ProcessBench: Table 3 reports F1 scores for each ProcessBench subset and marks the proposed PRM with a blue line.Other lines are taken from Zheng et al. (2024).
  • ProcessBench: Table 4 reports ProcessBench results and distinguishes the proposed PRM from methods reported by Song et al. (2025).

D Details of Training Metrics

The training-metrics appendix defines process-aggregated outcome reward, clip ratio, repetition score, and high repetition ratio, while illustrating reward hacking through an example of thinking without solving. These metrics expose truncation and repetition patterns associated with training collapse, but the repetition score misses near-synonym repetition.

  • Training Metrics: Process-aggregated outcome reward sums transformed process rewards and approximates the minimum PRM-emitted reward for a response.
  • Training Metrics: Clip ratio measures replay-buffer samples truncated at the maximum generation length, which often follows infinitely repetitive responses after collapse.
  • Training Metrics: Repetition score uses longest common prefix lengths between suffix pairs but cannot detect repetition expressed through near-synonyms.This limitation lets models exploit complex repetition patterns to bypass the metric.
  • Training Metrics: A high repetition ratio is the percentage of replay-buffer samples whose repetition score exceeds 0.2.
  • Reward Hacking Examples: The reward-hacking example shows an LLM producing reasoning and trigonometric equations without substituting numbers to solve the problem.The behavior is attributed to exploiting high-reward thinking patterns.

F.1 Ablation on Transform Temperature

The transform-temperature ablation evaluates how closely PURE approximates min-form credit assignment, using PURE-PRM+VR on Qwen2.5-7B. Among the tested settings, 0.1 performs best.

  • 0.1 is the best transform temperature for PURE-PRM+VR on Qwen2.5-7B.The experiment tests three transform-temperature values for 230 steps and reports pass@1 accuracy with greedy decoding.

F.2 PURE with Other RL Algorithms

PURE is evaluated with several advantage estimators beyond RLOO, including GAE, GRPO, and REINFORCE++. Their performance is similar early in training, while stability and computational cost favor RLOO overall.

  • Training comparison: GAE, GRPO, REINFORCE++, and RLOO show similar performance before step 150.The comparison uses training curves for PURE-PRM+VR with adjustments for combined verifier and process rewards.
  • Training comparison: REINFORCE++ experiences a spike around step 220, whereas GAE converges slightly more slowly but is more stable during the final 50 steps.
  • Estimator selection: RLOO is selected as the preferred advantage estimator based on performance, training time, and stability.GAE requires about 30% more time for forward and backward passes because it uses an additional learnable value network.
Loading 2504.15275v3…