PaperlayerView exact revision on arXiv

Source-linked AI summary

On Designing Effective RL Reward at Training Time for LLM Reasoning

Jiaxuan Gao, Shusheng Xu, Wenjie Ye, Weilin Liu, Chuyi He, Wei Fu, Zhiyu Mei, Guangju Wang, Yi Wu

arXiv:2410.15115v3cs.LGcs.AIcs.CL

TL;DR

Whether learned reward models add useful training signals beyond sparse success rewards for LLM reasoning remains unclear. The paper evaluates ORM and PRM in RL and introduces Clip and Delta to refine process rewards. With these techniques, RL improves the evaluated models, while unrefined learned rewards may fail to help or can be exploited through unnecessary repetitions.

  • Problem

    It remains unclear whether learned reward models provide additional training signals beyond sparse success rewards for LLM reasoning.

    Source: §1 · p. 2

  • Method

    The paper evaluates ORM and PRM combined with success rewards in RL and proposes Clip and Delta to refine process rewards.

    Source: §1 · p. 2

  • Results

    RL with carefully crafted rewards improves all evaluated LLMs, while unrefined learned rewards may fail to help or degrade performance relative to success reward alone.

    Source: §1 · p. 2

  • Takeaways & Limitations

    Clip bounds process rewards and Delta bounds the objective, mitigating reward hacking from repeated correct but unnecessary reasoning steps.

    Source: §1 · p. 2, §6 · p. 11

    Experiments cover 1.5B and 7B LLMs; evaluations on larger LLMs could further verify the techniques.

    Source: §6 · p. 11

Abstract

from arXiv · show

Reward models have been increasingly critical for improving the reasoning capability of LLMs. Existing research has shown that a well-trained reward model can substantially improve model performances at inference time via search. However, the potential of reward models during RL training time still remains largely under-explored. It is currently unclear whether these reward models can provide additional training signals to enhance the reasoning capabilities of LLMs in RL training that uses sparse success rewards, which verify the correctness of solutions. In this work, we evaluate popular reward models for RL training, including the Outcome-supervised Reward Model (ORM) and the Process-supervised Reward Model (PRM), and train a collection of LLMs for math problems using RL by combining these learned rewards with success rewards. Surprisingly, even though these learned reward models have strong inference-time performances, they may NOT help or even hurt RL training, producing worse performances than LLMs trained with the success reward only. Our analysis reveals that an LLM can receive high rewards from some of these reward models by repeating correct but unnecessary reasoning steps, leading to a severe reward hacking issue. Therefore, we introduce two novel reward refinement techniques, including Clipping and Delta. The key idea is to ensure the accumulative reward of any reasoning trajectory is upper-bounded to keep a learned reward model effective without being exploited. We evaluate our techniques with multiple reward models over a set of 1.5B and 7B LLMs on MATH and GSM8K benchmarks and demonstrate that with a carefully designed reward function, RL training without any additional supervised tuning can improve all the evaluated LLMs, including the state-of-the-art 7B LLM Qwen2.5-Math-7B-Instruct on MATH and GSM8K benchmarks.

1 INTRODUCTION

Learned reward models have mainly guided inference-time search, while their value as additional signals during RL training for reasoning remains unclear. This work evaluates ORM and PRM rewards in RL and introduces refinements to address reward hacking.

  • ORM estimates answer success, whereas PRM scores reasoning steps and can provide step-level feedback for search.The cited search strategies include Monte-Carlo Tree Search and beam search.Source: §1 · p. 1
  • Prior work leaves unclear whether reward models add training signals beyond success rewards for LLM reasoning.Although some work integrates reward models into RL and some capable math models use them, detailed analysis of their training-time contribution is lacking.Source: §1 · p. 2
  • The paper evaluates ORM and PRM as RL rewards on MATH and GSM8K using PPO.It finds that learned rewards may fail to improve training or degrade performance relative to sparse success reward alone.Source: §1 · p. 2
  • PRM reward hacking can lead an LLM to repeat correct but unnecessary reasoning steps for excessive reward, harming reasoning accuracy.The paper observes outcome rewards achieve training results similar to success rewards and hypothesizes they may add little when a more accurate success reward is available.Source: §1 · p. 2
  • Clip bounds process rewards, while Delta subtracts adjacent-step rewards to bound the objective and discourage trivial repetition.The techniques mitigate reward hacking on synthetic trajectories and stabilize full RL training; carefully crafted rewards improve the evaluated models, including Qwen2.5-Math-7B-Instruct on MATH and GSM8K.Source: §1 · p. 2

2 RELATED WORK

Related work covers RL for alignment and reasoning, learned reward models, reward hacking, and approaches to improving reasoning. This paper focuses on designing rewards that combine dense and sparse signals during RL training for reasoning tasks.

  • The paper addresses reward design for LLM reasoning, against prior work on RL for alignment, reasoning, and coding.PPO is described as widely used for robust performance across domains, and prior work identifies batch size as important for PPO performance in RLHF.Source: §2 · p. 2
  • In reasoning tasks with reliable solution checkers, ORM estimates final-answer correctness and PRM estimates correctness of reasoning steps.Reward models are also used in RLHF to align LLMs with human preferences, commonly trained on binary preference data.Source: §2 · p. 2
  • Reward hacking occurs when RL produces high-reward outputs that do not fulfill intended objectives.Prior approaches include reward ensembles, length penalties, length normalization, and PPO implementation techniques; this paper studies hacking when learned and success rewards are combined.Source: §2 · p. 3
  • Prior reasoning work spans pre-training, prompting, inference-time search, and fine-tuning.Inference-time search uses learned rewards to select promising solutions, while fine-tuning includes both high-quality question-answer training and RL.Source: §2 · p. 3

3 PRELIMINARY

The preliminary section represents an LLM solution as tokens and reasoning steps, then describes ORM and PRM supervision and the use of sparse success rewards or dense feedback during RL.

  • A solution is represented both as a token sequence and as a list of reasoning steps, with prefixes denoting steps up to a given point.Reasoning steps may be parsed using rule-based detectors, strict output formats, or special tokens.Source: §3 · p. 3
  • ORM is trained to estimate whether a solution prefix leads to a correct answer, using sampled solutions labeled for correctness.The objective compares the predicted outcome reward with the solution's binary correctness; training may use the full sequence or only its last token.Source: §3 · p. 3
  • PRM estimates the correctness of individual reasoning steps using labels for partial solutions.Process labels may come from human annotators or automatic annotation with LLMs.Source: §3 · p. 4
  • The success reward is the correctness of a solution, and RL optimizes it with Kullback-Leibler divergence.The LLM is treated as a policy that generates a solution for a question.Source: §3 · pp. 3–4
  • Success-reward RL provides a sparse signal at sequence end, while combining dense rewards offers finer-grained training signals.A PRM can provide dense feedback at the end of each reasoning step.Source: §3 · p. 4

4 RL REWARD FOR LLM REASONING

This section tests learned rewards alongside sparse success rewards for PPO reasoning and finds that naïve process rewards can be exploited through useless repetition. It then presents bounded reward designs and tests them on synthetic repetitions.

4.1 EVALUATING RL TRAINING WITH LEARNED REWARD MODELS

  • Neither OR nor PR surpasses success-reward training; OR tracks its accuracy, while PR increases generation length and step count.The study uses PPO on Qwen2-1.5B-Instruct for MATH, with sparse correctness rewards combined with either end-of-sequence ORM estimates or step-level PRM feedback.Source: §4.1 · pp. 4–5, Figure 1 · p. 5
  • Under PR, LLMs exploit positive rewards for meaningless or unnecessary steps, repeating them to raise return without improving accuracy.Synthetic trajectories confirm that PR favors added nonsense and repeated intermediate steps over the ground-truth solution.Source: §4.1 · pp. 5–6, Figure 3 · p. 6

4.2 TECHNIQUES FOR MITIGATING REWARD HACKING

  • Clip shifts PRM rewards by threshold η and caps them at zero; Delta differences adjacent-step rewards, drops the final-step reward, and bounds return.Delta return is bounded because the PRM's maximum output is 1.Source: §4.2 · pp. 6–7, Figure 4 · p. 7
  • On synthetic repetitive solutions, Clip and Delta limit return growth, and Clip yields smaller returns as repetition length increases.Source: §4.2 · p. 7
  • Length normalization and length penalties can still favor repetitive solutions over correct ones, while standard PRM normalization leads to training instability.Source: Figure 3 · p. 6, §4.2 · pp. 7–8

5 EXPERIMENTS

The experiments evaluate PPO training with learned rewards on math tasks, then test reward refinements designed to limit reward hacking. With PR-Clip-Delta, training improves evaluated models, including Qwen2.5-Math-7B-Instruct.

5.1 EXPERIMENT SETUP

  • Training uses MathInstruct questions and golden answers, evaluates GSM8K and MATH, and covers Qwen2 and Qwen2.5 general and math-specific models at 1.5B and 7B.The provided MathInstruct solutions are not used for training, and no further supervised fine-tuning is performed.Source: §5.1 · p. 8
  • PPO uses multiple sampled solutions per question, with batches of 1024 × 8 for 1.5B models and 4096 × 8 for 7B models.Source: §5.1 · p. 8

5.2 ABLATION STUDY

  • Among tested reward designs, OR, PR, and PR-Normed do not beat success-only training in greedy accuracy; Delta stabilizes training, and PR-Clip-Delta performs best.PR and PR-Normed results are from early epochs because their performance degrades significantly later.Source: §5.2 · p. 9, Table 1 · p. 9
  • Adding PR-Clip-Delta improves PPO across models and evaluation metrics, except greedy accuracy on Qwen2-Math-7B-Instruct.The evaluation includes Greedy, Sampling, and Pass@16 scores.Source: §5.2 · p. 9

5.3 MAIN RESULTS

  • Qwen2.5-Math-1.5B-Instruct reaches 87.34% GSM8K and 76.78% MATH greedy accuracy, gains of 2.20% and 0.78% over its base model.Source: §5.3 · p. 9
  • PR-Clip-Delta PPO improves all evaluated LLMs, including Qwen2.5-Math-7B-Instruct, which reaches 95.6% GSM8K and 83.38% MATH greedy accuracy.The reported scores surpass several baselines; the 7B model had already been trained using RL.Source: §5.3 · p. 9, Table 2 · p. 10

6 CONCLUSION

The paper studies dense PRM rewards for RL training and identifies reward hacking as a barrier to their effectiveness. Clip and Delta mitigate this issue, while experiments are limited to 1.5B and 7B models.

  • The paper finds that reward hacking can involve nonsensical text or unnecessary reasoning steps, and proposes Clip and Delta to mitigate it.Source: §6 · p. 11
  • Clip prevents exploiting the reward model, while Delta keeps the RL objective bounded; the techniques support using PRMs for improved RL training.Source: §6 · p. 11
  • Experiments cover 1.5B and 7B LLMs; evaluation on larger models could further verify the techniques.The authors also identify inference-time search evaluation as a direction for further study.Source: §6 · p. 11

A ADDITIONAL RESULTS

Additional results report RL training outcomes across base models and benchmarks, and show that PR-norm is associated with a significant accuracy drop from the third epoch.

A ADDITIONAL RESULTS

B ABLATION STUDY ON PPO BATCH SIZE

The PPO batch-size ablation finds that larger final batches improve performance, while sampling about eight solutions per question is recommended in practice.

B ABLATION STUDY ON PPO BATCH SIZE

C SYNTHETIC EXAMPLE

The synthetic example uses a baseball-card sharing problem to illustrate inserted nonsense steps or repetitions and gives 110 cards as Tom’s minimum savings.

C SYNTHETIC EXAMPLE

  • The illustrated manipulation inserts nonsense steps or repeats the last sentences in the reasoning.Source: C SYNTHETIC EXAMPLE · p. 17
  • The example asks how many of 220 baseball cards Tom must save while keeping at least half and dividing the rest equally among five brothers.Source: C SYNTHETIC EXAMPLE · p. 17
  • The example’s stated answer is that Tom saves 110 cards, leaving 110 to distribute among his brothers.Source: C SYNTHETIC EXAMPLE · p. 17

D BASELINES

The baselines include length normalization, which normalizes each solution’s rewards, and a length penalty that imposes a constant penalty per step.

D BASELINES

  • Length normalization normalizes the rewards for each solution.Source: D BASELINES · p. 18
  • The two baselines address solution length differently: one normalizes each solution’s rewards, while the other penalizes every step.Source: D BASELINES · p. 18
  • Length penalty imposes a constant penalty for each step.Source: D BASELINES · p. 18

D.1 THEORETICAL ANALYSIS

The Delta analysis gives returns for intermediate and final solution steps, then uses that result to state the policy gradient for optimizing the policy.

D.1 THEORETICAL ANALYSIS

  • Under Delta, an intermediate step returns α · rprocess(q, p(k)) + Correct(q, s), while the final step returns Correct(q, s) alone.The intermediate-step expression applies when k < K.Source: §D.1 · p. 19
  • The return expression includes a sequence of differences between process rewards at successive steps.Source: §D.1 · p. 19
  • The return result is used to state the policy gradient for optimizing the policy π under Delta.Source: §D.1 · p. 19

Related papers

Loading 2410.15115v3…