PaperlayerView exact revision on arXiv

Source-linked AI summary

Reward Shaping to Mitigate Reward Hacking in RLHF

Jiayi Fu, Xuandong Zhao, Chengyuan Yao, Qi Han, Yuxia Wang

arXiv:2502.18770v7cs.LGcs.AIcs.CL

TL;DR

RLHF reward hacking motivates a systematic study of reward shaping and its design principles. The paper proposes PAR, which uses centered preference signals with bounded sigmoid shaping, and reports stronger, more stable performance across several settings, while noting limits in peak gains, datasets, and base-model generality.

  • Problem

    Reward shaping for RLHF has not been systematically compared, leaving its underlying design principles unclear amid vulnerability to reward hacking.

  • Method

    PAR uses the reward model’s latent preference between policy and reference responses, applying a sigmoid to the centered proxy reward.

  • Results

    PAR consistently outperforms other shaping methods and the unshaped baseline in reported settings, with strongest effects for PPO and A2C and smaller effects for GRPO.

  • Takeaways & Limitations

    Bounded, rapidly growing, gradually saturating rewards can stabilize RLHF training, reduce variance, and widen the practical early-stopping window.

  • Takeaways & Limitations

    PAR does not substantially improve peak best-checkpoint win rate, and its effectiveness is not obvious on Qwen3-4B-Base.

Abstract

from arXiv · show

Reinforcement learning from human feedback (RLHF) is widely used to align large language models (LLMs) with human preferences. However, RLHF remains vulnerable to \emph{reward hacking}, whereby a policy exploits imperfections in the reward function instead of learning the intended behavior, thereby undermining alignment. Although reward shaping can stabilize RLHF training and partially mitigate reward hacking, shaping methods and their underlying design principles have not been systematically investigated. To address this gap, we conduct a comprehensive study of prevalent reward-shaping techniques. Our analysis identifies two key design principles: (1) the reinforcement-learning reward should be bounded, and (2) it should grow rapidly at first and then gradually saturate. Motivated by these principles, we propose Preference as Reward (PAR), a novel method that uses the latent preferences encoded in the reward model as the reinforcement-learning signal. We further show that PAR possesses two variance-reduction properties that stabilize RLHF training and substantially widen the practical window for early stopping. Our evaluation consists of two parts. First, we compare PAR with several other reward-shaping strategies using Proximal Policy Optimization (PPO) as the reinforcement-learning algorithm and Gemma2-2B as the base model. Second, we compare PAR with the vanilla baseline (i.e., unshaped reward) across four base models and four reinforcement-learning algorithms. In the first set of experiments, PAR consistently outperforms other reward-shaping methods and also reflects high data efficiency and robustness. The second set of experiments shows that PAR is particularly effective for actor-critic RL algorithms when value estimates become unstable and demonstrates its effectiveness across different base models. The code is available at https://github.com/PorUna-byte/PAR.

1 Introduction

RLHF can suffer reward hacking, and reward-shaping methods lack systematic comparison and clear design principles. This paper proposes PAR, grounded in bounded, saturating rewards, and reports robust performance across evaluation settings.

  • RLHF policies can exploit reward-model weaknesses, producing high proxy rewards without genuinely improving alignment or response quality.
  • Existing reward-shaping methods commonly clip or rescale proxy rewards, but their comparative effectiveness and design principles remain unclear.
  • The paper identifies bounded rewards and rapid initial growth followed by gradual saturation as principles for effective RLHF reward shaping.
  • Preference as Reward (PAR) applies a sigmoid to the centered difference between policy and reference rewards, converting proxy scores into preference-based RL signals.
  • PAR reduces return and policy-gradient variance, delays reward hacking, and widens the practical early-stopping window.
  • PAR achieves strongest performance among tested shaping methods and consistently outperforms unshaped rewards under PPO and A2C, while effects are smaller under GRPO.

2 Related Work

Related work addresses reward hacking through reward modeling, policy constraints, and reward transformations. PAR belongs to reward transformation methods and uses preference probabilities relative to reference responses.

  • Characterizing reward hacking: Reward hacking reflects discrepancies between proxy rewards and intended objectives, with prior work linking overoptimization to declining gold reward and response-length effects.
  • Robust reward modeling: Robust reward-modeling approaches include ensembles, weight-space averaging, information bottlenecks, hidden-state regularization, causal augmentation, and length-specific defenses.
  • Policy-side mitigation: Policy-side mitigations constrain optimization, incorporate reward uncertainty, regularize value estimates, penalize final-layer energy, or filter unreliable samples.
  • Reward transformation: Reward transformation modifies the optimization signal without retraining the reward model; PAR uses Bradley–Terry preference probability relative to a reference response.
  • PPO loss curves show greater training stability with PAR than Vanilla, particularly for critic loss, attributed to PAR’s bounded RL reward.

3 Method

The method motivates bounded, rapidly growing, gradually saturating rewards through PPO loss analysis and implements PAR with sigmoid-centered rewards. Theoretical results connect boundedness and sigmoid shaping to variance reduction and training stability.

  • 3.1 Design Principles: The method evaluates PPO policy and critic losses to explain why bounded, rapidly growing, gradually saturating RL rewards support stable optimization.
  • 3.1 Design Principles: GAE uses temporal-difference errors, while the policy and critic losses depend on the current policy, previous policy, and previous critic value function.
  • 3.1 Design Principles: The return is the discounted sum of per-token rewards, which supplies the target used in critic training.
  • 3.1 Design Principles: Excessively large rewards increase return variance, complicate critic regression, and destabilize advantage estimates.
  • 3.2 Reward Shaping: The study focuses on sigmoid-like shaping functions because sigmoid shaping minimizes policy-gradient variance among unbiased functions under logistic preference noise.
  • 3.2 Reward Shaping: PAR applies a sigmoid to centered rewards; its steep slope at zero accelerates early learning, while saturation limits extreme rewards.
  • 3.3 Theoretical Analysis: PAR’s guarantees include an upper bound on return variance and minimum policy-gradient variance among unbiased shaping functions under logistic preference noise.
  • 3.3 Theoretical Analysis: Together, the two variance-reduction properties support stable RLHF training and a longer period of high policy performance before reward hacking emerges.

4 Comparison of Reward-Hacking Mitigation Methods

The experiments validate bounded and rapidly saturating reward shaping, then compare PAR with mitigation methods under PPO. PAR provides the strongest reported performance, data efficiency, and robustness in the evaluated setting.

  • Bounded rewards: Increasing the KL coefficient from 0.01 to 0.1 lowers proxy reward while improving win rate, and smaller reward ceilings produce a similar effect.These results support limiting reward magnitude to improve training stability.
  • Rapid growth and gradual saturation: Centered reward transformations consistently achieve higher win rates than uncentered transformations because they place early rewards near the sigmoid’s largest gradients.Right-shifted SgFc has a small initial gradient and lower early win rates.
  • Mitigation methods: Vanilla PPO’s proxy reward increases while its win rate declines, whereas WARM, Minmax, and PAR mitigate this divergence, with PAR achieving the highest final win rate.ODIN, Reg, Meanstd, Clip, and LSC do not prevent the divergence in this setting.
  • Benchmark performance: PAR achieves the highest AlpacaEval 2.0 win rates and MT-Bench scores among the evaluated shaping methods.The comparison uses checkpoints selected after one epoch of PPO training.
  • Efficiency and robustness: PAR requires only one reference reward, remains robust after two training epochs, and shows calibrated win rate and latent preference before reward hacking.The reported ablations assess data efficiency, robustness, and calibration.

5 Validation of PAR across Base Models and Optimization Algorithms

The study extends PAR-versus-vanilla comparisons across four base models and four optimization algorithms on HH-RLHF. PAR improves PPO robustness on models prone to reward hacking while retaining competitive performance on a more stable model.

  • Experimental setting: The extended evaluation uses HH-RLHF with four base models and compares PPO, A2C, GRPO, and DPO, while PAR is compared with vanilla reward use.The base models are Gemma2-2B, Gemma2-9B, Qwen3-4B-Base, and Llama3.1-8B.
  • Results: For Gemma2-2B, vanilla PPO begins reward hacking after approximately 10k steps, whereas PAR maintains a steadily improving win rate and outperforms DPO.
  • Results: Gemma2-9B and Llama3.1-8B show reward hacking followed by proxy-reward and win-rate collapse under vanilla PPO, while PAR remains stable or preserves win rate above 0.6.For Gemma2-9B, vanilla win rate falls to zero; Llama3.1-8B exhibits collapse over approximately 2.6k–4.2k steps.
  • Results: Qwen3-4B-Base shows no evident reward collapse, with both PAR and vanilla improving gradually.The passage reports that PAR’s effectiveness is competitive on this more stable model.
  • Results: DPO is reported as a simple and effective algorithm for preference datasets, while additional A2C and GRPO results appear in Appendix D.

6 Human Evaluation

The human evaluation tests whether LLM-as-a-judge assessments are reliable for PPO with PAR on HH-RLHF using Gemma2-2B. Two authors independently annotate 200 checkpoint-comparison examples.

  • Human evaluation: The evaluation pool contains 200 examples sampled from test-set comparisons across all training checkpoints.Each example includes a prompt, policy response, reference response, and GPT-5-nano pairwise judgment.
  • Human evaluation: Two authors independently annotate all 200 examples to assess the reliability of GPT-5-nano’s pairwise judgments.
  • Evaluation protocol: The human evaluation uses PPO with PAR on HH-RLHF and Gemma2-2B, with ties counted as half a win for win rates and separately for agreement metrics.

7 Limitations

The paper reports that PAR mitigates reward hacking and stabilizes RLHF training but does not substantially improve peak performance. It also identifies unresolved shaping dynamics and scope limitations across datasets and base models.

  • Performance limitation: PAR does not substantially improve peak performance when measured by the win rate of the best checkpoint.
  • Open questions: The optimal rate of initial reward growth and appropriate saturation pace remain insufficiently understood.
  • Scope boundary: The evaluation uses preference dialogue datasets where DPO already performs well, leaving reasoning datasets for future research.
  • Scope boundary: PAR’s effectiveness is not obvious on Qwen3-4B-Base, indicating that it works on some base models but not all.

8 Conclusion

The paper identifies boundedness and rapid initial growth followed by gradual saturation as principles for reward shaping, and presents PAR as a stable, robust approach across models, datasets, and algorithms.

  • Effective reward shaping requires bounded rewards that grow rapidly at first and then gradually saturate.
  • PAR applies these principles and establishes two variance-reduction properties that contribute to stable RLHF training.
  • PAR achieves strong performance after one training epoch, maintains a high win rate after two epochs, and requires only a single reference reward.
  • PAR demonstrates effectiveness across different base models and Critic-based RL algorithms.

B.1 Part-1 Experiments

The Part-1 experiments use filtered preference data, Gemma2-2B, and PPO-based training, with pairwise judging and benchmark evaluation procedures specified for comparison.

  • Dataset preprocessing: Part-1 experiments use UltraFeedback Binarized data filtered by sequence length, preference scores, and the absence of the substring "confidence".
  • Models and optimization: Gemma2-2B is the base model, with SFT, reward-model, policy, and critic training using the stated learning rates, durations, and gradient clipping.
  • Models and optimization: Policy and critic models are optimized with PPO for one epoch, while the RL setup specifies sampling, KL-penalty, clipping, advantage-estimation, and discount parameters.
  • Evaluation: Responses are compared by pairwise judges using reversed presentation orders to mitigate positional bias.

C.2 Benchmark Evaluation

The benchmark evaluation compares policy responses with SFT responses using win-rate and MT-Bench metrics, while examining data efficiency, prolonged training, and reward–quality calibration.

  • Benchmark metrics: AlpacaEval 2.0 reports length-controlled win rate, raw win rate, and average response length against the SFT model.
  • Benchmark metrics: MT-Bench reports first-turn, second-turn, and overall scores, with overall equal to the average of the two turn scores.
  • Data efficiency: A single reference reward is sufficient for effective PAR training, whereas the uncentered sigmoid baseline performs substantially worse with one reference reward.
  • Robustness: PAR maintains a comparatively high and stable win rate during two PPO epochs, suggesting delayed reward hacking and a wider early-stopping window.
  • Reward calibration: Preference scores track judged win rates initially but become poorly calibrated above approximately 0.8, which bounded PAR limits during optimization.

D.4 Results on A2C and GRPO

Across A2C and GRPO, reward hacking and collapse vary by base model and algorithm; PAR often prevents severe failures, while DPO or vanilla training can achieve higher final win rates in some settings.

  • Gemma2-2B: For Gemma2-2B, vanilla A2C shows reward hacking and severe collapse, while PAR maintains a win rate near 0.74.
  • Gemma2-9B: For Gemma2-9B, vanilla A2C collapses to zero win rate, whereas PAR avoids collapse; DPO achieves the best terminal win rate.
  • Llama3.1-8B: For Llama3.1-8B, PAR reduces GRPO reward–win-rate divergence and ends near 0.62, while DPO reaches approximately 0.74.
  • Qwen3-4B-Base: For Qwen3-4B-Base, GRPO remains stable, with terminal win rates of approximately 0.81 for PAR and 0.84 for vanilla training.
  • Reward-shaping comparisons: The paper contrasts PAR with prior contrastive, response-centric, and log-sigmoid reward transformations while targeting reward hacking and stable policy optimization.
  • PAR properties: PAR applies a sigmoid transformation, has bounded rewards under a sufficiently small η, and is linked to a smaller, more stable critic loss.

F.2 The Justification of the Sigmoid Function

The paper proves that sigmoid-shaped rewards uniquely minimize variance among unbiased shaping estimators under logistic preference noise. It also describes the training algorithms, reward transformations, and observed reward-hacking patterns used in the study.

  • Sigmoid justification: Theorem F.2 identifies the unique minimum-variance unbiased REINFORCE estimator under logistic preference noise.The estimator may use additional randomness but must satisfy pointwise unbiasedness.
  • Sigmoid justification: The minimum-variance estimator is deterministic given the prompt, policy response, and reference response, with reward σ(z(x, y, yref)).Equality requires zero conditional reward variance, forcing ˜r = σ(z(x, y, yref)) almost surely.
  • Sigmoid justification: Theorem 3.2 motivates sigmoid-like functions because their properties minimize policy-gradient variance.This provides the paper’s theoretical justification for bounding reinforcement-learning rewards with sigmoid-like transformations.
  • Reward hacking: Vanilla PPO exhibits reward hacking through high-reward phrases in incoherent text, excessive verbosity, and topical responses that miss user intent.Representative examples include meaningless combinations of technical and aerospace terms.
  • Training methods: The study evaluates PPO, GRPO, A2C, and DPO, with the first three optimizing rewards from on-policy responses and DPO learning from offline preferences.PPO uses actor-critic updates with stability mechanisms including advantage normalization, clipped value loss, replay, and KL penalties.
  • Training methods: GRPO estimates advantages from response groups without a separate critic, assigning each response-level advantage to every token.A small εA is used for numerical stability.
Loading 2502.18770v7…