PaperlayerView exact revision on arXiv

Source-linked AI summary

The Energy Loss Phenomenon in RLHF: A New Perspective on Mitigating Reward Hacking

Yuchun Miao, Sen Zhang, Liang Ding, Yuqi Zhang, Lefei Zhang, Dacheng Tao

arXiv:2501.19358v3cs.LG

TL;DR

RLHF reward hacking arises when optimization under an imperfect proxy reward model diverges from human preferences, and excessive final-layer energy-loss increases characterize this phenomenon. The paper proposes EPPO, which penalizes those increases during reward calculation, and reports that it mitigates reward hacking and improves RLHF performance across models and tasks.

  • Problem

    RLHF reward hacking occurs when optimization under an imperfect proxy reward model diverges from human preferences, producing responses that overfit reward-model-favored patterns.

    Source: §1 · p. 1

  • Method

    EPPO penalizes increases in the LLM’s final-layer energy loss during reward calculation.

    Source: §1 · p. 2

  • Results

    Across four popular LLMs and two representative tasks, EPPO mitigates reward hacking and improves RLHF performance.

    Source: §1 · p. 2

  • Takeaways & Limitations

    Excessive final-layer energy-loss increases characterize reward hacking, and EPPO’s penalty targets this phenomenon without directly constraining output space.

    Source: §1 · p. 2

    The theory leaves open how increased energy loss affects contextual relevance when it exceeds the specified threshold.

    Source: §6.1 · p. 7

Abstract

from arXiv · show

This work identifies the Energy Loss Phenomenon in Reinforcement Learning from Human Feedback (RLHF) and its connection to reward hacking. Specifically, energy loss in the final layer of a Large Language Model (LLM) gradually increases during the RL process, with an excessive increase in energy loss characterizing reward hacking. Beyond empirical analysis, we further provide a theoretical foundation by proving that, under mild conditions, the increased energy loss reduces the upper bound of contextual relevance in LLMs, which is a critical aspect of reward hacking as the reduced contextual relevance typically indicates overfitting to reward model-favored patterns in RL. To address this issue, we propose an Energy loss-aware PPO algorithm (EPPO) which penalizes the increase in energy loss in the LLM's final layer during reward calculation to prevent excessive energy loss, thereby mitigating reward hacking. We theoretically show that EPPO can be conceptually interpreted as an entropy-regularized RL algorithm, which provides deeper insights into its effectiveness. Extensive experiments across various LLMs and tasks demonstrate the commonality of the energy loss phenomenon, as well as the effectiveness of EPPO in mitigating reward hacking and improving RLHF performance.

1. Introduction

The paper links excessive increases in final-layer energy loss during RL to reward hacking and reduced contextual relevance. It proposes EPPO, which penalizes those increases, and reports improved RLHF performance across multiple models and tasks.

1. Introduction

  • Hacking responses show more excessive energy loss than normal responses, while RLHF models generally show higher energy loss than SFT models.The phenomenon is described as a gradual increase in final-layer energy loss during RL, with an excessive increase characterizing reward hacking.Source: §1 · pp. 1–2
  • Reward hacking occurs when optimization under an imperfect proxy reward model diverges from human preferences, producing well-formed but less relevant responses.Existing RL regularizations often constrain output space, which can restrict the policy model’s optimization landscape and compromise RLHF performance.Source: §1 · pp. 1–2
  • Under mild conditions, increased final-layer energy loss reduces the upper bound of contextual relevance, a critical aspect of reward hacking.Reduced contextual relevance typically indicates overfitting to reward-model-favored patterns.Source: §1 · p. 2
  • EPPO outperforms output-constraining RLHF baselines and improves performance across four LLMs and two representative tasks.The response comparison is assessed by GPT-4 on AlpacaFarm; the paper also reports effectiveness in mitigating reward hacking.Source: Figure 2 · p. 2, §1 · p. 2
  • EPPO penalizes increases in final-layer energy loss during reward calculation and can be interpreted theoretically as entropy-regularized RL.Unlike direct output-space constraints, its focus on final-layer energy loss is described as allowing broader policy exploration and optimization.Source: §1 · p. 2

2. Preliminaries

RLHF optimizes a policy using a learned reward proxy, but proxy-reward optimization may diverge from human preferences. Existing regularizations often constrain outputs rather than address internal mechanisms.

2.1. Reinforcement Learning from Human Feedback

  • RLHF uses a learned reward model to score policy responses, with a KL penalty often limiting deviation from the SFT policy; this work uses PPO to optimize the policy.The objective is based on prompts and responses sampled from a dataset.Source: §2.1 · p. 3

2.2. Reward Hacking in RLHF

  • Reward hacking occurs when proxy-reward optimization improves while alignment with human preferences deteriorates, because the reward model is only an imperfect proxy.Existing regularizations often target output-space constraints such as KL divergence and response-length penalties, which may limit the optimization landscape and degrade RLHF performance.Source: §2.2 · p. 3

3. The Energy Loss Phenomenon

The paper defines energy loss and the energy loss phenomenon, then presents evidence that final-layer energy loss rises during RL and is especially high in hacking responses. It also reports a theoretical connection to contextual relevance.

3. The Energy Loss Phenomenon

  • Energy loss is the difference between the L1 norms of a layer’s input and output hidden states, averaged across response tokens for generation tasks.The paper defines the energy loss phenomenon as final-layer energy loss gradually increasing during RL, with an excessive increase indicating reward hacking.Source: §3 · p. 3

3.1. Empirical Evidence on Energy Loss Phenomenon

  • Hacking responses show a more excessive energy-loss increase than normal responses, while EPPO suppresses the increase in the displayed PPO/EPPO distributions.The Figure 4 caption identifies rows as PPO and EPPO and columns as Llama3-8B, Mistral-7B, and DeepSeek-7B; GPT-4 identifies hacking samples.Source: Figure 4 · p. 4, §3.1 · p. 4
  • Final-layer energy loss gradually increases during RL across the examined models and datasets, supporting the first characteristic of the energy loss phenomenon.The main analysis covers Llama3-8B, Mistral-7B, DeepSeek-7B, and AlpacaFarm; additional models and datasets are reported in appendixes.Source: §3.1 · p. 4
  • The theoretical discussion links excessive final-layer energy loss with suppressed contextual relevance, a trend associated with overfitting to reward-model-favored patterns.The paper states this connection as a theoretical foundation for the empirical findings.Source: §3.1 · p. 4

4. Energy Loss-Aware PPO (EPPO)

EPPO penalizes energy-loss increase in reward calculation, aiming to curb excessive increases and reward hacking. The paper also describes EPPO as conceptually entropy-regularized and compatible with reward-modeling methods.

4. Energy Loss-Aware PPO (EPPO)

  • EPPO precomputes SFT final-layer energy loss, computes the RLHF counterpart, and uses their energy-loss variation to penalize the reward-model score.The variation is empirically equivalent to an increase during RL; η is the trade-off parameter.Source: §4 · p. 4
  • The paper presents EPPO as compatible with advanced reward-modeling techniques, with combinations described as further enhancing RLHF performance.Table 2 compares methods on Llama3-8B under GPT-4 evaluation.Source: Table 2 · p. 5
  • EPPO suppresses excessive energy-loss increases and is reported to mitigate reward hacking.The paper interprets EPPO as conceptually an entropy-regularized RL algorithm.Source: §4 · p. 5

5. Experiments

Across four LLMs and two tasks, the experiments evaluate EPPO against PPO, reward-hacking mitigation methods, and reward-modeling approaches. EPPO improves RLHF performance and is reported to mitigate reward hacking across several evaluations.

5.1. Setup

  • The setup evaluates four LLMs on dialogue and summarization, using ShareGPT and Anthropic-HH for dialogue and Reddit TL;DR for summarization; evaluation includes in- and out-of-distribution dialogue data.Baselines include SFT, PPO, PPO with KL or length penalties, and six reward-modeling methods; GPT-4, Claude-3.5-Sonnet, and human evaluation are used.Source: §5.1 · p. 5

5.2. Main Results of RLHF Performance

  • EPPO outperforms reward-hacking mitigation and reward-modeling baselines, combines with reward-modeling methods for further gains, and consistently improves performance across additional LLMs.The cross-model comparison is reported under GPT-4 evaluation.Source: §5.2 · p. 6, Table 3 · p. 6

5.3. Main Results of Reward Hacking Mitigation

  • GPT-4 identification finds that penalizing energy loss with EPPO effectively prevents reward hacking, with additional results reported across LLMs and datasets.Hacking samples are identified using summarized common behaviors and GPT-4 feedback; the authors report over 95% human agreement for this procedure.Source: §5.3 · p. 6, §5.1 · p. 6
  • On AlpacaFarm win-rate trajectories, PPO and PPO with length penalties decline in later RL stages, while EPPO mitigates hacking and boosts final performance.PPO with KL mitigates hacking but restricts policy exploration and limits performance gains.Source: §5.3 · p. 6, Figure 5 · p. 7
  • EPPO reduces outliers in InfoRM’s latent response space, which the analysis uses to identify hacking responses.The visualization covers SFT and RLHF responses from Llama3-8B on AlpacaFarm.Source: §5.3 · pp. 6–7, Figure 6 · p. 7

6. Discussion

The discussion connects energy loss to contextual relevance and output entropy, then reports empirical patterns across models and layers. It identifies a threshold condition on the theoretical relationship between energy loss and contextual information.

6.1. Theoretical Analysis on Energy Loss Phenomenon

  • When energy loss is below a threshold, an increase reduces the upper bound on contextual mutual information; above that threshold, its effect on contextual relevance remains open.The theory connects energy loss with contextual information, while the empirical analysis reports a negative correlation between final-layer energy loss and response relevance.Source: §6.1 · p. 7

6.2. EPPO vs. Entropy-Regularized RL

  • EPPO’s energy-loss penalty also limits excessive drops in output entropy, motivating its conceptual interpretation as entropy-regularized RL with potential stability and exploration benefits.The claimed inverse relationship between energy loss and the entropy upper bound applies below a threshold.Source: §6.2 · p. 8

6.3. Energy Loss vs. Response-Context Relevance

  • Across three representative LLMs, excessive final-layer energy loss suppresses response-context relevance, potentially accompanying overfitting to reward-model-favored patterns.Contextual dependency strength is measured by the perplexity difference of a response with and without context.Source: §6.3 · p. 8, Figure 7 · p. 8

6.4. Energy Loss Distribution across All LLM Layers

  • The final layer shows the most pronounced and stable energy-loss pattern across models and datasets; earlier-layer patterns are less consistent across architectures, training schemes, and data.The authors hypothesize that the final layer is a more reliable indicator because it is closest to the output and captures richer semantic information.Source: §6.4 · p. 8

7. Related Work

The related work groups reward-hacking mitigation into reward-model robustness and policy regularization, then positions EPPO as an internal-representation approach. It also connects the study to representation engineering.

7.1. Reward Hacking Mitigation in RLHF

  • Prior mitigation enhances reward-model robustness or regularizes policy updates, whereas EPPO regularizes final-layer energy loss rather than constraining output space.The authors state that this focus permits a broader policy optimization space and more effective reward-hacking mitigation.Source: §7.1 · pp. 8–9, Figure 8 · p. 9

7.2. Representation Engineering for Neural Networks

  • Building on representation-engineering work, the paper treats reward hacking as manifesting in LLM representations and introduces a tailored mitigation strategy.The cited related work includes regulating representations for honesty, fairness, and harmlessness.Source: §7.2 · p. 9

8. Conclusion

The paper identifies increasing final-layer energy loss during RL as a phenomenon associated with reward hacking and proposes EPPO to penalize its increase. It reports theoretical, empirical, and performance findings supporting this approach.

8. Conclusion

  • Final-layer energy loss gradually increases during RL, and excessive energy loss empirically corresponds to reward hacking.The paper characterizes this pattern as an underlying mechanism of reward hacking.Source: §8 · p. 9
  • EPPO adds an energy-loss-based penalty to rewards and focuses on the LLM’s final layer rather than constraining output space.The paper interprets EPPO theoretically as entropy-regularized RL.Source: §8 · p. 9
  • The paper reports that increased energy loss suppresses contextual response relevance and that experiments show the phenomenon across LLMs and tasks alongside improved RLHF performance with EPPO.Source: §8 · p. 9

Impact Statement

The paper frames reward hacking as apparent success under a proxy reward model that departs from human objectives, and proposes energy loss-aware PPO to mitigate it.

Impact Statement

  • Reward hacking is policy optimization that appears effective under a proxy reward model but deviates from true human objectives.The paper says EPPO mitigates reward hacking and enhances RLHF performance, and reports no ethical concerns or adverse societal effects.Source: Impact Statement · p. 10

A. Proof of Theorem and Corollary

The appendix extends the energy-loss analysis across models and tasks, reports that EPPO suppresses excessive energy-loss growth and hacking, and presents theoretical bounds relating energy loss to information measures.

A. Proof of Theorem and Corollary

  • A variational upper bound is derived for I(X;Y). Theorem 6 states that contextual mutual information I(X;Y) satisfies a bound for any transformer-block layer ℓ, with σ a positive constant.Source: A. Proof of Theorem and Corollary · pp. 14–15
  • Under a threshold condition, energy loss is negatively correlated with output entropy.Corollary 5 states that energy loss also contributes to an upper bound for output entropy.Source: A. Proof of Theorem and Corollary · p. 14

B. More Results on Energy Loss Dynamics.

C. More Results on Energy Loss Distribution.

D. More Results on Reward Hacking Mitigation of EPPO

  • Across the additional settings, EPPO consistently mitigates reward hacking, as assessed using GPT-4 identification and win-rate dynamics.PPO shows a sharp late-training GPT-4 win-rate decline; PPO with KL or length penalties improves stability but limits policy exploration and performance gains.Source: §D.1 · p. 21, §D.2 · p. 21, Figure 14 · p. 21

D.2. GPT-4 Win Rates Dynamics in RL

  • On Llama3-8B, PPO's GPT-4 win rate declines sharply late in training, whereas EPPO achieves superior final RLHF performance while constraining final-layer energy loss.The comparison covers AlpacaFarm, Anthropic Helpful, Anthropic Harmless, and Reddit TL;DR.Source: §D.2 · p. 21, Figure 14 · p. 21

D.3. Representation Analysis using InfoRM

The appendix compares EPPO with PPO and reward-modeling approaches using InfoRM representations, evaluator win-tie-loss results, and hyperparameter sensitivity.

D.3. Representation Analysis using InfoRM

  • InfoRM latent-space visualizations show EPPO consistently reduces outliers relative to PPO across additional evaluation datasets.The paper treats InfoRM-space outliers as typical hacking samples, following Miao et al. (2024).Source: §D.3 · p. 22, Figure 15 · p. 22

E. Reward Hacking Mitigation of More Compared Methods

F. More Win-Tie-Lose Results under Various Evaluators

  • EPPO consistently outperforms constrained RLHF algorithms across evaluators in win-tie-loss comparisons.The evaluation uses Llama3-8B and representative general-dialogue and summarization datasets, with Claude-3.5 (Sonnet), GPT-4, and human evaluation.Source: F. More Win-Tie-Lose Results under… · p. 23, Table 4 · p. 23

G. Sensitivity Analysis of Hyperparameter in EPPO

H. Comparison Results with Different KL Penalties

The appendix reports EPPO's advantages over KL-constrained PPO, validates hacking annotators against human judgments, and describes experimental methods and qualitative hacking patterns.

H. Comparison Results with Different KL Penalties

I. Human Evaluation in Reward Hacking Identification

  • Human annotations agree with GPT-4 at 97.5% for general dialogue and with InfoRM at 95% for summarization.Two expert annotators reviewed 200 cases each from AlpacaFarm and Reddit TL;DR; disagreements were reassessed to reach consensus.Source: §I · p. 24

J.1. Training Setup

  • The experimental setup initializes SFT models from pretrained checkpoints and trains reward models by replacing the SFT model's final layer with a linear reward-scoring layer.The supplied training details include single-epoch SFT and reward-model training, each with global batch size 64.Source: §J.1 · p. 25

J.2.1. RL ALGORITHMS FOR MITIGATING REWARD HACKING

  • The appendix describes PPO, KL and length-penalty variants, ensemble and averaged reward models, ODIN, and InfoRM as comparison methods.The methods address reward hacking through output-space constraints or reward-modeling strategies.Source: §J.2.1 · p. 25, §J.2.2 · p. 25

J.4. Contextual Dependency Strength

  • The contextual-dependency metric is tailored to general dialogue and treats the instruction as x and the LLM response as y.The appendix also includes GPT-4 prompts for evaluation and hacking identification.Source: §J.4 · p. 26, Figure 18 · p. 26

K. Hacking Examples

  • Hacking examples include excessive caution or verbosity in dialogue and overly detailed, irrelevant, or repetitive summaries.Most examples share weakened contextual relevance, with responses gradually deviating from user intent and overfitting reward-model-favored patterns.Source: K. Hacking Examples · p. 26, Figure 19 · p. 27

L. Qualitative Examples

  • Output-space constraints can limit policy exploration, while EPPO's final-layer energy-loss constraint is reported to mitigate hacking and enable broader optimization.The paper also notes that length-penalty PPO remains somewhat vulnerable to hacking such as overly cautious responses.Source: L. Qualitative Examples · p. 28, Figure 20 · p. 28, Figure 21 · p. 29
Loading 2501.19358v3…