PaperlayerView exact revision on arXiv

Source-linked AI summary

ODIN: Disentangled Reward Mitigates Hacking in RLHF

Lichang Chen, Chen Zhu, Davit Soselia, Jiuhai Chen, Tianyi Zhou, Tom Goldstein, Heng Huang, Mohammad Shoeybi, Bryan Catanzaro

arXiv:2402.07319v1cs.LGcs.AIcs.CL

TL;DR

RLHF reward models can favor verbose responses that score well without becoming more helpful, making response-length hacking a concern. ODIN uses a two-head reward model and evaluates score–length trade-offs; the paper reports higher Pareto fronts than previous results across PPO and ReMax, with reduced reward–length correlation.

  • Problem

    RLHF policies can exploit imperfect reward models to obtain high reward without fulfilling the actual objective, and verbosity can gain scores without improving response quality.

    Source: §1 · pp. 1–2

  • Method

    The paper evaluates training configurations through score–length Pareto fronts and trains a shared-representation, two-head reward model that separates length from quality, using only the quality reward in RL.

    Source: §1 · p. 2, §3.3 · p. 6

  • Results

    ODIN achieves a higher Pareto front than previous results with a less expensive tuning budget for both PPO and ReMax, while reducing reward correlations with length and maintaining most preference-prediction accuracy.

    Source: §1 · p. 2, §4.2 · p. 10

  • Takeaways & Limitations

    The reported findings suggest reward disentanglement can reduce length hacking while improving policy score–length trade-offs across the studied RL-tuning algorithms.

    Source: §1 · p. 2, §6 · p. 11

    Perfect reward correlation and decorrelation are difficult in practice because training uses minibatches and the reward model must generalize to out-of-distribution RL examples.

    Source: §3.3 · p. 7

Abstract

from arXiv · show

In this work, we study the issue of reward hacking on the response length, a challenge emerging in Reinforcement Learning from Human Feedback (RLHF) on LLMs. A well-formatted, verbose but less helpful response from the LLMs can often deceive LLMs or even human evaluators to achieve high scores. The same issue also holds for some reward models in RL. To address the challenges in both training and evaluation, we establish a more reliable evaluation protocol for comparing different training configurations, which inspects the trade-off between LLM evaluation score and response length obtained by varying training hyperparameters. Based on this evaluation, we conduct large-scale studies, where the results shed insights into the efficacy of hyperparameters and tricks used in RL on mitigating length bias. We further propose to improve the reward model by jointly training two linear heads on shared feature representations to predict the rewards, one trained to correlate with length, and the other trained to decorrelate with length and therefore focus more on the actual content. We then discard the length head in RL to prevent reward hacking on length. Experiments demonstrate that our approach almost eliminates the reward correlation with length, and improves the obtained policy by a significant margin.

1 Introduction

RLHF can produce high-reward responses that miss the actual objective, with verbosity a common form of length hacking. The paper evaluates score–length trade-offs and proposes ODIN, which separates length and quality rewards and uses only the quality reward during RL.

1 Introduction

  • Reward hacking occurs when a policy obtains high reward without fulfilling the actual objectives, partly because reward models imperfectly proxy human preferences and generalize poorly out of distribution.Verbosity is a common form: responses become longer or better formatted without improved quality, exploiting human preference for longer answers.Source: §1 · pp. 1–2
  • ODIN achieves a higher Pareto front than prior results despite a less expensive tuning budget, with the conclusion holding for PPO and ReMax.The paper’s evaluation compares score against response length across training configurations.Source: §1 · p. 2
  • The paper establishes a large-scale grid-search protocol that compares evaluation score and response length through Pareto fronts to reduce conclusions’ sensitivity to length bias.It then studies reward clipping and length penalties, but reports difficulty drawing simple tuning principles across the hyperparameters.Source: §1 · p. 2
  • ODIN’s two-head reward model separates length and quality signals, then discards the length reward during RL fine-tuning.It is trained on the same human preference data as a vanilla reward model.Source: Figure 1 · p. 2

2 Preliminaries

The paper outlines the RLHF pipeline and the reward-model and RL objectives used in its studies. It compares REINFORCE with baseline, including ReMax, and PPO, whose clipping objective can help limit reward over-optimization.

2 Preliminaries

  • RLHF comprises supervised fine-tuning, reward modeling from the SFT checkpoint, and RL using that checkpoint as initialization with reward-model feedback.Source: §2 · p. 2
  • Figure 2 compares PPO and ReMax policies trained with vanilla reward models or ODIN against DPO on human preference data.The PPO* and ReMax* comparisons aggregate reward-clipping and length-penalty results using a broader, more compute-intensive search than ODIN.Source: Figure 2 · p. 3
  • The reward model is initialized from a supervised fine-tuned language model with a linear scalar-reward head, and trained on chosen–rejected response pairs under the Bradley–Terry model.The loss maximizes the log sigmoid of the reward difference between chosen and rejected responses.Source: §2 · p. 3
  • RL fine-tuning maximizes reward-model feedback while KL regularization limits drift from the SFT policy; the paper uses a naïve KL estimator to define an auxiliary reward.Source: §2 · p. 3
  • The study compares REINFORCE with baseline, including ReMax, and PPO; PPO’s clipping objective can limit optimization on samples once their likelihood ratio exceeds 1 + ϵ.The text says this clipping can help reduce reward hacking.Source: §2 · pp. 3–4
  • Figure 3 reports hyperparameter sweeps: smaller PPO clipping ϵ can improve results, larger off-policy batches are not beneficial, and some reward-clipping thresholds outperform the unclipped baseline but require tuning.The reported effects of β are marginal near SFT initialization, and clipping’s trend becomes complicated when reward clipping is enabled.Source: Figure 3 · p. 4

3 Mitigating Reward Hacking in Practice

The section compares RLHF configurations using evaluation-score versus response-length trade-offs, then examines RL tuning and a reward-model approach that separates length from quality. ODIN’s disentangled reward maintains validation accuracy while reducing length correlation and improving policies.

3.1 Evaluation

  • The evaluation protocol compares model-based win scores against average response length, treating a higher Pareto front as better at a given length.A win score of at least 50 means the test model is no worse than the SFT baseline.Source: §3.1 · p. 5

3.2 How much hacking can we mitigate by tuning RL?

  • Greater KL weight limits length growth but worsens win score; smaller PPO ε improves the Pareto front by around 2.5 points, though reward clipping makes ε harder to tune.Reward clipping also marginalizes KL’s effect.Source: §3.2 · p. 5
  • More off-policy sampling (N > b) can produce longer responses and lower win scores near SFT length, and RL tuning offers no simple heuristic for a much better Pareto front.The reported comparison finds pure on-policy sampling has a higher win score near the SFT response length.Source: §3.2 · p. 5
  • Reward clipping can improve PPO and ReMax at a suitable threshold, but aggressive clipping such as c = 1 can hinder learning; length penalty improves Pareto fronts less than ODIN.Length penalty uses a tunable coefficient, whereas ODIN incorporates length decorrelation into reward-model training without adding RL hyperparameters.Source: §3.2 · pp. 5–6, Figure 4 · p. 6

3.3 Reward Disentanglement: a more reliable approach

  • ODIN uses shared reward-model representations to predict separate length and quality rewards, then uses the quality reward in RL; orthogonal projection weights further separate the rewards.The length reward is trained to correlate with response length while the quality reward is trained to decorrelate from it.Source: §3.3 · pp. 6–7
  • Perfect correlation and decorrelation may be difficult to achieve in practice because training uses minibatches and the reward model must generalize to OOD examples in RL.Source: §3.3 · p. 7
  • ODIN maintains validation accuracy while drastically reducing reward–length correlation; orthogonality further reduces length correlation and improves RL policies.Rank correlations are also eliminated despite training to minimize Pearson correlation with length.Source: §3.3 · p. 7

4 Experiments

The experiments compare reward models and training configurations using evaluation score against response length, then test ODIN against baselines on model-based, human, and benchmark evaluations. ODIN’s evaluation front is higher than the respective baselines at lengths of at least 210, and human raters prefer its models at matched lengths.

4.1 Settings

  • The study trains reward models and policies from Vicuna-7B using OpenAssistant preference data, and tests ODIN with ReMax and PPO.The processed data contain 22,065 reward-model examples and 7,494 RL prompts; the policy training compares ODIN across two RL algorithms.Source: §4.1 · pp. 7–8
  • GPT-4 evaluation uses Win Score and judges policies by the score–length Pareto front; selected models are also compared in human studies.The evaluation uses LIMA prompts, while capability benchmarks cover BBH, MMLU, DROP, and TruthfulQA.Source: §4.1 · pp. 8–9

4.2 Results

  • At roughly matched lengths, human raters preferred every examined ODIN model over the baselines, with the difference increasing at longer lengths.Figure 6 compares models at lengths close to 220, 238, and 265.Source: Figure 6 · p. 9, §4.2 · p. 10
  • At every length scale, ODIN policies outperform vanilla-reward-model policies; they improve TruthfulQA and maintain other benchmark performance relative to SFT.Table 3 reports the benchmark results for policies selected at different average response lengths.Source: Table 3 · p. 9, §4.2 · p. 10
  • ODIN eliminates reward–length rank correlations while retaining most preference-prediction accuracy, despite 66% of test pairs having a longer chosen response.The reported reduction in length correlation is substantial; the accuracy drop is described as insignificant in that context.Source: Table 1 · p. 8, §4.2 · p. 10
  • ODIN’s policy Pareto front is higher than the respective PPO* and ReMax* baselines when response length L(y) ≥ 210.The authors note that lengths below 210 may indicate lower quality because the SFT model has L = 220.Source: §4.2 · p. 10

5 Related Works

Related work examines reward hacking, length bias in evaluators, and methods for reducing reward-model or policy vulnerabilities. The paper focuses on online RL and evaluates policies through score–length trade-offs.

Learning from Feedbacks

  • Offline feedback-learning methods update from annotated generations or preferences without sampling from the policy during training, while the paper studies reward models in online algorithms.Offline methods may be less prone to hacking as experiences update less frequently, but the passage notes long-term hacking can still occur.Source: §5 · p. 10

Mitigating Reward Hacking in RLHF

  • Prior mitigation work includes PPO regularization and reward-model data changes, smaller bias-learning reward models, ensembles, and weight-averaged reward models.Reported shortcomings include residual length increase, lower reward, reduced reward-model accuracy, or remaining reward hacking.Source: §5 · p. 11

LLM evaluations

  • Because LLM evaluators can reward excessive verbosity, the paper compares evaluation-score versus response-length Pareto fronts and also tests capability benchmarks.The listed benchmarks are BBH, MMLU, DROP, and TruthfulQA.Source: §5 · p. 11

6 Conclusion

The conclusion presents ODIN as a way to disentangle content quality from response length and reports improvements across Pareto fronts and two RL algorithms. The appendix also gives a PPO procedure for sampling responses and updating the value network and policy.

6 Conclusion

  • ODIN improves the score–verbosity Pareto front, and its gains transfer across ReMax and PPO.The conclusion describes hyperparameter and trick studies but says their combined tuning makes definitive tuning conclusions difficult; generalizing to other hacking types remains future work.Source: §6 · p. 11
  • The conclusion identifies evaluation and generalization of ODIN on other forms of reward hacking as future work.Source: §6 · p. 11

A Appendix

  • The appendix PPO algorithm samples prompt–completion batches from the current policy, computes rewards and advantages, then updates the value network and clipped policy.It initializes policy parameters from the SFT model and refreshes the old policy after the update loops.Source: A Appendix · p. 16

B Human Study

The human-study interface presents participants with an evaluation screen after consent. It includes session tracking and response-evaluation guidance, with criteria listed separately.

B Human Study

C Correlation Metric

The appendix names three correlation metrics and gives formulas for computing them. It also defines the rank difference used in Spearman’s rank correlation and the observation count.

C Correlation Metric

  • The paper uses Pearson ρ, Spearman’s rank correlation r_s, and Kendall’s τ as its three correlation metrics.It computes all three using the formulas that follow.Source: C Correlation Metric · p. 16
  • For Spearman’s rank correlation, d_i is the difference between the two ranks for observation i, and n is the number of observations.Source: C Correlation Metric · p. 16

D Evaluation Prompt

Table 5 presents the GPT-4 evaluation prompt.

D Evaluation Prompt

E.1 Model generation config.

The generation settings differ between RLHF training and evaluation: training encourages exploration, while evaluation settings aim to avoid over-randomness.

E.1 Model generation config.

  • Training generation uses top_p=0.9 and T=1.0 for exploration; evaluation uses top_p=0.8 and T=0.8 to avoid over-randomness.Source: §E.1 · p. 18

E.2 PPO config

The appendix reports PPO and ReMax fine-tuning configurations and the ReMax reward-clipping sweep.

E.2 PPO config

  • PPO uses full-model actor and critic fine-tuning for three Open Assistant epochs, with 702 updates, batch size 32, and a shared learning rate.The PPO search varies η, ϵ, β, c, and N; experiments with β=2.5e−3 are partial.Source: §E.2 · p. 18

E.3 ReMax Config

  • ReMax uses full-model fine-tuning for three prompt-set epochs; because actor lengths mostly exceeded 225, β values were expanded at η=5e−7.The added sweep was intended to improve comparisons between Pareto fronts across different lengths.Source: §E.3 · p. 18
  • Figure 8 shows the effect of reward clipping on ReMax across swept η and β values.Source: Figure 8 · p. 19

E.4 Configs for Length Penalty Experiments

The appendix documents experimental settings and evaluation choices, then reports a case-study comparison in which ODIN models generate more accurate answers with shorter length.

E.4 Configs for Length Penalty Experiments

  • For Fig. 4, ReMax and PPO used separate α sweeps while other RL hyperparameters were matched across settings.The length-penalty setting consequently had more data points.Source: §E.4 · p. 19

E.5 Dialogue Format

  • OpenAssistant prompts and responses were formatted as Human/Assistant turns, with all prior turns included as model input for multi-turn dialogue.Source: §E.5 · p. 19

F.1 Why do not use other prompt sets for evaluating the models’ capability on free-form QA?

  • LIMA, with 300 prompts, was used to evaluate instruction following; main evaluations used GPT-4, with selected Pareto-front models also assessed by humans.Source: §F.1 · p. 19

F.2 Why do you choose Vicuna-7B as the base model or the starting point of the RL?

  • Vicuna-7B was selected for its instruction-following capability and as a well-known SFT model, avoiding the authors’ own OpenAssistant SFT checkpoint selection.The authors say using Vicuna-7B and their reward model could make results easier to reproduce.Source: §F.2 · p. 19
  • Across the hyperparameter sweeps, reward clipping improved results, smaller β could yield better results, and larger experience batches were not beneficial.The effect of ϵ was complicated; ϵ clipping was ineffective at N = 32.Source: Figure 10 · p. 20, Figure 9 · p. 20, Figure 11 · p. 20

G Case Study

Loading 2402.07319v1…