PaperlayerView exact revision on arXiv

Source-linked AI summary

Scaling Laws for Reward Model Overoptimization in Direct Alignment Algorithms

Rafael Rafailov, Yaswanth Chittepu, Ryan Park, Harshit Sikchi, Joey Hejna, Bradley Knox, Chelsea Finn, Scott Niekum

arXiv:2406.02900v2cs.LGcs.AIcs.CL

TL;DR

Direct Alignment Algorithms (DAAs) avoid explicit reward modeling but still lack a clear characterization of reward over-optimization. This paper unifies DAA methods and empirically shows consistent degradation across KL budgets, often within a single training epoch, while larger models are less prone to over-optimization.

  • Problem

    Recent alignment methods bypass reward modeling, leaving their reward over-optimization phenomena insufficiently characterized.

  • Method

    The paper unifies DPO, IPO, and SLiC under a DAA framework and evaluates them across objectives, KL budgets, training regimes, and model scales.

  • Results

    DAAs consistently show hump-shaped over-optimization across KL budgets, with performance often declining after 25% of training data and larger models generally less prone to degradation.

  • Takeaways & Limitations

    Over-optimization remains a substantive concern for DAAs, whose under-constrained objectives can produce undesirable behavior on out-of-distribution responses.

  • Takeaways & Limitations

    The analysis assumes a nontrivial intersection between the null space and data span and the existence of an unseen out-of-distribution response for every prompt.

Abstract

from arXiv · show

Reinforcement Learning from Human Feedback (RLHF) has been crucial to the recent success of Large Language Models (LLMs), however, it is often a complex and brittle process. In the classical RLHF framework, a reward model is first trained to represent human preferences, which is in turn used by an online reinforcement learning (RL) algorithm to optimize the LLM. A prominent issue with such methods is reward over-optimization or reward hacking, where performance as measured by the learned proxy reward model increases, but true quality plateaus or even deteriorates. Direct Alignment Algorithms (DDAs) like Direct Preference Optimization have emerged as alternatives to the classical RLHF pipeline by circumventing the reward modeling phase. However, although DAAs do not use a separate proxy reward model, they still commonly deteriorate from over-optimization. While the so-called reward hacking phenomenon is not well-defined for DAAs, we still uncover similar trends: at higher KL budgets, DAA algorithms exhibit similar degradation patterns to their classic RLHF counterparts. In particular, we find that DAA methods deteriorate not only across a wide range of KL budgets but also often before even a single epoch of the dataset is completed. Through extensive empirical experimentation, this work formulates and formalizes the reward over-optimization or hacking problem for DAAs and explores its consequences across objectives, training regimes, and model scales.

1 Introduction

RLHF has enabled major LLM advances but is complex and vulnerable to reward over-optimization. This work shows that Direct Alignment Algorithms, despite bypassing reward modeling, exhibit similar over-optimization and degradation patterns across methods, model scales, and KL budgets.

  • RLHF improves LLM capabilities but requires a complex, computationally intensive pipeline that trains a reward model before reinforcement-learning optimization.Users evaluate responses to train a reward model, after which the LLM maximizes its learned reward function.
  • RLHF suffers reward over-optimization because optimizing an imperfect learned reward amplifies the reward model’s shortcomings.This limitation is framed through Goodhart’s Law and has been repeatedly observed empirically.
  • Direct Alignment Algorithms bypass traditional reward-model fitting by directly re-parameterizing the reward through the optimal policy, reducing computational demands while retaining similar over-optimization trends.Direct Preference Optimization is identified as a prominent example of this algorithm class.
  • The work unifies several recent methods under the DAA framework and experimentally finds RLHF-consistent reward over-optimization across model scales and KL-divergence budgets.The study specifically investigates over-fitting phenomena in DAAs through extensive experimentation.

2 Preliminaries

This section introduces the three-stage RLHF pipeline, explains reward over-optimization from surrogate rewards, and presents DAAs as direct policy-updating alternatives that nevertheless exhibit similar overfitting behaviors.

  • Standard RLHF pipeline: Standard RLHF trains an instruction-following model, fits a preference reward model from human rankings, and optimizes the policy with on-policy reinforcement learning.The final objective uses a KL penalty against a reference policy to balance reward exploitation with deviation from initialization.
  • Reward over-optimization: RLHF can increase surrogate reward while decreasing output quality because the policy optimizes the learned reward rather than the true reward.This over-optimization arises from issues including out-of-distribution reward queries and reward mis-specification.
  • Direct Alignment Algorithms: Direct Alignment Algorithms update the language-model policy directly from user feedback, avoiding explicit reward modeling, on-policy sampling, and reinforcement learning.DPO is a prominent example derived from the closed-form RLHF solution.
  • Direct Alignment Algorithms: The section analyzes DPO, IPO, and SLiC-HF using convex classification losses, including logistic, quadratic, and hinge objectives.DPO uses the Bradley-Terry logistic loss, IPO uses a quadratic loss, and SLiC-HF uses a hinge loss.
  • DAA over-optimization: Despite avoiding an explicit reward model, DAAs exhibit strong overfitting and reward-hacking-like behavior, with DPO responses lengthening without continued ground-truth win-rate gains.The section’s evaluation compares DPO, IPO, and SLiC using model win-rates judged by GPT-4 across training checkpoints.

3 Empirical Analysis of Overoptimization in DAAs

DAAs exhibit systematic reward over-optimization across objectives, model sizes, KL budgets, training stages, and implicit-reward diagnostics. The analyses further show scaling-law behavior, feature exploitation, and weak links between training statistics and downstream performance.

  • Model Over-Optimization: DAA performance follows a hump-shaped KL pattern, with wider-KL configurations often peaking after 25% of an epoch before degrading as KL divergence rises.This pattern appears across DPO, IPO, and SLiC, with 6.9B models shown in the intra-epoch analysis.
  • Effect of Training Objective and Model Size: IPO is less prone to over-optimization than DPO and SLiC, while larger Pythia models show more favorable win-rate–KL trade-offs than the 1B model.The 1B model reaches higher KL values under the same constraints and quickly exhibits over-optimization across all three objectives.
  • Scaling Laws: The classical reward scaling law relates KL divergence to DAA GPT4 winrates and halves RMSE versus a quadratic KL fit.A quadratic fit using the transformed scaling variable achieves similar error to the reported scaling law.
  • Feature Exploitation: Length regularization changes the KL–win-rate Pareto frontier but does not alleviate over-optimization, while weaker models extrapolate more strongly through length.The length-extrapolation effect is clearest for Pythia 1B compared with the 2.8B and 6.9B models.
  • Implicit Reward Diagnostics: Implicit reward-model accuracy has little to no relationship with downstream performance for DPO and SLiC, while IPO’s weak positive relationship is explained by model-size scaling.Optimization loss and other training statistics likewise do not exhibit a strong relationship with checkpoint win rates.
  • Implicit Reward Dynamics: Decreasing implicit rewards during training are compatible with improvement, but performance eventually dips after a threshold, revealing nonlinear over-optimization dynamics.The analysis interprets decreasing likelihoods as expected as the optimization policy moves away from the reference policy.

4 Reward Exploitation in Direct Alignment Algorithms

DAA objectives are under-constrained because finite preference data leave a large null space, allowing equally optimal policies to assign probability to out-of-distribution responses. This implicit-reward ambiguity is reflected in both a formal proposition and toy-MDP training behavior.

  • Implicit Reward Ambiguity: Because DAAs deterministically map their implicit reward to a policy, under-constrained reward modeling directly links policy OOD behavior to the implicit reward’s OOD behavior.DAAs do not train a separate reward model; instead, they fit an implicit reward to preference data.
  • Rank Deficiency with Finite Preferences: Finite preference datasets make the DAA objective non-strictly convex, yielding multiple optimal policies, including solutions that place high weight on out-of-distribution responses.The prompt-response space is exponentially large, while the preference matrix has far fewer rows, creating a non-trivial null space.
  • Rank Deficiency with Finite Preferences: Under the proposition’s null-space and unseen-response conditions, the DAA loss has infinitely many minima that place weight on out-of-distribution responses.The first condition can arise from conflicting preferences, while unseen responses are common because datasets cover only a tiny portion of response space.
  • Understanding OOD behavior for DAA algorithms with a Toy MDP: In a toy tree MDP, DAA training assigns substantial probability mass to some OOD trajectories while reducing probability on in-distribution preference-pair trajectories.The experiment uses a single preference and updates an RNN policy after supervised fine-tuning on preferred responses.

5 Related Work

Over-optimization spans supervised learning and reinforcement learning, where it can harm generalization, enable adversarial vulnerability, or produce reward-hacking behavior. In LLM alignment, both standard RLHF and direct alignment algorithms have persistent over-optimization issues, motivating methods for more robust reward functions and further study of DAA-specific failures.

  • Broader over-optimization: Over-optimization appears across supervised learning and reinforcement learning, including overfitting that harms generalization or enables adversarial attacks and reward hacking that diverges from intended goals.The paper situates its contribution within broader work on over-optimization, overfitting, and reward hacking.
  • RLHF: Standard RLHF can suffer both reward-function overfitting and exploitation by the reinforcement-learning algorithm, motivating ensemble and robustness approaches.Related work includes RLHF techniques pioneered for control and methods using ensembles to reduce reward overfitting or improve reward robustness.
  • Direct alignment algorithms: Unlike standard RLHF, direct alignment algorithms have also shown persistent over-optimization across language modeling, image generation, and control, including overfitting to length at the expense of performance.The paper distinguishes its focus from Gao et al. and notes theoretical and applied DAA work across several domains.

6 Conclusion

The work analyzes over-optimization in Direct Alignment Algorithms, finding consistent trends across DPO, IPO, and SLIC, model scales from 1B to 6.9B, and different KL-divergence budgets.

  • 6 Conclusion: Across DPO, IPO, and SLIC, experiments at 1B, 2.8B, and 6.9B model scales show consistent over-optimization trends across different KL-divergence budgets.The authors frame this analysis as a first step and identify larger model scales as an avenue for further characterization.

A Limitations and Societal Impacts

The work identifies theoretical and practical issues in direct alignment algorithms but does not provide solutions. It also relies on an imperfect underlying model of human preferences.

  • A Limitations and Societal Impacts: The work highlights unresolved issues in direct alignment algorithms, offers theoretical underpinning rather than solutions, and retains reliance on imperfect human-preference models.The authors frame the study as motivating progress toward better alignment rather than resolving the identified problems.

B Experiment Details

Experiments largely followed the DPO setup on the curated OpenAI TL;DR dataset, evaluating DAA models across three Pythia model sizes after supervised fine-tuning. Training used consistent optimization settings across SFT and DAA runs, with evaluations performed on 256 held-out prompts.

  • Implementation: The implementation largely followed the DPO setup and reused its code without changing hyperparameters unless otherwise noted.
  • Dataset and evaluation: DAA experiments used the curated OpenAI TL;DR dataset with 92K preferred-dispreferred summary completions and evaluated loss, accuracy, KL, winrates, and length on 256 held-out prompts.Evaluations used temperature 1.0 and a maximum length of 512.
  • Models and training: Experiments covered 1B, 2.8B, and 6.9B Pythia models initialized from pretrained weights and supervised-fine-tuned on TL;DR before direct alignment.All SFT and DAA runs used batch size 128, RMSProp with learning rate 0.5 × 10^-6, 150-step linear warmup, and one training epoch.

C Appendix A: Complete Intra-Epoch Training Dynamics

Across 2.8B and 1B models, DPO, SLiC, and IPO show intra-epoch dynamics similar to the 6.9B models: KL divergence rises almost monotonically while performance tends to degrade after the first quarter epoch, especially under low KL budgets.

  • C Appendix A: Complete Intra-Epoch Training Dynamics: The appendix reports intra-epoch KL-divergence and winrate evolution results across all model sizes.
  • C Appendix A: Complete Intra-Epoch Training Dynamics: After the first quarter epoch, GPT4 winrate tends to degrade while KL divergence increases almost monotonically across DPO, SLiC, and IPO for 2.8B and 1B models, particularly under low KL budgets.These dynamics are similar to those observed for the 6.9B models.

D Overoptimization from the lens of Implicit Bootstrapping · E Understanding Behavior of DAAs on OOD sequences

The analysis explains DAA overoptimization as implicit OOD bootstrapping in token-level value estimates, despite offline training, and examines how probability mass shifts toward OOD sequences. A toy tree MDP is introduced to study this allocation behavior.

  • D Overoptimization from the lens of Implicit Bootstrapping: A DAA-trained policy admits a dense per-token reward representation satisfying a soft Bellman equation, with sequence reward obtained by summing token-level rewards.The dense formulation is more general than sparse terminal rewards because intermediate rewards can be set to zero.
  • D Overoptimization from the lens of Implicit Bootstrapping: DAA logits can be interpreted as soft Q-values, whose estimates include a log-probability term associated with OOD bootstrapping.This interpretation follows from substituting the implicit reward representation into the soft value formulation.
  • D Overoptimization from the lens of Implicit Bootstrapping: DAAs can exhibit classical offline-RL OOD bootstrapping because their token-level value estimates effectively evaluate unseen-token actions despite fully offline training.This connects DAA overoptimization to the classical RLHF issue of querying potentially OOD samples, even without explicit online reward-model queries.
  • D Overoptimization from the lens of Implicit Bootstrapping: Smaller beta values produce more optimistic estimates and consequently higher levels of OOD bootstrapping, potentially making the resulting LLM suboptimal despite fitting the implicit reward.The interpretation is presented as explaining counter-intuitive behavior associated with beta and training dynamics.
  • E Understanding Behavior of DAAs on OOD sequences: Because OOD response probability must increase when total probability mass is conserved, DPO may decrease likelihoods for both preferred and dis-preferred responses.This behavior is described as consistent with a maximum-entropy reinforcement-learning perspective.
  • E Understanding Behavior of DAAs on OOD sequences: The study uses a toy deterministic tree MDP, originating from one start state, to mimic the LLM setting and analyze how DAAs allocate probability mass to OOD sequences.The toy MDP is illustrated in Figure 6.

E.1 Designing a toy LLM MDP

The toy LLM MDP models token-level alignment as a tree with deterministic terminal transitions and deliberately limited demonstrations and preferences. Across DPO, IPO, and SLiC, alignment shifts substantial probability mass toward a few out-of-distribution trajectories, with DPO admitting multiple such solutions.

  • MDP design: The MDP is a tree rooted at one start state, with every leaf deterministically transitioning to an absorbing terminal state, mirroring token-level DPO settings.This construction represents shared-prompt preferred and dispreferred responses in the broader LLM-alignment context.
  • Experimental setup: The protocol uses three SFT demonstrations, one preference comparison, an RNN policy, and DPO, IPO, and SLiC losses under β values 0.01, 0.1, and 0.5.The deliberately low-data setup probes OOD probability mass and reflects preference datasets that are small relative to deployed LLMs.
  • Results: DPO, IPO, and SLiC allocate substantial probability mass to OOD trajectories, while in-distribution mass can diminish and concentrate on a few OOD paths.These patterns are illustrated across trajectory-probability plots for all three algorithms and tested β values.
  • Results: DPO’s unconstrained loss admits multiple solutions that distribute varying amounts of probability mass to OOD trajectories.The DPO plots are noisy, even when substantial mass remains on the selected preference trajectory.

F Overoptimization Trends in the Gemma2-2b Model and Anthropic-HH Dataset

Figure 19 examines overoptimization trends for DPO and SLiC on Gemma2-2b with the Anthropic-HH dataset, showing KL divergence versus GPT-4 win rate across β values. These results indicate that trends observed with Pythia models are not specific to one model or dataset.

  • F Overoptimization Trends in the Gemma2-2b Model and Anthropic-HH Dataset: Figure 19 shows the KL-divergence–GPT-4-win-rate trade-off for DPO and SLiC across different β values on Gemma2-2b and Anthropic-HH.The experiment extends overoptimization analysis beyond the Pythia models.
  • F Overoptimization Trends in the Gemma2-2b Model and Anthropic-HH Dataset: The observed overoptimization trends are not specific to a single model or dataset, as demonstrated with Gemma2-2b and Anthropic-HH.The passage frames DPO and SLiC results as sufficient to illustrate this broader pattern.

NeurIPS Paper Checklist

The paper reports that its claims, theoretical results, experimental details, reproducibility guidance, open-source materials, and compute requirements are documented, but it does not report statistical significance because configurations were not run with multiple seeds.

  • Limitations: The paper states that its limitations are discussed in a separate section at the beginning of the appendix.
  • Theory Assumptions and Proofs: The paper reports that its theoretical results are supported by proofs and empirical evidence.
  • Experimental Result Reproducibility: The paper answers Yes on reproducibility, providing datasets, code, hyperparameters, training and test details, and compute-resource information.It uses open-source models and datasets and refers to appendix sections for reproduction and experimental details.
  • Open access to data and code: The paper uses open-source models and datasets and directs readers to Section B for reproducing the results.
  • Experiment Statistical Significance: The paper does not report error bars or statistical significance because it avoids multiple seeds per configuration under limited computing and financial resources.Instead, the authors emphasize evaluation across multiple configurations.
Loading 2406.02900v2…