PaperlayerView exact revision on arXiv

Source-linked AI summary

Disentangling Length from Quality in Direct Preference Optimization

Ryan Park, Rafael Rafailov, Stefano Ermon, Chelsea Finn

arXiv:2403.19159v2cs.CLcs.LG

TL;DR

Preference-based alignment can reward verbosity, but whether this length exploitation occurs in DPO was previously unclear. The paper analyzes DPO’s implicit reward, derives explicit length regularization, and evaluates it on dialogue and summarization. The regularized method controls verbosity while maintaining performance and improves length-corrected win rates by up to 15-20%.

  • Problem

    Length exploitation is established in classical RLHF but under-explored in DPO, where the lack of an explicit reward model makes reward overoptimization harder to define.

  • Method

    The paper interprets DPO through its implicit reward and adds an explicit α|y| length penalty to derive a length-regularized DPO objective.

  • Results

    15-20% improvement in length-corrected win rates was achieved while maintaining GPT-4-evaluated performance without significant increases in verbosity.

  • Takeaways & Limitations

    Length-regularized DPO can disentangle answer quality from length across dialogue and summarization evaluations despite GPT-4’s length bias.

  • Takeaways & Limitations

    The study evaluates one model size and two smaller public human-feedback datasets, leaving scaling behavior and dependence on model capability and data quality unclear.

Abstract

from arXiv · show

Reinforcement Learning from Human Feedback (RLHF) has been a crucial component in the recent success of Large Language Models. However, RLHF is know to exploit biases in human preferences, such as verbosity. A well-formatted and eloquent answer is often more highly rated by users, even when it is less helpful and objective. A number of approaches have been developed to control those biases in the classical RLHF literature, but the problem remains relatively under-explored for Direct Alignment Algorithms such as Direct Preference Optimization (DPO). Unlike classical RLHF, DPO does not train a separate reward model or use reinforcement learning directly, so previous approaches developed to control verbosity cannot be directly applied to this setting. Our work makes several contributions. For the first time, we study the length problem in the DPO setting, showing significant exploitation in DPO and linking it to out-of-distribution bootstrapping. We then develop a principled but simple regularization strategy that prevents length exploitation, while still maintaining improvements in model quality. We demonstrate these effects across datasets on summarization and dialogue, where we achieve up to 20\% improvement in win rates when controlling for length, despite the GPT4 judge's well-known verbosity bias.

1 Introduction

RLHF and DPO improve LLM alignment but can exploit preference biases toward verbosity. This paper studies length exploitation in DPO, links it to out-of-distribution behavior, and proposes regularization to control verbosity while preserving quality.

  • DPO-generated answers can become twice as long as feedback-data responses and significantly out of distribution, whereas length-regularized DPO stays closer to SFT lengths.
  • RLHF can produce significantly longer answers without necessarily improving quality, partly because preference data favors longer responses.
  • GPT-4 and automated evaluations exhibit verbosity bias, making length-corrected performance important when comparing open-source and proprietary models.
  • DPO is an increasingly used alternative to RLHF that trains offline without a separate reinforcement-learning stage, improving simplicity, speed, and resource efficiency.
  • Length exploitation had been studied in classical RLHF but remained unexplored in DPO, where the absence of an explicit reward model makes overoptimization harder to define.
  • The paper shows that length exploitation is prevalent in DPO, connects it to out-of-distribution extrapolation, and develops regularization that controls verbosity with minimal performance impact.

2 Preliminaries

RLHF trains a reward model from human-ranked responses and then optimizes an LLM with reinforcement learning. DPO instead reparameterizes the optimal reward through the policy and trains directly with a classification objective.

  • Reinforcement Learning From Human Feedback: The standard RLHF pipeline has three stages: supervised fine-tuning, reward-model training from ranked answers, and reinforcement-learning optimization.
  • Reinforcement Learning From Human Feedback: Human preferences are represented as rankings of preferred and dispreferred answers, commonly modeled with a Bradley-Terry preference distribution driven by latent reward.
  • Reinforcement Learning From Human Feedback: The RLHF objective trades off maximizing learned reward against divergence from a reference distribution, typically using a hyperparameter β and an RL algorithm such as PPO.
  • Direct Preference Optimization: DPO analytically transforms the optimal reward-policy relationship and substitutes it into the reward objective, enabling direct training on offline preference data.
  • Direct Preference Optimization: DPO uses a standard classification loss to recover the optimal model without on-policy sampling or substantial hyperparameter tuning.
  • Direct Preference Optimization: DPO’s implicit reward is a valid reward-model parameterization, and the resulting policy is the closed-form optimum of the corresponding constrained RL problem.

3 Building in Explicit Regularization in DPO

The paper derives a length-regularized DPO objective by adding a token-length penalty to the implicit reward, yielding a modified preference-learning loss. This regularizer adjusts learning according to length differences and is intended to disentangle answer quality from verbosity.

  • Explicit length regularization: The method adds a general regularization term to the DPO derivation, focusing on a length penalty α|y| for answer token length.α is the hyper-parameter controlling the length term, and |y| denotes answer token length.
  • Regularized reward: The regularized reward subtracts the length penalty from the latent reward before deriving the equivalent DPO formulation.The derivation assumes r(x, y) remains the latent reward driving human preferences.
  • Regularized DPO objective: The resulting DPO objective is the standard objective with an additional regularization margin α|yw| − α|yl| in the binary-classification logit.The margin compares the preferred and dispreferred answers’ lengths within each feedback pair.
  • Interpretation: The regularizer is presented as the DPO implicit-reward counterpart to length-regularized classical RLHF and as a way to improve length-controlled performance.The paper frames the approach as explicitly linking regularized reward modeling with an equivalent regularized reinforcement-learning setup.
  • Optimization effect: The added term increases the likelihood of preferred answers while decreasing the likelihood of dispreferred answers, with the length difference modulating the update.Pairs with shorter selected answers receive up-weighted gradients, whereas pairs with longer selected answers receive down-weighted gradients.

4 Experiments

Experiments on HH and TL;DR show that standard DPO generates unusually long, out-of-distribution responses, while length regularization controls verbosity and can improve quality under length-matched evaluation. The results also connect length exploitation to out-of-distribution reward extrapolation and show that regularization supports continued training improvement.

  • 4.2 Length Exploitation in DPO and Effectiveness of Regularization: DPO-generated answers are significantly longer than preferred and rejected answers and are out-of-distribution in length relative to the offline preference data.Smaller β values produce longer responses; length regularization keeps outputs closer to the SFT model and avoids significantly out-of-distribution lengths.
  • 4.2 Length Exploitation in DPO and Effectiveness of Regularization: Length regularization efficiently reduces verbosity on both HH and TL;DR while preserving the evaluation setup’s comparison of response quality.The experiments use GPT-4 head-to-head evaluation against preferred answers, with 90% confidence intervals for win rates versus average length.
  • 4.3 Length Versus Quality Trade-Offs: Close to 20% improvement on HH and close to 15% improvement on TL;DR occurs when regularized models match the average lengths of β = 0.5 runs while achieving statistically significant higher win rates.The comparison uses β = 0.05 and β = 0.01 regularized runs against corresponding β = 0.5 lengths; GPT-4 is known to have a significant length bias.
  • 4.5 DPO and Early Convergence: The KL budget is only weakly correlated with win rates and length for TL;DR, while length-regularized models can achieve higher win rates with statistically significant KL differences.For HH and TL;DR, β = 0.05 and β = 0.01 regularized models match the average length of β = 0.5 runs but have higher KL divergences and win rates.
  • 4.5 DPO and Early Convergence: The length-regularized model achieves higher final win rates at less than 40% of the KL budget and almost half the response length, while standard DPO peaks early and then stops improving.Across two epochs on HH, regularized training improves steadily, whereas standard DPO’s continued length and KL increases do not yield further win-rate gains.

5 Related Work

Prior work identifies reward exploitation and verbosity bias in RLHF, but length exploitation had not been studied for DPO. This work positions itself as a DPO counterpart to approaches that disentangle length from quality.

  • Reward Exploitation in RLHF: RLHF reward exploitation occurs when expected reward improves while model quality later degrades.Increased verbosity has been explicitly linked to this phenomenon.
  • Mitigating Length Biases in RLHF: Prior RLHF methods address reward exploitation or length bias, but do not directly transfer to direct alignment algorithms such as DPO.The paper frames its approach as a DPO equivalent counterpart to length-bias methods.
  • DPO: This work presents the first study of length exploitation for Direct Alignment Algorithms such as DPO.The novelty claim specifically concerns studying the length exploitation problem in the DPO setting.

6 Conclusion

The paper studies length exploitation in DPO, links it to out-of-distribution bootstrapping, and introduces length regularization. Across two human-feedback datasets, the approach preserves GPT-4-evaluated performance while improving length-corrected win rates by up to 15–20%.

  • 6 Conclusion: DPO exhibits significant length hacking across a range of hyperparameters on two standard human feedback datasets.The paper extends analysis of length exploitation from classical RLHF to DPO.
  • 6 Conclusion: The paper links DPO length exploitation specifically to out-of-distribution bootstrapping.
  • 6 Conclusion: 15–20% improvement in length-corrected win rates is achieved while maintaining GPT-4-evaluated model performance without significant verbosity increases.
  • 6 Conclusion: The authors hypothesize that length exploitation may affect open-source models trained on strongly length-biased public feedback datasets.They suggest this could relate to observations in Fig. 1 and motivate length-corrected comparisons with proprietary models.

7 Limitations

The study is limited to length exploitation in DPO and evaluates one model size on two smaller-scale public human-feedback datasets. Its regularizer and scaling behavior may not generalize beyond these settings.

  • 7 Limitations: The regularization objective requires an explicit penalty function such as length and may not address exploitation along other axes.
  • 7 Limitations: The study evaluates only the DPO objective, which may behave differently from other direct alignment algorithms.
  • 7 Limitations: Experiments use one model size and two smaller-scale public human-feedback datasets, leaving scaling behavior and dependence on model and data properties unclear.

Ethics Statement

The work focuses on increasing verbosity bias during DPO training using publicly available data and pretrained models. It releases no new models for public use and reports no ethical concerns.

  • Ethics Statement: The work focuses on alleviating empirical extrapolation issues during DPO training, specifically increasing verbosity bias.
  • Ethics Statement: Experiments use publicly available data and pretrained models.
  • Ethics Statement: The authors do not release any new models for public use and state that there should be no ethical concerns.

A Phi-2 UltraFeedback Experiments

Experiments with Phi-2 on UltraFeedback further validate the length-regularization approach, while reward–length correlations across datasets and settings show consistent out-of-distribution extrapolation.

  • Phi-2 experiments use the 2.7B Microsoft model on the UltraFeedback binarized dataset, containing 64K prompts with LLM-generated and GPT-ranked completions.Chosen responses are the highest-scoring completions; rejected responses are sampled randomly from the other three completions.
  • The additional Phi-2 evaluation provides a small-model validation of the proposed length-regularization strategy on preference data ranked by GPT.The passage describes this as further validation rather than a broad benchmark.
  • Across TL;DR and HH experiments, varying α and β produces a similar pattern of out-of-distribution extrapolation along the length axis.The additional experiments extend the reward–length correlation analysis shown in Fig. 6.

C Experimental Details

The experiments use standard DPO settings on TL;DR and HH, evaluate responses with GPT-4 prompts, and visualize KL divergence and implicit reward across α and β configurations.

  • Training follows the original DPO codebase with one epoch, batch size 128, 16 gradient-accumulation steps, RMSProp, and learning rate 0.5 × 10^-6.The learning rate uses linear warm-up for 150 steps.
  • Main experiments use 92K preferred/dispreferred pairs from OpenAI TL;DR, evaluate 256 held-out prompts, and initialize models from Pythia 2.8B after one SFT epoch.Evaluations use temperature 1.0 and maximum length 512.
  • GPT-4 evaluates TL;DR summaries by asking which better captures the most important points and HH responses by asking which is more helpful.Both prompts require a one-sentence comparison followed by a binary choice.
  • Figure 7 plots HH KL divergence and DPO implicit reward for preferred, dispreferred, and model-generated responses across α values in rows and β values in columns.Preferred responses are blue, dispreferred responses red, and model-generated responses green.
  • Figure 8 provides the corresponding TL;DR KL-divergence and implicit-reward analysis under α values 0.0, 0.02, and 0.05 and β values 0.05 and 0.5.

D Sample Responses Across Models

The paper compares responses from RDPO, standard DPO, and SFT on HH and TL;DR prompts, using examples selected around GPT-4 evaluation outcomes.

  • Examples compare length-regularized DPO, standard DPO, and the initial SFT policy on responses to prompts from the HH and TL;DR evaluation sets.All samples use the standard generation procedure and the same set of 256 prompts.
  • The selected cases highlight prompts where RDPO beats the dataset response according to GPT-4, while both DPO and SFT do not.
  • The HH examples include advice about bathing a child, renting a U-Haul without a license, relationship messaging, and marital neglect.
Loading 2403.19159v2…