Length Desensitization in Direct Preference Optimization

1 INTRODUCTION

• The paper reports that earlier approaches either failed to achieve significant length control or compromised performance, while prior explanations emphasized length bias in training data.The paper frames DPO’s own length sensitivity as a theoretical issue that remains inadequately explained in prior work.Source: §1 · p. 2
• DPO’s likelihood-based objective favors longer chosen responses and can skew optimization toward length while overlooking other human-like preferences.The paper attributes this to likelihood decreasing rapidly with sequence length and to differences in chosen and rejected response lengths.Source: §1 · p. 2
• LD-DPO decomposes the longer response’s likelihood into a shared-length portion and an excessive portion, separating verbosity preference from other preferences.The adjustment aims to reduce length’s influence on DPO’s optimization direction.Source: §1 · p. 2
• Across two settings of Llama2-13B, Llama3-8B, and Qwen2-7B, LD-DPO outperforms DPO and other baselines, reduces response length 10–40%, and improves reasoning performance.Evaluations include MT-Bench and AlpacaEval 2.Source: §1 · pp. 2–3
• The authors define DPO length sensitivity and provide theoretical validation for the phenomenon.They also propose decoupling length preference from reward as the basis for LD-DPO.Source: §1 · p. 3

2.1 REINFORCEMENT LEARNING FROM HUMAN FEEDBACK

• In RLHF, SFT produces a conversational model, human annotators label generated response pairs, and a Bradley-Terry reward model predicts those preferences.The labeled preferred and dispreferred responses are denoted yw and yl.Source: §2.1 · p. 3
• The RL stage uses the reward function to provide feedback, with β controlling deviation from the reference model and PPO given as an example optimizer.The language model is initialized from the SFT model.Source: §2.1 · p. 3

2.2 DIRECT PREFERENCE OPTIMIZATION

• DPO is an offline preference-optimization method that reparameterizes the reward with a closed-form expression, yielding a pairwise objective without additional reward-model training.The objective incorporates the reparameterized reward into the Bradley-Terry ranking objective.Source: §2.2 · p. 3

3.1 LENGTH SENSITIVITY OF DPO

• DPO’s inverse-likelihood gradients and shrinking sequence likelihood make it sensitive to length differences, potentially prioritizing length over other human-like preferences.When the preferred response is longer, its likelihood may be lower; when the dispreferred response is longer, avoiding it does not necessarily favor the shorter answer.Source: §3.1 · pp. 4–5

3.2 DERIVATION OF LD-DPO

• LD-DPO rewrites response likelihood around the pair’s public length, identifying the excess-length term as a source of reward reduction and optimization bias.The public length is the shorter of the two response lengths.Source: §3.2 · p. 5
• LD-DPO uses α to remove some verbosity preference from the over-length portion while retaining other preferences, including some length preference.At α = 1 it matches vanilla DPO; at α = 0 only the public-length portion is considered.Source: §3.2 · p. 5
• For any α < 1, LD-DPO smooths the likelihood ratio compared with DPO, with a stronger effect at smaller α and less influence from length disparity.Source: §3.2 · p. 6

Models and training settings

• Experiments use Llama2-13B, Llama3-8B, and Qwen2-7B under Base and Instruct/Chat setups.Base models are trained on UltraChat-200k for SFT, while Instruct setups start from corresponding instruct models.Source: §4 · p. 6
• Preference optimization uses 60,000 UltraFeedback preference pairs designed for conversational preferences and helpfulness.Source: §4 · p. 6

Evaluation benchmarks

• Evaluation uses MT-Bench, AlpacaEval 2, and Arena-Hard, reporting average MT-Bench score, length-controlled AlpacaEval 2 win rate, and response length.Arena-Hard analysis is presented in Appendix B.1.Source: §4 · p. 6

Baselines and training hyperparameters

• Baselines are DPO, R-DPO, SamPO, WPO, and SimPO; LD-DPO’s α is explored from 0.1 to 1.0.The general setup uses β = 0.1 for DPO, R-DPO, SamPO, WPO, and LD-DPO; SimPO uses β = 2.0 and γ = 1.0, and R-DPO uses α = 0.05.Source: §4 · p. 6

5.1 MAIN RESULTS

• LD-DPO significantly improves MT-Bench and AlpacaEval 2 over all baselines, with average response length 7.8%–37.9% lower than DPO.Table 2 covers six model settings; its LC-winrate metric is designed to mitigate judge length preference.Source: Table 2 · p. 7, §5.1 · p. 7
• DPO increases response verbosity by 10%–40% across Base and Instruct settings, while LD-DPO significantly alleviates it.The paper notes that verbosity can potentially impair model performance.Source: §5.1 · pp. 7–8

5.2 LENGTH SENSITIVITY ANALYSIS OF VARIED MODEL

• Optimal α varies by model: 0.3 for Llama2-13B, 0.5 for Llama3-8B, and 0.6 for Qwen2-7B in Instruct; Base optima are about 0.1–0.2 lower.The paper defines γ = 1 − α_s; γ is 0.7 for Llama2-13B-Chat and 0.5 for Llama3-8B-Instruct, with the former described as more length-sensitive.Source: §5.2 · pp. 8–9
• Under DPO, predicted probability differences vary with data length; when chosen responses are longer, the difference falls below zero and continues decreasing.The authors infer that this makes optimization toward the chosen response easier, leading to verbose outputs; the pattern reverses when rejected responses are longer.Source: §5.2 · p. 8
• With DPO-Pub, predicted probability differences are positive for a larger share of data; average differences shift by 19.66 and 14.01 for Llama2-13B-Chat, and 18.37 and 13.68 for Llama3-8B-Instruct.The authors interpret the shifts as showing that Llama2-13B-Chat is more affected by data length.Source: §5.2 · pp. 8–9

5.3 ABLATION STUDY

• Applying length decoupling to only chosen or only rejected responses performs worse than LD-DPO on benchmarks and length control, indicating both need decoupling.The authors explain that chosen-side decoupling can alleviate redundant output when chosen is longer, while rejected-side decoupling may redirect optimization toward a shorter chosen response when rejected is longer.Source: §5.3 · p. 9

5.4 HYPERPARAMETER ANALYSIS

• As α decreases from 1, length sensitivity and average output length decline; the response-length decrease is pronounced from α = 1 to 0.5 and less pronounced from 0.5 to 0.The authors suggest verbosity-preference decoupling is effectively complete at α = 0.5.Source: §5.4 · p. 10
• Performance on AlpacaEval 2 and MT-Bench first rises and then falls as α decreases, peaking at α = 0.5.The authors attribute the decline at smaller α to excessive decoupling losing human-like preferences, while larger α leaves DPO length sensitivity in place.Source: §5.4 · p. 10

6.1 OFFLINE PREFERENCE OPTIMIZATION

• Offline preference optimization includes ORPO, KTO, NCA, IPO, and WPO, alongside work on reference models, reward fitting, and update weights.DPO avoids online reward models but requires preference datasets to be constructed in advance.Source: §6.1 · p. 10

6.2 LENGTH CONTROL STRATEGY

• Length-control approaches include R-DPO’s rule term, SimPO’s length-normalized rewards, and SamPO’s down-sampling of KL divergence to mitigate length bias.The cited work attributes lengthy DPO outputs to length-dependent over- or underestimation of implicit rewards.Source: §6.2 · p. 10

7 CONCLUSION

• The paper provides a theoretical proof that DPO’s optimization process is length-sensitive.It proposes LD-DPO, reparameterizing likelihood to decouple verbosity preference while preserving human-like preferences.Source: §7 · p. 10
• LD-DPO consistently outperforms existing algorithms across training settings while reducing output length by 10%–40%, especially in reasoning ability.The conclusion says the results suggest previous optimization algorithms overemphasized length at the expense of quality.Source: §7 · pp. 10–11

A.1 DERIVATION OF OPTIMIZATION DIRECTION IN DPO

• DPO’s preference-direction gradient relative to the non-preference direction is inversely proportional to prediction probability.The derivation treats chosen and rejected response likelihoods and reference likelihoods as variables, with the reference likelihoods held constant.Source: §A.1 · pp. 14–15
• The appendix states that more extensive training data causes faster parameter updates, exacerbating DPO’s length sensitivity.The analysis examines partial derivatives of the gradient functions with respect to chosen and rejected likelihoods.Source: §A.1 · pp. 15–16

B.1 ARENA-HARD RESULTS

• On Arena-Hard, LD-DPO surpasses five offline preference-optimization strategies overall; Qwen2-Instruct reaches a 31.2% win rate, 7.8% above DPO, with responses 10% shorter.For Llama2-13B-Chat and Llama3-8B-Instruct, LD-DPO is suboptimal but produces responses over 10% shorter than the SOTA algorithm; the judge exhibits a clear preference for longer responses.Source: Table 4 · p. 16, §B.1 · p. 16

B.2 RESULTS ON DOWNSTREAM TASKS

• LD-DPO achieves the highest average score across benchmarks in every setting, outperforming SFT, DPO, and other baselines.Source: §B.2 · p. 17
• Preference optimization may not significantly improve every downstream task, and its methods perform comparably to SFT on MMLU and CommonsenseQA.The reported slight fluctuations on MMLU and CommonsenseQA are described as showing that knowledge is maintained during preference optimization.Source: §B.2 · pp. 17–18
• LD-DPO significantly improves HellaSwag and ProofWriter performance over SFT, DPO, and other baselines.Source: §B.2 · p. 18

B.3 MT-BENCH RESULT

• On MT-Bench, LD-DPO leads the baselines on average and significantly outperforms them in reasoning despite slight fluctuations across other dimensions.MT-Bench reports average scores across two Q&A rounds for 80 questions spanning eight dimensions.Source: §B.3 · pp. 18–19, Table 6 · p. 18

• ProofWriter tests problems from direct reasoning to tasks requiring more than five steps, under both open-world and closed-world assumptions.It divides these problems into 14 data subsets.Source: C IMPROVEMENT OF REASONING CAPABILITY · p. 19
• For Llama3-8B-Instruct, ProofWriter’s average score rises from 55.19 to 58.72 with LD-DPO, which improves across all 14 subsets.The paper reports that this outperforms five classes of preference optimization algorithms, including DPO.Source: C IMPROVEMENT OF REASONING CAPABILITY · p. 19
• The authors connect concise chain-of-thought style with improved reasoning, arguing that overly lengthy responses can include incorrect derivations or meaningless descriptions that interfere with reaching a conclusion.Source: C IMPROVEMENT OF REASONING CAPABILITY · p. 19

• Figure 7 presents performance across 14 ProofWriter subsets for Llama3-8B-Instruct.Source: Figure 7 · p. 19
• In two AlpacaEval 2 examples, LD-DPO produces comparably accurate outputs with fewer tokens than vanilla DPO.The paper notes that longer outputs can contain redundant or unimportant information.Source: D CASE STUDIES · p. 19, Figure 8 · p. 20
• In two ProofWriter examples, LD-DPO produces more direct, clearer chain-of-thought, while DPO’s verbose chains lead to incorrect answers.Source: D CASE STUDIES · p. 19, Figure 9 · p. 21