PaperlayerView exact revision on arXiv

Source-linked AI summary

Earlier Tokens Contribute More: Learning Direct Preference Optimization From Temporal Decay Perspective

Ruichen Shao, Bei Li, Gangao Liu, Yang Chen, Xiang Zhou, Jingang Wang, Xunliang Cai, Peng Li

arXiv:2502.14340v1cs.CL

TL;DR

DPO can favor longer responses when chosen and rejected examples differ in length, while existing approaches still weight rewards uniformly across sequence positions. D2PO introduces gamma-controlled temporal decay to prioritize earlier tokens. It reports improvements over DPO across benchmarks and model architectures, including gains on AlpacaEval 2 and Arena-Hard.

  • Problem

    DPO’s length bias can make generations longer than the reference model when chosen responses are often longer than rejected responses, and existing methods still treat token contributions uniformly.

    Source: §1 · p. 1, §3.2 · p. 4

  • Method

    D2PO adds a gamma-controlled temporal decay factor to DPO, reducing later-token contributions so earlier tokens receive greater weight.

    Source: §1 · p. 2, §3.2 · p. 5

  • Results

    D2PO consistently outperforms vanilla DPO across benchmarks and model architectures, with reported on-policy gains of 5.9–8.8 win-rate points on AlpacaEval2 and 3.3–9.7 points on Arena-Hard.

    Source: §1 · p. 2, §7 · p. 10

  • Takeaways & Limitations

    D2PO retains DPO’s computational efficiency while prioritizing earlier tokens, and the authors report improved mathematical and reasoning benchmark performance without compromising general capabilities.

    Source: §1 · p. 2

    The instruction-tuning status of the evaluated models is not fully disclosed; the paper refers to them as SFT models for simplicity.

    Source: §5.1 · p. 7

Abstract

from arXiv · show

Direct Preference Optimization (DPO) has gained attention as an efficient alternative to reinforcement learning from human feedback (RLHF) for aligning large language models (LLMs) with human preferences. Despite its advantages, DPO suffers from a length bias, generating responses longer than those from the reference model. Existing solutions like SimPO and SamPO address this issue but uniformly treat the contribution of rewards across sequences, overlooking temporal dynamics. To this end, we propose an enhanced preference optimization method that incorporates a temporal decay factor controlled by a gamma parameter. This dynamic weighting mechanism adjusts the influence of each reward based on its position in the sequence, prioritizing earlier tokens that are more critical for alignment. By adaptively focusing on more relevant feedback, our approach mitigates overfitting to less pertinent data and remains responsive to evolving human preferences. Experimental results on several benchmarks show that our approach consistently outperforms vanilla DPO by 5.9-8.8 points on AlpacaEval 2 and 3.3-9.7 points on Arena-Hard across different model architectures and sizes. Furthermore, additional experiments on mathematical and reasoning benchmarks (MMLU, GSM8K, and MATH) confirm that our method enhances performance without compromising general capabilities. Our codebase would be available at \url{https://github.com/LotuSrc/D2PO}.

1 INTRODUCTION

DPO offers a single-stage alternative to RLHF, but imbalanced chosen and rejected response lengths can bias it toward longer generations. D2PO applies temporal decay to prioritize earlier tokens, motivated by evidence that DPO affects them more, and reports gains over DPO across benchmarks.

1 INTRODUCTION

  • DPO’s length bias can produce responses longer than the reference model when chosen responses are often longer than rejected ones.SimPO removes the reference model and uses average sequence log probability as an implicit reward; SamPO limits reward computation to the shorter chosen/rejected sequence range.Source: §1 · pp. 1–2
  • Across Llama3, Gemma2, and Mistral-NeMo, DPO variants show larger KL divergence at earlier positions, which the authors interpret as greater effects on earlier tokens.The authors relate this observation to prior findings that earlier tokens are more critical for alignment.Source: §1 · p. 2, Figure 1 · p. 2
  • D2PO adds a gamma-controlled temporal decay factor to preference optimization, giving earlier feedback more weight than later tokens.The introduction describes this as a dynamic weighting mechanism and provides a token-level Markov decision process analysis.Source: §1 · p. 2
  • D2PO reports on-policy gains of 5.9–8.8 win-rate points on AlpacaEval2 and 3.3–9.7 points on Arena-Hard over DPO.The introduction also reports improvements in off-policy setups and on mathematical and reasoning benchmarks without compromising general capabilities.Source: §1 · p. 2

2 RELATED WORK

Classical RLHF trains a reward model from human preference comparisons, then optimizes a policy with reinforcement learning. Direct alignment algorithms simplify this pipeline, with methods distinguished in part by whether they use a reference model.

2.1 REINFORCEMENT LEARNING FROM HUMAN FEEDBACK(RLHF)

  • Classical RLHF has two stages: reward modeling from human-labeled preference pairs, followed by policy optimization that maximizes reward while constraining divergence from a reference policy.The reward-modeling phase uses a Bradley–Terry preference likelihood.Source: §2.1 · p. 3
  • The two-stage RLHF framework is associated with reward over-optimization, training instability, and efficiency challenges.Source: §2.2 · p. 3

2.2 DIRECT ALIGNMENT ALGORITHMS (DAAS)

  • Direct alignment algorithms include reference-based methods such as DPO, KTO, IPO, and R-DPO, which use a pre-existing model as a comparison point.The cited methods differ in their preference assumptions and strategies for addressing length bias.Source: §2.2 · p. 3
  • Reference-free methods forgo a pre-existing reference model; examples include CPO, ORPO, and SimPO, which use different preference or likelihood objectives.SimPO uses average log probability as an implicit reward and adds a target reward margin.Source: §2.2 · p. 3
  • Figure 2 contrasts uniform DPO coefficients, length-normalized SimPO coefficients, SamPO coefficients based on the shorter response, and D2PO’s position-dependent decay.Its D2PO illustration uses gamma = 0.9.Source: Figure 2 · p. 4

3 METHODOLOGY

The methodology starts from DPO’s implicit-reward preference objective, then weights token contributions by position; it also motivates a reference-free loss for on-policy data.

3.1 DIRECT PREFERENCE OPTIMIZATION (DPO).

  • DPO reparameterizes reward through the policy model and optimizes preference data with a Bradley–Terry ranking objective.The reward uses the log ratio of response likelihoods under the current policy and an SFT model.Source: §3.1 · p. 4

3.2 TEMPORAL DECAY MATTERS IN PREFERENCE LEARNING.

  • D2PO weights token contributions in the DPO loss by exponential γ^t (0 < γ < 1), smoothly reducing later-token influence and prioritizing earlier tokens.This addresses the uniform token treatment in existing preference methods.Source: §3.2 · pp. 4–5

3.3 REFERENCE-FREE IS CONSISTENT WITH ON-POLICY SETUPS.

  • On-policy UltraFeedback margins have lower variance and a mean closer to zero than off-policy margins, motivating removal of DPO’s reference constraint and a reference-free loss.The comparison uses original responses for off-policy data and responses regenerated by the same model for on-policy data.Source: §3.3 · pp. 5–6

4 THEORETICAL ANALYSIS

The theoretical analysis decomposes policy suboptimality to distinguish discount-factor effects from policy differences. Its upper-bound analysis identifies a trade-off in γ, with an interior value balancing opposing terms.

4.1 SUBOPTIMALITY DECOMPOSITION

  • The decomposition separates suboptimality differences due to discount factors from those due to policy discrepancies.Suboptimality is defined relative to the optimal policy under an evaluation discount factor γe = 1.0.Source: §4.1 · p. 6

4.2 SUBOPTIMALITY ANALYSIS

  • The upper bound makes suboptimality depend on discount factor γ and the mismatch between π and π*, assuming bounded rewards, finite horizon, and evaluation at γe = 1.0.The proof is included in Appendix F.Source: §4.2 · p. 6
  • The two terms vary monotonically in opposing directions, so an interior γ* in (0, 1) balances them to minimize overall suboptimality.Source: §4.2 · p. 6

5 EXPERIMENTS

The experiments compare D2PO with DPO and other preference-optimization baselines across off-policy and on-policy settings. D2PO improves win rates in both settings, with on-policy results also reporting shorter responses.

5.1 EXPERIMENTAL SETUPS

  • Experiments use Llama3-8B, Gemma2-9B, and Mistral-12B instruction-tuned models, with UltraFeedback off-policy pairs and model-generated, reward-ranked on-policy pairs.For on-policy data, five responses per prompt are sampled at temperature 0.8; the highest- and lowest-scoring responses form the chosen and rejected pair.Source: §5.1 · p. 7

5.2 EXPERIMENTAL RESULTS

  • The comparison includes DPO and its variants alongside reference-free baselines SimPO and ORPO, with code and experimental settings kept consistent across evaluated methods.The experiments include both off-policy and on-policy data; the method is also evaluated in a reference-free on-policy setting.Source: §5.2 · p. 8
  • D2PO exceeds DPO on Llama3 by 6.1% and 2.9% in standard and length-controlled evaluations, and on Mistral-NeMo by 6.9% and 5.1%, respectively.The authors report that reference-free methods can lose performance on off-policy data, particularly SimPO.Source: §5.2 · p. 8
  • On-policy, D2PO consistently improves results across setups and produces shorter responses; without a reference model, it exceeds SimPO by 2.4–7.4 in LC win rate and 0.7–4.2 in win rate on AlpacaEval 2 and Arena-Hard.The authors note that reward-model selection favors longer responses and that standard evaluations favor longer outputs.Source: §5.2 · p. 8

6 ANALYSES

The analyses examine how temporal-decay choices affect benchmark performance, response length, and general task performance. They report that decay below 1.0 generally performs well, while alternative decay strategies and weighting later tokens can be less effective.

Plays an Important Role

  • Nearly all tested γ values below 1.0 outperform DPO, and γ = 0.98 gives the highest performance across three benchmarks for the three tested open-source models.The authors describe performance as robust to the choice of γ, reducing the need for extensive hyperparameter tuning.Source: §6 · p. 8

γ Larger than 1 is Harmful

  • When γ exceeds 1.0, later-token rewards receive larger coefficients, and performance falls behind standard DPO on AlpacaEval 2 and Arena-Hard.The authors interpret this result as evidence that overemphasizing later tokens can degrade preference-optimization performance.Source: §6 · pp. 8–9

Evaluations on OpenLLM Benchmark

  • D2PO performs better than DPO and SimPO on MMLU and MATH in the Llama3-8B configuration, and its MATH score exceeds the Instruct model by 0.55 points.On GSM8K, D2PO shows less performance degradation than SimPO despite both controlling output length.Source: §6 · p. 9
  • Gemma2-9B shows a similar pattern, with D2PO performing significantly better on MATH; the authors report effectiveness across specialized datasets and task types.The evaluation covers reasoning and mathematical test sets as well as other OpenLLM Benchmark tasks.Source: §6 · p. 9, Table 3 · p. 9

Comparisons of Various Decay Strategies

  • Among the tested decay strategies, 1-0 and linear decay perform worse than temporal decay and even vanilla DPO, while power-law decay shows promise but does not control response length as effectively as exponential decay.Exponential decay is the default strategy, following the authors’ stated connection to the classic Markov Decision Process.Source: §6 · p. 9, Table 4 · p. 10

Lengthy Debias

  • DPO loss is larger for verbosity-biased pairs than brevity-biased pairs, whereas D2PO reduces this loss imbalance and controls output length.The analysis visualizes losses for 1,000 samples grouped by the length gap between chosen and rejected responses.Source: §6 · p. 10, Figure 6 · p. 10

Human Evaluations

  • Human evaluators report D2PO win rates of 67% on Arena-Hard and 69% on AlpacaEval 2 against DPO for Gemma2-9B.Four evaluators each assessed 50 samples per benchmark, judging accuracy, completeness, and relevance.Source: §6 · p. 10

7 CONCLUSIONS

The conclusion presents temporal decay as a way to prioritize earlier tokens in preference optimization. The authors report consistent improvements over vanilla DPO and extension to reference-free, on-policy training.

7 CONCLUSIONS

  • Table 5 presents human-evaluation results on two benchmarks.Source: Table 5 · p. 10
  • D2PO introduces γ-controlled temporal weighting to prioritize earlier tokens, and the authors report consistent gains over vanilla DPO across benchmarks and model architectures.They also report extending the method to a reference-free, on-policy setting, where it outperforms existing approaches.Source: §7 · p. 10

A EXPERIMENTAL SETUPS

The appendix describes benchmark evaluation, comparison baselines, and selected training details. AlpacaEval 2 uses GPT-4-Turbo-FN as annotator, while the reported settings include fixed β and shared training resources.

A EXPERIMENTAL SETUPS

  • Evaluation covers MT-Bench, AlpacaEval 2, and Arena-Hard, using GPT-4-Turbo as the MT-Bench judge and reporting raw and length-controlled win rates for AlpacaEval 2.Arena-Hard reports win rate against a baseline; the benchmarks cover conversational queries, with Arena-Hard including 500 technical problem-solving queries.Source: A EXPERIMENTAL SETUPS · p. 15
  • The baselines include IPO, KTO, ORPO, SimPO, and SamPO; SimPO and SamPO address verbosity through length normalization, and each baseline’s hyperparameters are tuned.Source: A EXPERIMENTAL SETUPS · p. 15
  • AlpacaEval 2 uses the GPT-4-Turbo-FN annotator, which the authors report has 68.1% agreement with human annotation data.The appendix contrasts this with the default weighted GPT-4-Turbo annotator’s 65.7% agreement.Source: A EXPERIMENTAL SETUPS · p. 15

B EXPERIMENTAL DETAILS

  • The decay-coefficient comparison fixes β at 0.1; runs use 32 A100s and a global batch size of 128 with four gradient-accumulation steps.The hyperparameter search ranges are listed in Table 6.Source: B EXPERIMENTAL DETAILS · p. 15

C MORE ANALYSES

The analyses test D2PO across decay settings, judge models, style-controlled evaluation, and a stronger instruct model. They report gains across these settings, while noting the evaluation configurations and benchmark-specific scope.

When to Decay

  • Both decay-from-prompt and decay-from-generated-token settings outperform DPO, but the default prompt-based setting performs much better.The authors state that the first generated token receives a different initial scaling coefficient under the two settings.Source: C MORE ANALYSES · p. 15, Figure 7 · p. 16

Effect of Different Judge Models

  • D2PO delivers consistent performance gains with both GPT-4-Turbo-0409 and GPT-4-preview-1106 as judge models.Table 7 reports results with GPT-4-preview-1106; the main evaluation used GPT-4-Turbo-0409.Source: C MORE ANALYSES · p. 15, Table 7 · p. 16

Full evaluation results

Arena-Hard Style Control Evaluation

  • D2PO consistently achieves superior performance on Arena-Hard style control evaluation with Gemma2-9B.The evaluation focuses on style-control capabilities.Source: C MORE ANALYSES · p. 17, Table 8 · p. 17

Stronger Instruct Model Evaluation

  • D2PO maintains an advantage over DPO and SimPO on three benchmarks with the stronger Gemma2-Instruct (27B) model.The comparison uses on-policy setups and is limited to those two baselines because of computing resources.Source: C MORE ANALYSES · p. 19, Table 11 · p. 19

D CASE ANALYSIS

The case analyses contrast DPO and D2PO outputs on a travel-time calculation and an ocean question. They illustrate differences in calculation accuracy and answer explanation in these examples.

Travel-time example

  • In the travel-time example, DPO omits a stopover from its final calculation after mistakenly treating it as already included, while D2PO correctly sums the listed times.The comparison is shown in Figure 8.Source: D CASE ANALYSIS · p. 19, Figure 8 · p. 19

Largest-ocean example

  • For the largest-ocean question, DPO answers without explanation, whereas D2PO gives a structured answer with detailed explanation.The comparison is shown in Figure 9.Source: D CASE ANALYSIS · p. 19, Figure 9 · p. 20

E DERIVATION OF D2PO

The derivation extends a KL-constrained Q–V relation with temporal decay and uses it to form a D2PO objective. The theoretical analysis derives bounds under bounded rewards and finite-horizon assumptions.

E DERIVATION OF D2PO

  • D2PO incorporates temporal decay into the Q–V relation under a KL-divergence constraint, then uses the resulting reward formulation in a Bradley-Terry ranking loss.The method empirically sets gamma < 1 to emphasize short-term over long-term return.Source: E DERIVATION OF D2PO · p. 21

F THEORETICAL ANALYSIS

  • The theoretical analysis frames suboptimality as the performance difference between the optimal policy and a given policy under specified discount factors.The derivation introduces a suboptimality upper bound for a finite horizon.Source: F THEORETICAL ANALYSIS · p. 22

F.1 THE UPPER BOUND OF ∆1 AND ∆3

  • The ∆1 and ∆3 bounds are analyzed together because both compare expected returns for the same policy under different discount factors.The derivation assumes bounded rewards.Source: §F.1 · p. 22

F.2 THE UPPER BOUND OF ∆2

  • The ∆2 bound uses the finite-horizon Performance Difference Lemma and assumes bounded rewards.Source: §F.2 · p. 22

F.3 SUBOPTIMAL ANALYSIS

The analysis identifies an interior gamma trade-off and distinguishes D2PO from related work in task setting and motivation. It also emphasizes flexible decay strategies and tests on standard RLHF benchmarks.

F.3 SUBOPTIMAL ANALYSIS

  • An optimal gamma* lies within (0, 1), balancing two terms that vary monotonically in opposing directions.Source: §F.3 · p. 23

G COMPARISON WITH STRONGLY RELATED WORK

Related papers

Loading 2502.14340v1…