PaperlayerView exact revision on arXiv

Source-linked AI summary

Masking Frequent Tokens Sharpens Direct Preference Optimization

Harshvardhan Saini, Samyak Jha, Yiming Tang, Dianbo Liu

arXiv:2609.32445v1cs.CLcs.LG

TL;DR

DPO’s sequence-level preference scores can include frequent token types that have weaker preference-side distinction, raising the question of which token contributions should shape preference updates. The paper proposes a fixed frequency mask that suppresses those tokens’ direct score contributions while retaining context and the full softmax. Evaluations report consistent alignment improvements over standard DPO across two models and multiple benchmarks.

  • Problem

    The paper investigates whether frequent token types with weaker preference-side distinction should directly contribute to DPO preference scores.

    Source: §1 · pp. 1–2

  • Method

    ADPO uses a fixed, label-agnostic frequency mask to exclude high-frequency types from preference scores while retaining full context and the softmax.

    Source: §1 · p. 2

  • Results

    ADPO consistently improves alignment outcomes over standard DPO across Qwen2.5-7B-Instruct and Llama-3-8B-Instruct on multiple rigorous benchmarks.

    Source: §6 · p. 10

  • Takeaways & Limitations

    Selective masking of frequent token types is presented as a zero-overhead mechanism for preference optimization without discarding context, modifying pairs, or adding learned parameters.

    Source: §1 · p. 2, §6 · p. 10

    Absolute performance of edited response fragments is difficult to interpret because editing can remove length, context, and coherence.

    Source: §3.4 · p. 6

Abstract

from arXiv · show

Direct Preference Optimization (DPO) aligns language models by optimizing over sequence-level sums of token-wise implicit reward differences. However, we identify a pervasive pathology in this formulation: a disproportionately small subset of high-frequency token types dominates cumulative sequence scores while appearing symmetrically across both preferred and dispreferred responses. Specifically, under canonical Qwen tokenization on Anthropic HH-RLHF, merely 69 token types account for $55.1\%$ of all response tokens and $85.9\%$ of within-pair shared token mass, exhibiting substantially lower preference-side specificity than the remaining vocabulary. This symmetric ubiquity induces gradient entanglement and dilutes the discriminative preference signal propagated through the objective. To resolve this issue, we introduce \emph{Anisotropic DPO} (\textsf{ADPO}) and its canonical realization, \emph{Frequency-Hard DPO}. Using a fixed, label-agnostic vocabulary mask, our method zeroes the implicit reward contribution of high-frequency response tokens while assigning unit weight to informative positions, thereby suppressing gradient interference without modifying preference pairs, discarding context, or introducing learned parameters. Here, \emph{anisotropy} designates non-uniform token-level objective weighting rather than representational geometry. Extensive empirical evaluations on AlpacaEval, MT-Bench, and Arena-Hard demonstrate that Frequency-Hard DPO consistently outperforms standard DPO across Qwen-2.5-7B-Instruct and Llama-3-8B-Instruct, establishing that selectively masking shared high-frequency tokens offers an effective, zero-overhead mechanism for robust preference alignment.

1 INTRODUCTION

The introduction asks which token-level contributions should directly determine DPO preference updates, given gradient entanglement and weaker preference-side distinction among frequent types. It proposes ADPO, which uses a fixed frequency mask to suppress those score contributions while preserving context and the full softmax.

1 INTRODUCTION

  • ADPO uses a fixed, label-agnostic frequency mask shared across chosen and rejected responses, excluding high-frequency types from preference scores while retaining full context and softmax.It adds no learned selector, auxiliary reward model, or architecture change.Source: §1 · pp. 2–3
  • ADPO analyzes chosen–rejected gradient cosine similarities and high-frequency-token preference signals to formalize a preference-tuning objective.The work presents this as a transparent, computationally inexpensive regularizer for preference fitting.Source: §1 · p. 2
  • Frequent token types occupy substantial shared response mass yet show weaker token-identity preference signals than retained types in the studied datasets.This analysis motivates testing whether suppressing their direct score contributions can reduce the alignment component of gradient entanglement.Source: §1 · p. 2
  • ADPO characterizes changes to both score-gradient selection and the shared sigmoid gate, and reports reduced chosen–rejected score-gradient alignment at fixed checkpoints.The intervention changes direct gradient terms and the pair-level sigmoid multiplier on retained terms.Source: §1 · p. 2, Figure 1 · p. 2

2 RELATED WORK

Related work spans alternative preference objectives, gradient-coupling analyses, token-level weighting, and frequency-based approaches. ADPO retains DPO’s frozen reference and logistic pairwise loss while using a fixed frequency selector and defining anisotropy through unequal token-score coefficients.

2 RELATED WORK

  • ADPO retains DPO’s frozen reference and logistic pairwise loss, changing which realized token terms enter the margin rather than introducing a new preference model.Other reviewed approaches include reference-free objectives and alternatives to DPO’s preference-learning objective.Source: §2 · p. 3
  • ADPO uses a fixed frequency selector, whereas related work addresses gradient entanglement with pairwise gradient normalization or sparsity-regularized contextual token masking.Other cited work examines likelihood displacement, reward-shape changes, and normalized-reward objectives.Source: §2 · p. 3
  • Unlike token-level methods using reward estimators or learned sparse masks, ADPO uses neither a token-reward estimator nor importance-sampling correction and deliberately changes the objective.The reviewed methods include contextual token contributions, token-level reward guidance, and token-level KL control.Source: §2 · p. 3
  • ADPO defines anisotropy through unequal token-score coefficients, distinct from representation anisotropy concerning learned representation geometry.The related-work discussion also distinguishes occurrence mass, token-identity specificity, contextual prediction, and reward-model sensitivity in its analyses.Source: §2 · p. 3

3 ANALYSIS

DPO applies the same global coefficient to every token, despite token contributions producing different gradients. The analysis finds that frequent types account for much shared response content but carry weaker preference-side signal, and examines frequency masking as a way to restrict direct preference scoring while retaining context.

3.1 THE ISOTROPIC SCALING ASSUMPTION IN DPO

  • Standard DPO applies the same global β to every realized token, although token contributions can produce different gradients through shared model parameters.The analysis frames this equal weighting as DPO’s isotropic scaling assumption.Source: §3.1 · p. 4

3.2 A STRUCTURAL FREQUENCY PRIOR

  • Frequent types make up much of chosen–rejected overlap, while held-out preference-side associations reproduce more strongly for retained types across the examined tokenizers.The frequent-type set is frozen on training data and evaluated on disjoint held-out responses from the same release.Source: §3.2 · p. 4

3.3 EFFECT OF FREQUENCY MASKING ON GRADIENT ALIGNMENT

  • Removing frequent score terms lowers mean chosen–rejected gradient cosine at every measured DPO checkpoint, suggesting reduced directional coupling under frequency-hard scoring.Comparisons use identical weights, response pairs, and context; this is a diagnostic, not a claim that masking generally eliminates gradient entanglement.Source: §3.3 · pp. 4–5

3.4 PREFERENCE SIGNAL IN FREQUENT TOKENS

  • Frequent-only fragments yield a 0.08779 reward margin and 51.7% human-label agreement, while removing frequent tokens retains a 0.97335 margin and 61.4% agreement.These comparisons use a held-out HH sample scored with frozen PairRM; the table caption specifies that margins are raw PairRM logits, not reward probabilities.Source: Table 1 · p. 6, §3.4 · p. 6

4 ANISOTROPIC DPO

ADPO, also called Frequency-hard DPO, uses empirical response-token frequencies to apply a fixed binary weighting to token-score terms. Its canonical mask excludes frequent types from the preference score while retaining the original pairwise loss and all tokens as context.

4 ANISOTROPIC DPO

  • The method is fixed binary weighting of realized token-score terms based on empirical response frequencies.Source: §4 · p. 6

4.1 THE DPO TOKEN SCORE

  • The intervention changes which terms enter the score while retaining the logistic pairwise loss and frozen reference.Source: §4.1 · p. 6

4.2 THE FIXED FREQUENCY MASK

  • The fixed mask counts response-token frequency on both sides of preference pairs, making selection invariant to swapping preference labels.The canonical configuration uses the first 4,000 training pairs, chat formatting, prompt/sequence caps of 256/512 tokens, and response EOS; counts are recomputed for each tokenizer.Source: §4.2 · pp. 6–7
  • At τ = 0.002, types above the frequency threshold receive zero weight, unseen types retain unit weight, and the mask stays fixed during training.The mask contains 50 vocabulary types whose empirical frequency exceeds the threshold.Source: §4.2 · p. 7, Table 2 · p. 7

4.3 THE FREQUENCY-HARD OBJECTIVE

  • The effective coefficient is βit = β wτ(yit), and the objective reduces exactly to standard DPO when no realized types are masked.Source: §4.3 · p. 7
  • ADPO combines score-gradient selection with pair-dependent gate reweighting, while retaining all original tokens in context and the full softmax vocabulary.Scores are not renormalized by retained token count; zero weight does not freeze an output row or mask attention, and shared parameters can still receive indirect gradients.Source: §4.3 · p. 7

5 RESULTS

Across Qwen2.5-7B-Instruct and Llama-3-8B-Instruct, ADPO reports higher scores than DPO on the listed benchmarks and the highest reported score among compared methods on each metric. The evaluation also tests frequency-threshold sensitivity and alternative token-selection strategies.

5.2 MAIN RESULTS

  • ADPO achieves the highest reported score among compared methods on every listed metric for both evaluated models.For Qwen2.5-7B-Instruct, FsFairX mean reward rises from 1.231 to 1.308, MT-Bench from 8.43 to 8.53, AlpacaEval 2.0 length-controlled win rate from 49.72% to 52.78%, and Arena-Hard win rate from 52.1% to 53.4%; for Llama-3-8B-Instruct, the corresponding values are 2.15 to 2.20, 7.66 to 7.89, 50.43% to 53.06%, and 34.9% to 36.5%.Source: Table 3 · p. 8, Table 4 · p. 8, §5.2 · p. 9

5.1 EXPERIMENTAL SETUP

  • The main evaluation uses on-policy preference pairs generated by sampling five responses per UltraFeedback prompt and selecting highest- and lowest-ranked responses.The methods are trained for one epoch with global batch size 128; configurations are selected through a β sweep except for separately scaled SimPO.Source: §5.1 · p. 8

5.2 MAIN RESULTS

  • On Llama-3-8B-Instruct, τ = 0.002, masking approximately 50 token types, yields the highest mean reward and AlpacaEval 2.0 length-controlled win rate among tested thresholds.Masking approximately 25 or 100 tokens gives lower scores on both metrics, and none of the alternative token-selection strategies benefits over standard DPO.Source: §5.2 · p. 9, Table 5 · p. 9, Table 6 · p. 9

6 CONCLUSION

The paper concludes that selectively masking frequent token types from the direct preference score mitigates gradient interference without discarding context, modifying preference pairs, or adding learned parameters. It reports improved alignment outcomes across multiple benchmarks and the two evaluated models.

6 CONCLUSION

  • The paper concludes that standard DPO's isotropic scaling allows a small subset of frequent tokens to dominate and entangle preference gradients.Source: §6 · p. 10
  • ADPO masks frequent token types from the direct preference score without discarding context, modifying pairs, or adding learned parameters.Source: §6 · p. 10
  • Evaluations on Qwen2.5-7B-Instruct and Llama-3-8B-Instruct report improved outcomes across multiple benchmarks.Source: §6 · p. 10

A.1 FREQUENCY STRUCTURE ACROSS TOKENIZERS

Across three tokenizers, training-derived frequency masks capture disproportionate held-out chosen–rejected overlap, while retained types show stronger reproducible preference-side associations.

A.1 FREQUENCY STRUCTURE ACROSS TOKENIZERS

  • The audit sets the frequent-type threshold at τ = 0.002, freezes masks from the first 4,000 training pairs, and calibrates side associations on pairs 4,001–8,000.The selected token IDs are evaluated on 1,961 disjoint test pairs.Source: §A.1 · p. 13
  • Frequency masks cover 36.06–40.94% of held-out response occurrences but capture 47.43–52.49% of shared occurrence mass, with weaker token-identity side specificity.The pattern holds across all three tokenizers.Source: §A.1 · p. 13, Table 7 · p. 14
  • Retained types have larger cross-fitted side signals under every tokenizer, with each paired interval for the difference above zero.Evaluation-pair bootstrap intervals keep the training-derived mask and calibration associations fixed.Source: §A.1 · p. 13, Table 8 · p. 14
  • Table 9 lists each tokenizer’s selected token IDs and vocabulary-piece strings in descending calibration-frequency order.The escaped strings preserve whitespace, byte, and special-token distinctions; positions across tokenizers do not imply matching token identities.Source: §A.1 · p. 14, Table 9 · p. 14

A.2 CONTROLLED TOKEN SELECTION

At a matched response-occurrence budget, frequency selection captures more shared chosen–rejected mass and weaker token-identity side specificity than randomized selection.

A.2 CONTROLLED TOKEN SELECTION

  • The Qwen2.5 frequency selector uses 52 types covering 1,194,145 of 3,176,166 response occurrences, and random selections match that occurrence count rather than type count.The 1,000 randomized selections are binary type-level selections; their vocabulary sizes may vary.Source: §A.2 · p. 15
  • 49.03% shared coverage for frequency selection exceeds the random mean of 37.46%, with shared enrichment of 1.304.The comparison uses randomized selections matched to the frequency selector’s occurrence budget.Source: §A.2 · pp. 15–16, Table 10 · p. 15
  • The occurrence-matched control indicates that the structural concentration is not explained by selected occurrence mass alone.Together with held-out replication, the token-count analyses support frequency as a transparent selector of shared, weakly side-specific response content.Source: §A.2 · pp. 15–16

Related papers

Loading 2609.32445v1…