PaperlayerView exact revision on arXiv

Source-linked AI summary

On the Sparsity of Direct Preference Optimization: Weight Disentanglement in the NTK Regime

Kensuke Sasaki, Issei Sato

arXiv:2610.37085v1cs.LG

TL;DR

The paper addresses the lack of a theoretical explanation for DPO’s sparse parameter updates. It analyzes DPO under NTK linearization, relating its implicit maximum-margin bias to dual sparsity and weight disentanglement. Experiments report sparsity from GPT-2 to 7–9B models and lower cross-task interference than SFT.

  • Problem

    Why DPO induces sparse parameter updates without explicit sparsity-promoting regularization, and whether that sparsity reduces interference between independently trained task vectors, remain open questions.

    Source: §3 · p. 3

  • Method

    The paper analyzes DPO under NTK linearization, formalizes dual sparsity, and relates it to weight disentanglement through a triplet-space framework.

    Source: §1 · p. 2, §3.3 · p. 6

  • Results

    DPO reduces cross-task interference by 11× to 35× relative to SFT, and both sources of sparsity hold from GPT-2 to 7–9B models.

    Source: §1 · p. 2, §6 · p. 10

  • Takeaways & Limitations

    The analysis provides a theoretical explanation for sparse DPO updates and relates that structure to weight disentanglement, a condition favorable for model merging.

    Source: §3.3 · p. 6

    The analysis relies on NTK linearization, and parameter-level sparsity is an empirically motivated assumption that may weaken or vanish when preferred and dispreferred responses have no token overlap.

    Source: §6 · p. 10

Abstract

from arXiv · show

Direct Preference Optimization (DPO) has been widely adopted for aligning language models with human preferences. Recent empirical work has revealed that DPO induces remarkably sparse parameter updates, yet no theoretical explanation for this phenomenon has been offered. We analyze DPO's optimization dynamics under a neural tangent kernel linearization and show that the logistic structure of the DPO loss gives rise to an implicit maximum-margin bias. Building on this observation, we introduce the concept of dual sparsity: the DPO update direction is a sparse linear combination (data-level sparsity, driven by support vectors) of sparse gradient feature vectors (parameter-level sparsity, arising from gradient cancellation between preferred and dispreferred responses). We formalize this structure in our main theorem, bound the number of support pairs by a margin-dependent spectral quantity, and present a framework relating dual sparsity to weight disentanglement---a condition favorable for model merging. Experiments on GPT-2 Small and Medium and on 7--9B instruction-tuned models (Llama-3-8B, Mistral-7B, Gemma-2-9B) with the Anthropic HH-RLHF and UltraFeedback datasets confirm both sources of sparsity, and experiments on GPT-2 show that DPO reduces cross-task interference by up to $35{\times}$ compared to supervised fine-tuning.

1 Introduction

The paper explains DPO’s sparse updates through an NTK maximum-margin analysis and connects dual sparsity to weight disentanglement. Experiments validate both forms of sparsity and report reduced cross-task interference relative to SFT.

1 Introduction

  • The paper addresses the poorly understood observation that DPO induces sparse updates and asks whether that structure favors composing task vectors.Prior work identifies weight disentanglement—task vectors acting on non-overlapping parameter regions—as a property enabling successful model merging.Source: §1 · p. 1
  • Under NTK linearization, DPO’s logistic loss drives its update toward a maximum-margin solution with data-level and parameter-level sparsity.Only a small subset of preference pairs contribute to the update, while each pair’s gradient feature vector is sparse through cancellation between preferred and dispreferred responses sharing tokens.Source: §1 · p. 2
  • The paper relates dual sparsity to weight disentanglement through a triplet-space formulation that reduces the condition to a per-triplet inner product governed by task overlap.The framework connects the magnitude of that inner product to parameter-space overlap between tasks.Source: §1 · p. 2
  • DPO reduces cross-task interference by 11× to 35× relative to SFT across all configurations, while both sparsity sources hold from GPT-2 to 7–9B models.DPO feature vectors are over 90% near-zero from 124M to 9B parameters, and time-integrated weights concentrate on a small subset of pairs within three epochs at 7–8B scale.Source: §1 · p. 2

2 Preliminaries

The preliminaries describe DPO’s preference-pair optimization and introduce weight disentanglement as a condition for composing task-specific updates. They also define task vectors and relate weight disentanglement to task arithmetic.

2.1 Direct Preference Optimization

  • DPO’s gradient is weighted by σ(−mi(θ)), so optimization focuses on hard preference pairs near the decision boundary as gradients for already-preferred pairs vanish.The setup uses a reference policy and preferred/dispreferred response pairs; the supplied dataset description is incomplete.Source: §2.1 · p. 2

2.2 Weight Disentanglement

  • A task vector is the difference between task-optimal and pretrained parameters; weight disentanglement characterizes an ideal condition for composing task vectors without cross-task interference.The text states that weight disentanglement implies the task arithmetic property.Source: §2.2 · pp. 2–3

3 Theoretical Analysis

The analysis models DPO under an NTK linearization, where logistic-loss optimization has a maximum-margin direction. It derives dual sparsity under stated assumptions and relates the structure to weight disentanglement through cross-task inner products.

3.1 DPO as Maximum-Margin Classification in Tangent Space

  • Under linear separability, the normalized DPO update converges to the L2 maximum-margin direction in the linearized model.The analysis assumes the reference-model initialization and uses the implicit-bias result for logistic regression.Source: §3.1 · p. 4

3.2 Dual Sparsity of DPO Updates

  • The support-pair count is bounded by a margin-dependent spectral quantity and can be much smaller than the dataset when the margin is large relative to feature scale.The cumulative gradient weight concentrates on pairs that remain near the margin boundary during training.Source: §3.2 · p. 4
  • Theorem 3 shows that, under linear separability and sparse gradient features, the update is a sparse combination of sparse vectors.Data-level sparsity comes from support pairs; parameter-level sparsity is assumed for each gradient feature, motivated by cancellation between shared response tokens.Source: §3.2 · pp. 4–5

3.3 Connection to Weight Disentanglement

  • Under NTK linearization, DPO's margin has the weight-disentanglement form, with the condition holding to the extent that cross-task inner products are small.Sparse active parameter supports can make these inner products small when their overlap is limited; whether naturally occurring preference tasks have such overlap is evaluated empirically.Source: §3.3 · pp. 5–6

4 Experiments

Experiments test DPO’s data- and parameter-level sparsity and its relationship to cross-task interference, using GPT-2 and instruction-tuned models across preference datasets. DPO concentrates updates on support pairs, yields sparse gradient features, and reduces cross-task interference relative to SFT.

4.1 Experimental Setup

  • The experiments use GPT-2 Small and Medium plus three 7–9B instruction-tuned models, with HH-RLHF and UltraFeedback preference data.Gradient-feature experiments use dataset subsamples because storing full-dimensional vectors is memory-intensive.Source: §4.1 · p. 6

4.2 Data-Level Sparsity

  • As training progresses, time-integrated weights become increasingly concentrated on a small subset of preference pairs, consistent with support-pair coefficients becoming nonzero only for support pairs.The concentration is seen for GPT-2 Small and Medium; top-10% mass grows from about 11% to 40–44% for GPT-2 and to 44–52% for 7–8B models within three epochs of full fine-tuning DPO.Source: §4.2 · pp. 6–7, Figure 1 · p. 7, Table 1 · p. 7

4.3 Parameter-Level Sparsity

  • From 124M to 9B parameters, more than 90% of DPO feature-vector entries are near zero, compared with 4–20% for SFT.The reported mechanism is gradient cancellation between preferred and dispreferred responses; SFT features are substantially denser.Source: §4.3 · pp. 7–8, Table 2 · p. 8, Figure 2 · p. 8

4.4 Cross-Task Interference

  • DPO suppresses cross-task interference by an order of magnitude or more in every tested configuration, and causes less performance degradation than SFT under merging.The comparisons cover two model scales and both within- and cross-dataset task pairs.Source: Table 3 · p. 9, §4.4 · p. 9
  • Cross-task interference is assessed using top-1% Jaccard overlap between active parameters and the inner product of one task’s DPO update direction with the other task’s mean feature vector.The mean-feature-vector summary makes the per-triplet disentanglement condition tractable to evaluate.Source: §4.4 · p. 9

5 Related Work

Related work studies alternative DPO objectives, DPO dynamics under NTK linearization, and task-vector merging. This paper distinguishes its focus on update directions and intrinsic sparsity from prior analyses and post-hoc sparsification.

Optimization dynamics of DPO

  • Prior work on IPO and KTO modifies the DPO loss to address specific failure modes, whereas this analysis concerns the original objective.IPO avoids the Bradley–Terry assumption; KTO adopts a prospect-theoretic utility.Source: §5 · p. 9
  • Ren and Sutherland [2025] analyze DPO under NTK linearization for the squeezing effect, while this work studies update directions and dual sparsity.The paper presents likelihood displacement and dual sparsity as complementary consequences of the dynamics, concerning changed probabilities and changed parameters, respectively.Source: §5 · p. 10

Task arithmetic, weight disentanglement, and model merging

  • Unlike post-hoc sparsification methods such as DARE [Yu et al., 2024] and TIES-Merging [Yadav et al., 2023], this framework treats DPO sparsity as emerging intrinsically from optimization.The related-work discussion says a larger-scale study of merging is left to future work.Source: §5 · p. 10
  • The paper relates approximate weight disentanglement under DPO to reduced cross-task interference and reports less performance degradation than SFT under task-vector merging.Prior work identifies disjoint parameter-space regions as a sufficient condition for task-vector composition.Source: §5 · p. 10, Table 4 · p. 10

6 Conclusion

The conclusion attributes DPO’s sparse updates to maximum-margin dynamics under NTK linearization and reports lower cross-task interference and merging degradation than SFT. It also identifies assumptions and scope boundaries that limit the analysis.

6 Conclusion

  • Under NTK linearization, DPO’s logistic loss drives updates toward a maximum-margin solution with data-level support-pair sparsity and parameter-level sparsity from gradient cancellation.The authors report both sources of sparsity from GPT-2 to 7–9B models.Source: §6 · p. 10
  • DPO reduces cross-task interference by 11× to 35× relative to SFT and degrades less under model merging.These empirical findings are reported alongside the sparsity results across GPT-2 to 7–9B models.Source: §6 · p. 10
  • The analysis relies on NTK linearization, which is not guaranteed to hold throughout large-model DPO training.Parameter-level sparsity is an empirically motivated assumption; its cancellation mechanism may weaken or vanish when preferred and dispreferred responses are entirely disjoint.Source: §6 · p. 10

A Proof and Numerical Evaluation of Lemma 2

The proof recasts the frozen-NTK DPO margin problem as hard-margin classification and bounds the number of support pairs using the feature-kernel spectrum and margin.

A.1 Proof

  • Under the frozen-NTK surrogate, DPO loss reduces exactly to logistic regression; with the reference policy equal to initialization, the offsets vanish and the margin problem has an active unit-norm constraint.The positive optimal margin follows from assumption 1.Source: §A.1 · p. 12
  • The maximum-margin problem is equivalent to minimizing the squared norm subject to each feature having margin at least one, with the normalized optimizer defining the unit update direction.The KKT conditions provide the representation used in the support-pair bound.Source: §A.1 · p. 12
  • The support-pair count is bounded by min{n, floor(λmax(K)/γ⋆^2)}, linking its size to the kernel’s largest eigenvalue and the squared margin.The spectral step uses that the support-set kernel is a principal submatrix of the positive semidefinite full kernel.Source: §A.1 · p. 12

A.2 Numerical Evaluation

The section evaluates the support-set bound using a finite-time margin, which is no greater than the max-margin value. On actual GPT-2 training trajectories, the resulting conservative estimate places the support set at roughly one-seventh to one-ninth of the 100 pairs.

A.2 Numerical Evaluation

  • The reported bound evaluation uses a finite-time margin rather than solving the max-margin problem exactly.For actual training trajectories, the convergence claim does not apply, but the inequality still holds because the normalized update has unit norm.Source: §A.2 · pp. 12–13
  • The finite-time margin is at most the max-margin value, and under frozen-NTK gradient flow it converges to that value.The finite-time margin is computed from the normalized parameter update.Source: §A.2 · pp. 12–13
  • For GPT-2 Small and Medium, the conservative support-set estimate is about 1/7 to 1/9 of n, roughly an order of magnitude below n.The values use the actual training trajectory, where the finite-time margin remains a lower bound on the max-margin value.Source: §A.2 · p. 13, Table 5 · p. 13

B Experimental Details

The experimental details summarize the models, datasets, and pair counts used across experiments. The 7–8B full-fine-tuning runs use up to eight NVIDIA A100 GPUs.

B Experimental Details

  • The accumulated pair weights A_i are computed from training-step terms involving σ(−m_i), so this quantity requires no gradient features.Source: B Experimental Details · p. 13
  • Table 6 summarizes the models, datasets, and preference-pair counts used for each experiment.“Pairs” denotes the number of preference pairs used to compute each quantity; HH and UF refer to Anthropic HH-RLHF and UltraFeedback Binarized, and ep. means epochs.Source: B Experimental Details · p. 13, Table 6 · p. 13
  • The 7–8B models' full-fine-tuning DPO runs use up to eight NVIDIA A100 GPUs.Source: B Experimental Details · p. 13

C Evolution of Log-Probabilities During DPO Training

During DPO training, both preferred and dispreferred response log-probabilities decrease, but the dispreferred response decreases faster. The widening per-pair gap increases the margin and stops easy pairs from contributing to the accumulated weights.

C Evolution of Log-Probabilities During DPO Training

D Model Merging Experiments

The model-merging experiments compare DPO and SFT task vectors by merging them with a target task vector and measuring performance change. DPO degrades less than SFT on both task pairs, though the performance gap is smaller than the reduction in cross-task inner product.

D Model Merging Experiments

  • For each target task, the merged model adds a validation-selected scaled sum of target and other task vectors to the initial model.The scale α is selected from 0 to 2.0 in increments of 0.1, and the reported change is single-task minus merged performance.Source: D Model Merging Experiments · p. 14, Table 8 · p. 14
  • The comparison uses GPT-2 Medium task vectors taken from each method's best checkpoint after fine-tuning to convergence.Source: D Model Merging Experiments · p. 14
  • DPO degrades less than SFT under task-vector merging on both task pairs.The performance gap is smaller than the order-of-magnitude reduction in cross-task inner product because merged performance also depends on other factors.Source: D Model Merging Experiments · p. 14, Table 8 · p. 14

Related papers

Loading 2610.37085v1…