PaperlayerView exact revision on arXiv

Source-linked AI summary

Disentangling Optimization Scale from Preference Scale in DPO

Ivan Kruzhilov

arXiv:2608.27032v1cs.LG

TL;DR

The paper examines how DPO’s β entangles preference-noise scaling with optimization step size, making β-sensitive training behavior and loss comparisons difficult to interpret. It introduces a centered-softplus reformulation that preserves the β > 0 optimum while separating these effects, with a continuous β→0 linear-margin endpoint. Experiments and analysis show non-monotone policy deviation at fixed learning rate and non-comparable standard loss values across β.

  • Problem

    DPO commonly treats β as controlling the preference–KL trade-off, but β also changes gradient magnitude and therefore the effective update scale.

    Source: §1 · p. 1

  • Method

    The paper proposes a centered-softplus reformulation that preserves the original DPO minimizer for β > 0 while making preference-noise and learning-rate effects independently tunable.

    Source: §1 · p. 2, §5 · p. 10

  • Results

    At fixed learning rate, policy deviation is non-monotone in β, while nearly identical standard DPO loss curves can correspond to several-fold differences in KL divergence.

    Source: §1 · p. 2, §3.1 · p. 3, §3.4 · p. 6

  • Takeaways & Limitations

    The reformulation exposes β’s two roles and provides a loss signal comparable across β, while β = 0 yields a less-regularized linear preference-margin regime.

    Source: §1 · p. 2, §5 · p. 10

    Empirical evidence covers three preference datasets and four LoRA-fine-tuned backbones, so transfer to larger models, full fine-tuning, and other data regimes remains to be verified.

    Source: Limitations and Broader Impacts · p. 11

Abstract

from arXiv · show

Direct Preference Optimization (DPO) is a widely used objective for aligning language models from preference data, with the coefficient $β$ commonly interpreted as controlling the KL constraint to a reference policy. We show that $β$ entangles two distinct roles: it governs the effective inverse preference-noise scale and simultaneously rescales the optimization dynamics, coupling this scale with the effective step size. As a consequence, at a fixed learning rate the achieved policy deviation is non-monotone in $β$: it vanishes in a dead zone at small $β$, reaches a peak at an intermediate value, and decreases again for larger $β$. Moreover, standard DPO loss values are not comparable across $β$: runs with nearly identical loss curves can differ several-fold in KL divergence from the reference model. This entanglement obscures the role of $β$, increases sensitivity to hyperparameter choices, and complicates learning-rate scheduling. We propose a centered-softplus reformulation that is argmin-equivalent to DPO for $β>0$, while making the inverse preference-noise-scale and learning-rate effects explicit and independently tunable. The normalized centered-softplus objective also admits a continuous $β\to0$ endpoint that reduces to a linear preference-margin objective.

1 Introduction

The paper argues that DPO’s β simultaneously controls preference-noise scaling and optimization update magnitude, contrary to treating it only as a preference–KL trade-off parameter. This entanglement produces weak updates at small β, obscures loss comparisons across β, and motivates a centered-softplus reformulation.

  • β controls both the preference–KL trade-off and the effective update scale, not only the regularization behavior.The paper identifies the optimizer-scale role as a previously unisolated aspect of standard DPO objectives.Source: §1 · p. 1
  • Small β reduces gradient magnitude, leaving the policy close to the reference model instead of producing larger policy deviation.Under SGD, decreasing β is equivalent to lowering the learning rate, complicating hyperparameter tuning.Source: §1 · p. 1
  • Standard DPO loss values are not comparable across β because the loss depends on β and the preference margin through their product.Runs with nearly identical loss curves can nevertheless differ several-fold in KL divergence from the reference model.Source: §1 · pp. 1–2
  • At fixed learning rate, policy deviation is non-monotone in β: it enters a small-β dead zone, peaks near βpeak ≈4.6τ, then decreases for larger β.Here τ denotes the effective stopping tolerance of optimization.Source: §1 · p. 2
  • The centered-softplus loss preserves DPO’s optimum for β > 0 while exposing and decoupling preference-noise-scale and learning-rate effects.Its β = 0 endpoint becomes a linear preference-margin objective.Source: §1 · p. 2

2 Background: KL-Regularized RLHF and DPO

DPO collapses KL-regularized RLHF’s reward-model fitting and policy optimization into a single objective over policy log-probabilities. Its coefficient β reflects the KL coefficient relative to preference annotator-noise scale, while DPO-family objectives commonly retain an explicit β-dependent gradient factor.

  • KL-regularized RLHF balances reward maximization against staying close to a reference policy using λ as the KL-regularization coefficient.Source: §2 · p. 2
  • DPO collapses the two-stage reward-model-and-PPO pipeline into a single objective over policy log-probabilities.The formulation avoids separately training a reward model.Source: §2 · p. 2
  • Under the Bradley–Terry model, β = λ/sa combines KL regularization λ with pairwise annotator-noise scale sa.The conventional unit-noise formulation sets sa = 1.Source: §2 · p. 2
  • The standard DPO objective is equivalently LDPO(∆; β) = log(1 + exp(−β∆)), with σ denoting the sigmoid function.Source: §2 · p. 3
  • Most DPO variants retain an explicit multiplicative β factor in the marginal gradient, so reducing β suppresses update magnitude.The same coupling is discussed for unpaired KTO, reference-free SimPO, and token-level TDPO, while IPO, ROPO, and AOT depart from the standard pattern.Source: §2 · p. 3

3 Gradient-Scale Coupling and Loss Incomparability in Standard DPO

Standard DPO’s β simultaneously scales preference gradients and shapes saturation, so fixed-learning-rate policy deviation becomes non-monotone and loss values become incomparable across β.

Dead Zone, Peak, and Saturation

  • Figure 1 reports final validation KL of 0.83 per token at β = 2 × 10−3 versus 0.54 at β = 10−2, violating a monotone KL-control reading.The β = 0.002 run overtakes β = 0.01 after epoch 3 in both gradient norm and KL.Source: §3.1 · p. 3, Figure 1 · p. 4

Gradient-Scale Coupling

  • 20× β sweeps show initial gradient ratios G0/β remain constant within roughly 2%, confirming linear gradient suppression.This result comes from HelpSteer3 SGD runs summarized in Table 2.Source: §3.1 · p. 4, Table 2 · p. 4
  • β directly multiplies the marginal gradient, suppressing update magnitude as β decreases and coupling preference scale to effective learning speed.At initialization, the full parameter-gradient norm is proportional to β.Source: §3 · p. 3, §3.2 · p. 5

Dead Zone, Peak, and Saturation

  • At fixed learning rate, final policy deviation has a dead zone at small β, peaks at an intermediate β, and decreases again at larger β.The stopping-tolerance model attributes these regimes to βσ(−β∆), including logarithmic margin growth and right-hand saturation.Source: §3.3 · p. 5
  • Figure 2 finds validation NLL, per-token KL, and mean margin follow the predicted dead-zone, intermediate-peak, and decaying-right-slope shape.Two descriptive τ estimates are approximately 5.99 × 10−4 and 6.37 × 10−4, with nearly indistinguishable overlays.Source: §3.3 · pp. 5–6, Figure 2 · p. 6

Loss Incomparability

  • Because standard DPO loss depends on β and margin only through u = β∆, nearly identical loss curves can correspond to policies differing several-fold in KL.Therefore neither training nor validation DPO loss supports comparing runs or selecting β by loss value.Source: §3.4 · pp. 6–7, Figure 3 · p. 7

4 Normalized Objective Formulation

The normalized centered-softplus objective removes β’s direct gradient scaling while preserving DPO minimizers for β > 0 and providing a continuous linear β = 0 endpoint.

4.1 Normalized DPO Loss

  • Dividing the standard softplus loss by β > 0 preserves its argmin while removing the divergent log(2)/β offset.The centered-softplus transformation supplies the corresponding reformulation.Source: §4.1 · p. 8, §4.2 · p. 8
  • In the normalized parameterization, β controls sigmoid shape and saturation but no longer directly rescales local gradient magnitude.β instead bounds curvature by β/4.Source: §4.1 · p. 8

4.2 Centered Softplus

  • For β > 0, centered softplus is argmin-equivalent to −log σ(β∆), while β → 0 yields sc(x; 0) = x/2.The β = 0 branch therefore extends the objective continuously.Source: §4.2 · p. 8
  • At β = 0, the objective becomes linear preference-margin maximization and does not saturate for already separated pairs.Its constant marginal gradient means it does not by itself impose a finite preferred margin.Source: §4.2 · p. 8
  • Unlike standard DPO, normalized centered softplus avoids small-β collapse toward a constant and maintains gradients over a margin range growing as 1/β.Its lower plateau is −ln 2/β and its marginal gradient stays near −1/2 across that range.Source: §4.2 · pp. 8–9
  • Under the normalized loss, achieved policy deviation is monotone in 1/β, without the classical dead zone or intermediate peak.The saturation margin scales as 1/β.Source: §4.2 · p. 9

4.3 Experiments

  • With lrnorm = β lrstandard, SGD validation trajectories are nearly equivalent, while normalized losses separate runs that standard losses make nearly indistinguishable.The experiment tests scale equivalence rather than superiority over a retuned DPO baseline.Source: §4.3 · p. 9
  • Across additional settings and datasets with AdamW, stronger policy evolution corresponds to faster decreases in normalized training loss, including the β = 0 branch.Standard DPO loss can instead approach log 2 regardless of actual policy movement.Source: §4.3 · pp. 9–10

5 Discussion

The reformulation separates preference-noise scaling from optimization scaling without changing the DPO optimum for β > 0, while clarifying β’s statistical interpretation.

Discussion

  • The centered-softplus reformulation preserves the original DPO minimizer for β > 0 and explicitly decouples preference-noise scale from update scale.The paper does not claim a different optimum or universal gain over retuned DPO.Source: §5 · p. 10
  • DPO’s β = λ/sa combines the KL coefficient λ and annotator-noise scale sa, which are not separately identifiable without reward normalization.This follows from substituting the reward–policy relation into a logistic preference model.Source: §5 · p. 10
  • Subtracting ln 2 gives the negative log-likelihood ratio against random choice, while dividing by β removes the inverse preference-noise prefactor from the marginal gradient.The interpretation uses the Bradley–Terry Gumbel-noise view.Source: §5 · p. 10

6 Conclusion

The paper recommends normalized centered softplus when β-dependent gradient scaling or cross-β loss comparability matters, presenting it as a practical loss rewrite rather than a universal performance upgrade.

Conclusion

  • The normalized objective resolves small-β gradient attenuation and makes preference-margin and validation pair-NLL changes monotone in β, equivalently in 1/β.Reported sweeps show validation KL co-moves with margin over a wide β range without claiming a universal KL–margin law.Source: §6 · p. 10
  • The reformulation is a local loss-evaluation change that can be integrated into existing DPO codebases without restructuring the trainer or data pipeline.The paper frames this as an implementation property, not a performance guarantee.Source: §6 · p. 10

Reproducibility, Data, and Compute

The experiments use multiple language-model backbones, public pairwise preference datasets, and LoRA fine-tuning, with additional downstream instruction-following checks.

Data and models

Datasets

Evaluation

Compute and protocols

Limitations and Broader Impacts

The paper’s scope is limited by moderate-scale LoRA experiments and optimizer- and implementation-dependent trajectories, while its broader impacts are those of language-model alignment generally.

Limitations

Broader impacts

Relation to variants

Optimizer dependence

Threshold scope

Training-stage dependence

C Additional Training Dynamics

Additional training dynamics reproduce β-dependent optimization effects across datasets, models, optimizers, and preference-noise settings, while showing stage-dependent compensation and link-function differences.

UltraFeedback

Cross-dataset replication

  • The same training and validation dynamics appear on PKU-processed HH-RLHF with Mamba-2.Source: Figure C2 · p. 20

Small-β behavior

  • Figure C3 shows that β ∈{10−8, 10−6} keeps gradients low and validation KL near zero, whereas β = 10−3 moves appreciably.Source: Figure C3 · p. 20

SGD dynamics

  • Figure C5 compares HH-RLHF SGD trajectories for β ∈ {0.001, 0.002, 0.005, 0.01, 0.02}, where smaller-β runs can show delayed gradient increases.Source: C Additional Training Dynamics · p. 21

Mechanism

Non-monotone dynamics

  • For sufficiently small β, delayed compensation remains weaker than early attenuation, producing non-monotone dynamics across practical tuning ranges.Source: C Additional Training Dynamics · p. 21

General link functions

Thurstone comparison

E Formal Proof of Proposition 1 and Relation to Prior Work

The centered-softplus reformulation preserves DPO’s optimizer for β > 0, extends continuously to β = 0, and approaches a hard-margin objective as β grows.

Proposition 1

β = 0 endpoint

Hard-margin limit

F Bradley–Terry Scale and Log-Likelihood-Ratio View

The Bradley–Terry view identifies β as an effective inverse preference-noise scale in policy-log-ratio coordinates, while standard DPO also couples β to optimization scale. Centered-softplus preserves the statistical role while decoupling optimizer scaling.

Bradley–Terry scale

  • β = λ/sa is the effective inverse preference-noise scale, combining KL/reward scale λ with annotator-noise scale sa.The DPO likelihood identifies their ratio rather than λ and sa separately.Source: F Bradley–Terry Scale and… · p. 26

Log-likelihood-ratio view

  • The DPO loss models an observed preference with probability σ(β∆), where ∆ is the policy’s log-ratio gap relative to the reference.The product β∆ acts as a scale-controlled evidence score for the observed preference.Source: F Bradley–Terry Scale and… · p. 26

Normalization

  • Subtracting the chance baseline and dividing by β removes the direct β prefactor from the marginal gradient.The normalized quantity equals annotator-noise units only under a fixed reward/KL normalization.Source: F Bradley–Terry Scale and… · p. 26

Disentangled roles

  • Standard DPO gives β two roles: statistical inverse scale β = λ/sa and an unintended optimization-scale gradient multiplier.The latter couples β to effective step size.Source: F Bradley–Terry Scale and… · p. 26
  • Centered-softplus removes the optimization-scale role while preserving the statistical coefficient, allowing the two effects to be tuned independently.Separating λ from sa still requires external normalization or additional information.Source: F Bradley–Terry Scale and… · p. 26

G Saturation Thresholds: Where Training Stops as a Function of β

A fixed-learning-rate threshold model explains why standard DPO training stops at a β-dependent margin: small β can create a dead zone, intermediate β a maximum, and large β declining reachable margins. The model is an approximate saturation description whose empirical interpretation requires care.

Standard DPO regimes

  • For β > 2τ, standard DPO’s saturation margin follows β ∆∗DPO = log(β/τ − 1), so the product grows logarithmically with β.This differs from the normalized objective, where β∆∗ is constant and ∆∗ is monotone in 1/β.Source: G Saturation Thresholds: Where… · p. 27
  • β ≤ 2τ defines a dead zone where the initial marginal gradient is below tolerance and training remains at initialization.The normalized objective has no corresponding dead zone because its initial marginal gradient is 1/2 for every β.Source: G Saturation Thresholds: Where… · p. 27

Peak location

  • βpeak ≈ 4.6τ is the unique maximum of the reachable margin under the fixed-learning-rate threshold model.The maximum is determined by the stopping tolerance τ.Source: G Saturation Thresholds: Where… · p. 28

Empirical illustration

  • In the HelpSteer3 SGD sweep, final per-token KL {0.04, 0.35, 0.83, 0.54} peaks at β = 2×10−3 among the four sampled values.On the dense grid, min NLL, max KL, and max mean ∆ occur between β = 2.5 × 10−3 and 3 × 10−3.Source: G Saturation Thresholds: Where… · p. 28

Estimating τ

  • The βpeak-based estimate of τ is primary, while fitting the β-shape of ∆ with a free vertical scale provides an internal consistency check.The two estimates are not out-of-sample validation because both use the same β-sweep.Source: §G.1 · pp. 28, 30

Scope and caveats

  • The threshold model applies only when training reaches saturation before the epoch budget ends and treats τ as a coarse effective tolerance under decaying learning rates.Very small-β runs may stop early because updates are weak, rather than because the loss has saturated.Source: §G.1 · p. 31

H Loss-Level Scale Invariance: Identical Loss Curves, Different Policies

Standard DPO loss depends on β and the preference margin only through u = β∆, so matched scale-equivalent runs can show nearly identical losses while producing different policy deviations and KL values.

Empirical comparisons

Dynamics

Structural invariance

  • The standard pairwise loss is invariant under β∆, making its level sets hyperbolas and obscuring physical margin and KL differences.For example, (β, ∆) = (0.01, 100) and (0.02, 50) have identical loss despite a twofold difference in policy displacement.Source: H Loss-Level Scale Invariance:… · p. 35

J Logged Gradients, Mean-Abs Magnitudes, and the Jacobian Factor g

The gradient analysis separates explicit β scaling, sigmoid saturation, and the margin Jacobian, while distinguishing mean-absolute logs from Euclidean gradient norms and cautioning against overinterpreting minibatch proxies.

Minibatch interpretation

  • The minibatch proxy gB represents a coherence-weighted norm of the batch-averaged margin Jacobian rather than a typical per-pair Jacobian norm.Cross-pair alignment and cancellation can make gB differ substantially from individual Jacobian norms.Source: J Logged Gradients, Mean-Abs… · pp. 39–40

Logging conventions

  • The logged mean-absolute magnitude G and Euclidean norm differ by a near-constant shape factor, with r ≈2.8–3.1 on HelpSteer3 and r ≈3.7–4.1 on UltraFeedback.Source: J Logged Gradients, Mean-Abs… · pp. 40–41
  • TRL reports Euclidean ∥∇θL∥2, whereas some published SGD curves retain mean-absolute G despite axes labelled ∥∇L∥, creating a residual logging inconsistency.Source: J Logged Gradients, Mean-Abs… · p. 40

Gradient decomposition

  • For one preference pair, the DPO parameter gradient factors into an explicit β term, a sigmoid saturation term, and the margin Jacobian.This decomposition explains why a falling loss-gradient magnitude need not imply that the Jacobian itself is shrinking.Source: J Logged Gradients, Mean-Abs… · pp. 39–40

Horizon estimate

  • The scalar horizon estimate is only an order-of-magnitude guide because Jacobian variation, cross-pair cancellation, and stochasticity prevent a precise cutoff.The highlighted βr = 0.027 value is described as a closed-form ceiling or characteristic scale, not a calibrated threshold.Source: J Logged Gradients, Mean-Abs… · pp. 41–42, §K.1 · p. 43, §K.2 · p. 44

K.4 Numerical estimate on HelpSteer3

On the HelpSteer3 sweep, the scalar horizon estimate places the right-hand crossover near βr ≈0.027, but trajectory-dependent minibatch dynamics make it a broad practical scale rather than a sharp boundary.

Numerical estimate

  • Using Tbudget = 9078, ηmax = 2 × 10−4, H = 0.9078, τ = 6.37 × 10−4, and gB(θ0) ≈250 yields a rough βr estimate.Source: §K.4 · pp. 44–45

Uncertainty

  • Replacing the initial proxy with the later reconstructed scale gB ≈360–370 moves βr downward by about a factor of two, to roughly 0.012–0.013.Source: §K.4 · p. 44

Numerical estimate

  • The empirical βr row has NLL ≈12.88, KL ≈0.221, and mean validation margin ∆≈38.5, interpolated between β = 0.02 and β = 0.04.Source: §K.4 · p. 44
  • βr ≈0.027 places the right-hand crossover within the observed logarithmic range of the HelpSteer3 sweep.Source: §K.4 · pp. 44–45

Operational interpretation

Related papers

Loading 2608.27032v1…