PaperlayerView exact revision on arXiv

Source-linked AI summary

Disentangling Optimization Scale from Preference Scale in DPO

Ivan Kruzhilov

arXiv:2608.27032v1cs.LG

TL;DR

The paper examines how DPO’s β entangles preference-noise scaling with optimization step size, making β-sensitive training behavior and loss comparisons difficult to interpret. It introduces a centered-softplus reformulation that preserves the β > 0 optimum while separating these effects, with a continuous β→0 linear-margin endpoint. Experiments and analysis show non-monotone policy deviation at fixed learning rate and non-comparable standard loss values across β.

  • Problem

    DPO commonly treats β as controlling the preference–KL trade-off, but β also changes gradient magnitude and therefore the effective update scale.

  • Method

    The paper proposes a centered-softplus reformulation that preserves the original DPO minimizer for β > 0 while making preference-noise and learning-rate effects independently tunable.

  • Results

    At fixed learning rate, policy deviation is non-monotone in β, while nearly identical standard DPO loss curves can correspond to several-fold differences in KL divergence.

  • Takeaways & Limitations

    The reformulation exposes β’s two roles and provides a loss signal comparable across β, while β = 0 yields a less-regularized linear preference-margin regime.

  • Takeaways & Limitations

    Empirical evidence covers three preference datasets and four LoRA-fine-tuned backbones, so transfer to larger models, full fine-tuning, and other data regimes remains to be verified.

Abstract

from arXiv · show

Direct Preference Optimization (DPO) is a widely used objective for aligning language models from preference data, with the coefficient $β$ commonly interpreted as controlling the KL constraint to a reference policy. We show that $β$ entangles two distinct roles: it governs the effective inverse preference-noise scale and simultaneously rescales the optimization dynamics, coupling this scale with the effective step size. As a consequence, at a fixed learning rate the achieved policy deviation is non-monotone in $β$: it vanishes in a dead zone at small $β$, reaches a peak at an intermediate value, and decreases again for larger $β$. Moreover, standard DPO loss values are not comparable across $β$: runs with nearly identical loss curves can differ several-fold in KL divergence from the reference model. This entanglement obscures the role of $β$, increases sensitivity to hyperparameter choices, and complicates learning-rate scheduling. We propose a centered-softplus reformulation that is argmin-equivalent to DPO for $β>0$, while making the inverse preference-noise-scale and learning-rate effects explicit and independently tunable. The normalized centered-softplus objective also admits a continuous $β\to0$ endpoint that reduces to a linear preference-margin objective.

1 Introduction

The paper argues that DPO’s β simultaneously controls preference-noise scaling and optimization update magnitude, contrary to treating it only as a preference–KL trade-off parameter. This entanglement produces weak updates at small β, obscures loss comparisons across β, and motivates a centered-softplus reformulation.

  • β controls both the preference–KL trade-off and the effective update scale, not only the regularization behavior.The paper identifies the optimizer-scale role as a previously unisolated aspect of standard DPO objectives.
  • Small β reduces gradient magnitude, leaving the policy close to the reference model instead of producing larger policy deviation.Under SGD, decreasing β is equivalent to lowering the learning rate, complicating hyperparameter tuning.
  • Standard DPO loss values are not comparable across β because the loss depends on β and the preference margin through their product.Runs with nearly identical loss curves can nevertheless differ several-fold in KL divergence from the reference model.
  • At fixed learning rate, policy deviation is non-monotone in β: it enters a small-β dead zone, peaks near βpeak ≈4.6τ, then decreases for larger β.Here τ denotes the effective stopping tolerance of optimization.
  • The centered-softplus loss preserves DPO’s optimum for β > 0 while exposing and decoupling preference-noise-scale and learning-rate effects.Its β = 0 endpoint becomes a linear preference-margin objective.

2 Background: KL-Regularized RLHF and DPO

DPO collapses KL-regularized RLHF’s reward-model fitting and policy optimization into a single objective over policy log-probabilities. Its coefficient β reflects the KL coefficient relative to preference annotator-noise scale, while DPO-family objectives commonly retain an explicit β-dependent gradient factor.

  • KL-regularized RLHF balances reward maximization against staying close to a reference policy using λ as the KL-regularization coefficient.
  • DPO collapses the two-stage reward-model-and-PPO pipeline into a single objective over policy log-probabilities.The formulation avoids separately training a reward model.
  • Under the Bradley–Terry model, β = λ/sa combines KL regularization λ with pairwise annotator-noise scale sa.The conventional unit-noise formulation sets sa = 1.
  • The standard DPO objective is equivalently LDPO(∆; β) = log(1 + exp(−β∆)), with σ denoting the sigmoid function.
  • Most DPO variants retain an explicit multiplicative β factor in the marginal gradient, so reducing β suppresses update magnitude.The same coupling is discussed for unpaired KTO, reference-free SimPO, and token-level TDPO, while IPO, ROPO, and AOT depart from the standard pattern.

3 Gradient-Scale Coupling and Loss Incomparability in Standard DPO

Standard DPO’s β simultaneously scales preference gradients and shapes saturation, so fixed-learning-rate policy deviation becomes non-monotone and loss values become incomparable across β.

  • Dead Zone, Peak, and Saturation: Figure 1 reports final validation KL of 0.83 per token at β = 2 × 10−3 versus 0.54 at β = 10−2, violating a monotone KL-control reading.The β = 0.002 run overtakes β = 0.01 after epoch 3 in both gradient norm and KL.
  • Gradient-Scale Coupling: 20× β sweeps show initial gradient ratios G0/β remain constant within roughly 2%, confirming linear gradient suppression.This result comes from HelpSteer3 SGD runs summarized in Table 2.
  • Gradient-Scale Coupling: β directly multiplies the marginal gradient, suppressing update magnitude as β decreases and coupling preference scale to effective learning speed.At initialization, the full parameter-gradient norm is proportional to β.
  • Dead Zone, Peak, and Saturation: At fixed learning rate, final policy deviation has a dead zone at small β, peaks at an intermediate β, and decreases again at larger β.The stopping-tolerance model attributes these regimes to βσ(−β∆), including logarithmic margin growth and right-hand saturation.
  • Dead Zone, Peak, and Saturation: Figure 2 finds validation NLL, per-token KL, and mean margin follow the predicted dead-zone, intermediate-peak, and decaying-right-slope shape.Two descriptive τ estimates are approximately 5.99 × 10−4 and 6.37 × 10−4, with nearly indistinguishable overlays.
  • Loss Incomparability: Because standard DPO loss depends on β and margin only through u = β∆, nearly identical loss curves can correspond to policies differing several-fold in KL.Therefore neither training nor validation DPO loss supports comparing runs or selecting β by loss value.

4 Normalized Objective Formulation

The normalized centered-softplus objective removes β’s direct gradient scaling while preserving DPO minimizers for β > 0 and providing a continuous linear β = 0 endpoint.

  • 4.1 Normalized DPO Loss: Dividing the standard softplus loss by β > 0 preserves its argmin while removing the divergent log(2)/β offset.The centered-softplus transformation supplies the corresponding reformulation.
  • 4.1 Normalized DPO Loss: In the normalized parameterization, β controls sigmoid shape and saturation but no longer directly rescales local gradient magnitude.β instead bounds curvature by β/4.
  • 4.2 Centered Softplus: For β > 0, centered softplus is argmin-equivalent to −log σ(β∆), while β → 0 yields sc(x; 0) = x/2.The β = 0 branch therefore extends the objective continuously.
  • 4.2 Centered Softplus: At β = 0, the objective becomes linear preference-margin maximization and does not saturate for already separated pairs.Its constant marginal gradient means it does not by itself impose a finite preferred margin.
  • 4.2 Centered Softplus: Unlike standard DPO, normalized centered softplus avoids small-β collapse toward a constant and maintains gradients over a margin range growing as 1/β.Its lower plateau is −ln 2/β and its marginal gradient stays near −1/2 across that range.
  • 4.2 Centered Softplus: Under the normalized loss, achieved policy deviation is monotone in 1/β, without the classical dead zone or intermediate peak.The saturation margin scales as 1/β.
  • 4.3 Experiments: With lrnorm = β lrstandard, SGD validation trajectories are nearly equivalent, while normalized losses separate runs that standard losses make nearly indistinguishable.The experiment tests scale equivalence rather than superiority over a retuned DPO baseline.
  • 4.3 Experiments: Across additional settings and datasets with AdamW, stronger policy evolution corresponds to faster decreases in normalized training loss, including the β = 0 branch.Standard DPO loss can instead approach log 2 regardless of actual policy movement.

5 Discussion

The reformulation separates preference-noise scaling from optimization scaling without changing the DPO optimum for β > 0, while clarifying β’s statistical interpretation.

  • Discussion: The centered-softplus reformulation preserves the original DPO minimizer for β > 0 and explicitly decouples preference-noise scale from update scale.The paper does not claim a different optimum or universal gain over retuned DPO.
  • Discussion: DPO’s β = λ/sa combines the KL coefficient λ and annotator-noise scale sa, which are not separately identifiable without reward normalization.This follows from substituting the reward–policy relation into a logistic preference model.
  • Discussion: Subtracting ln 2 gives the negative log-likelihood ratio against random choice, while dividing by β removes the inverse preference-noise prefactor from the marginal gradient.The interpretation uses the Bradley–Terry Gumbel-noise view.

6 Conclusion

The paper recommends normalized centered softplus when β-dependent gradient scaling or cross-β loss comparability matters, presenting it as a practical loss rewrite rather than a universal performance upgrade.

  • Conclusion: The normalized objective resolves small-β gradient attenuation and makes preference-margin and validation pair-NLL changes monotone in β, equivalently in 1/β.Reported sweeps show validation KL co-moves with margin over a wide β range without claiming a universal KL–margin law.
  • Conclusion: The reformulation is a local loss-evaluation change that can be integrated into existing DPO codebases without restructuring the trainer or data pipeline.The paper frames this as an implementation property, not a performance guarantee.

Reproducibility, Data, and Compute

The experiments use multiple language-model backbones, public pairwise preference datasets, and LoRA fine-tuning, with additional downstream instruction-following checks.

  • Data and models: Experiments train Qwen3-4B-Instruct-2507, Qwen2.5-3B-Instruct, Ministral-3B-Instruct, and Mamba-2 2.7B with LoRA.
  • Datasets: Training uses HelpSteer3-Preference, UltraFeedback Binarized, and PKU-processed HH-RLHF from public Hugging Face datasets.
  • Evaluation: Downstream checks use Qwen2.5-14B-Instruct as an LLM judge for AlpacaEval 2 and the official verifiable IFEval benchmark without a judge model.
  • Compute and protocols: All experiments run on NVIDIA A100-SXM4-80GB GPUs, with processing, splits, hyperparameters, KL protocols, and training times documented.

Limitations and Broader Impacts

The paper’s scope is limited by moderate-scale LoRA experiments and optimizer- and implementation-dependent trajectories, while its broader impacts are those of language-model alignment generally.

  • Limitations: Empirical evidence covers three preference datasets and four moderate-scale LoRA backbones, leaving larger models, full fine-tuning, and other data regimes unverified.
  • Limitations: Argmin equivalence for β > 0 does not imply identical finite-time trajectories across optimizers and implementations.
  • Broader impacts: The work reformulates an existing preference-optimization objective rather than introducing a new capability class.
  • Broader impacts: Potential risks include misuse of better-aligned systems and biases inherited from preference data and annotator judgments.
  • Relation to variants: Across DPO-family objectives, β commonly remains an explicit gradient-scale factor, although some variants mediate its effect through weighting or alter the coupling.
  • Optimizer dependence: Adam can fully compensate for β scaling in one regime but leaves updates proportional to β in another, with the transition characterized by βcrit.
  • Threshold scope: βcrit depends on observed gradient statistics, model architecture, dataset, trainable-parameter count, and training stage.
  • Training-stage dependence: At initialization, Adam-related suppression is consequential, but thresholds generally fall as gradient scales grow during training.

C Additional Training Dynamics

Additional training dynamics reproduce β-dependent optimization effects across datasets, models, optimizers, and preference-noise settings, while showing stage-dependent compensation and link-function differences.

  • UltraFeedback: UltraFeedback experiments compare β ∈{10−8, 7 × 10−7, 10−6, 10−3} with AdamW, learning rate 5 × 10−7, batch size 16, and five seeds.
  • Cross-dataset replication: The same training and validation dynamics appear on PKU-processed HH-RLHF with Mamba-2.
  • Small-β behavior: Figure C3 shows that β ∈{10−8, 10−6} keeps gradients low and validation KL near zero, whereas β = 10−3 moves appreciably.
  • SGD dynamics: Figure C5 compares HH-RLHF SGD trajectories for β ∈ {0.001, 0.002, 0.005, 0.01, 0.02}, where smaller-β runs can show delayed gradient increases.
  • Mechanism: A smaller β shrinks the gradient norm directly, while stage-dependent growth of the non-β factor can partially offset that reduction.
  • Non-monotone dynamics: For sufficiently small β, delayed compensation remains weaker than early attenuation, producing non-monotone dynamics across practical tuning ranges.
  • General link functions: The general-link analysis shows that any pairwise loss −log F(β∆) has an explicit β gradient prefactor, including the Thurstone case F = Φ.
  • Thurstone comparison: The Thurstone loss grows quadratically for large negative margins, unlike the logistic branch’s linear growth.

E Formal Proof of Proposition 1 and Relation to Prior Work

The centered-softplus reformulation preserves DPO’s optimizer for β > 0, extends continuously to β = 0, and approaches a hard-margin objective as β grows.

  • Proposition 1: The paper distinguishes its contribution by applying centered-softplus as a loss normalization that preserves the DPO minimizer.
  • Proposition 1: Subtracting ln 2/β from normalized softplus changes only the vertical offset and leaves the minimizer unchanged.
  • Proposition 1: For β > 0, the normalized and original objectives induce the same total order on parameter space and therefore the same argmin.
  • β = 0 endpoint: At β = 0, centered-softplus has value −∆/2 and gradient −1/2, with gradient continuity from the admissible side β ≥0.
  • Hard-margin limit: As β →∞, normalized centered-softplus converges to max{−∆, 0}, the hard hinge boundary with decision boundary ∆= 0.

F Bradley–Terry Scale and Log-Likelihood-Ratio View

The Bradley–Terry view identifies β as an effective inverse preference-noise scale in policy-log-ratio coordinates, while standard DPO also couples β to optimization scale. Centered-softplus preserves the statistical role while decoupling optimizer scaling.

  • Bradley–Terry scale: β = λ/sa is the effective inverse preference-noise scale, combining KL/reward scale λ with annotator-noise scale sa.The DPO likelihood identifies their ratio rather than λ and sa separately.
  • Log-likelihood-ratio view: The DPO loss models an observed preference with probability σ(β∆), where ∆ is the policy’s log-ratio gap relative to the reference.The product β∆ acts as a scale-controlled evidence score for the observed preference.
  • Normalization: Subtracting the chance baseline and dividing by β removes the direct β prefactor from the marginal gradient.The normalized quantity equals annotator-noise units only under a fixed reward/KL normalization.
  • Disentangled roles: Standard DPO gives β two roles: statistical inverse scale β = λ/sa and an unintended optimization-scale gradient multiplier.The latter couples β to effective step size.
  • Disentangled roles: Centered-softplus removes the optimization-scale role while preserving the statistical coefficient, allowing the two effects to be tuned independently.Separating λ from sa still requires external normalization or additional information.

G Saturation Thresholds: Where Training Stops as a Function of β

A fixed-learning-rate threshold model explains why standard DPO training stops at a β-dependent margin: small β can create a dead zone, intermediate β a maximum, and large β declining reachable margins. The model is an approximate saturation description whose empirical interpretation requires care.

  • Standard DPO regimes: For β > 2τ, standard DPO’s saturation margin follows β ∆∗DPO = log(β/τ − 1), so the product grows logarithmically with β.This differs from the normalized objective, where β∆∗ is constant and ∆∗ is monotone in 1/β.
  • Standard DPO regimes: β ≤ 2τ defines a dead zone where the initial marginal gradient is below tolerance and training remains at initialization.The normalized objective has no corresponding dead zone because its initial marginal gradient is 1/2 for every β.
  • Peak location: βpeak ≈ 4.6τ is the unique maximum of the reachable margin under the fixed-learning-rate threshold model.The maximum is determined by the stopping tolerance τ.
  • Empirical illustration: In the HelpSteer3 SGD sweep, final per-token KL {0.04, 0.35, 0.83, 0.54} peaks at β = 2×10−3 among the four sampled values.On the dense grid, min NLL, max KL, and max mean ∆ occur between β = 2.5 × 10−3 and 3 × 10−3.
  • Estimating τ: The βpeak-based estimate of τ is primary, while fitting the β-shape of ∆ with a free vertical scale provides an internal consistency check.The two estimates are not out-of-sample validation because both use the same β-sweep.
  • Scope and caveats: The threshold model applies only when training reaches saturation before the epoch budget ends and treats τ as a coarse effective tolerance under decaying learning rates.Very small-β runs may stop early because updates are weak, rather than because the loss has saturated.

H Loss-Level Scale Invariance: Identical Loss Curves, Different Policies

Standard DPO loss depends on β and the preference margin only through u = β∆, so matched scale-equivalent runs can show nearly identical losses while producing different policy deviations and KL values.

  • Empirical comparisons: On UltraFeedback Binarized, validation DPO losses stay within 0.009 nats while the KL ratio stabilizes near 6.5 under matched ηβ.
  • Empirical comparisons: On HH-RLHF, matched ηβ produces validation-loss differences up to 0.03 nats while the KL ratio remains approximately 1.8–3.1.
  • Empirical comparisons: Validation losses coincide to the third decimal from epoch 1 onward on UltraFeedback, while the KL ratio remains near 3.3–3.9.
  • Empirical comparisons: From epoch 1 onward, four-seed UltraFeedback validation losses differ by at most 0.005 nats while the KL ratio stays near 2.9–4.1.
  • Dynamics: Matching ηβ can produce a common late-time u trajectory, allowing losses to converge even though ∆ = u/β and KL remain different.The catch-up pattern succeeds on UltraFeedback, HelpSteer3, and Ministral-3B but does not fully occur on HH-RLHF.
  • Structural invariance: The standard pairwise loss is invariant under β∆, making its level sets hyperbolas and obscuring physical margin and KL differences.For example, (β, ∆) = (0.01, 100) and (0.02, 50) have identical loss despite a twofold difference in policy displacement.

J Logged Gradients, Mean-Abs Magnitudes, and the Jacobian Factor g

The gradient analysis separates explicit β scaling, sigmoid saturation, and the margin Jacobian, while distinguishing mean-absolute logs from Euclidean gradient norms and cautioning against overinterpreting minibatch proxies.

  • Minibatch interpretation: The minibatch proxy gB represents a coherence-weighted norm of the batch-averaged margin Jacobian rather than a typical per-pair Jacobian norm.Cross-pair alignment and cancellation can make gB differ substantially from individual Jacobian norms.
  • Logging conventions: The logged mean-absolute magnitude G and Euclidean norm differ by a near-constant shape factor, with r ≈2.8–3.1 on HelpSteer3 and r ≈3.7–4.1 on UltraFeedback.
  • Logging conventions: TRL reports Euclidean ∥∇θL∥2, whereas some published SGD curves retain mean-absolute G despite axes labelled ∥∇L∥, creating a residual logging inconsistency.
  • Gradient decomposition: For one preference pair, the DPO parameter gradient factors into an explicit β term, a sigmoid saturation term, and the margin Jacobian.This decomposition explains why a falling loss-gradient magnitude need not imply that the Jacobian itself is shrinking.
  • Horizon estimate: The scalar horizon estimate is only an order-of-magnitude guide because Jacobian variation, cross-pair cancellation, and stochasticity prevent a precise cutoff.The highlighted βr = 0.027 value is described as a closed-form ceiling or characteristic scale, not a calibrated threshold.

K.4 Numerical estimate on HelpSteer3

On the HelpSteer3 sweep, the scalar horizon estimate places the right-hand crossover near βr ≈0.027, but trajectory-dependent minibatch dynamics make it a broad practical scale rather than a sharp boundary.

  • Numerical estimate: Using Tbudget = 9078, ηmax = 2 × 10−4, H = 0.9078, τ = 6.37 × 10−4, and gB(θ0) ≈250 yields a rough βr estimate.
  • Uncertainty: Replacing the initial proxy with the later reconstructed scale gB ≈360–370 moves βr downward by about a factor of two, to roughly 0.012–0.013.
  • Numerical estimate: The empirical βr row has NLL ≈12.88, KL ≈0.221, and mean validation margin ∆≈38.5, interpolated between β = 0.02 and β = 0.04.
  • Numerical estimate: βr ≈0.027 places the right-hand crossover within the observed logarithmic range of the HelpSteer3 sweep.
  • Operational interpretation: The inferred τ depends on observation resolution and can also absorb finite-horizon and learning-rate-decay effects, so a plateau alone may not identify it.
  • Operational interpretation: The stochastic appendix treats τ as a drift–noise crossover estimated from aligned trajectories, without assuming convergence to an equilibrium distribution.
Loading 2608.27032v1…