PaperlayerView exact revision on arXiv

Source-linked AI summary

Exploring the Limits of the Delta Learning Hypothesis: Single-Pair Generator Biases in Direct Preference Optimization

Giuseppe Curci, Stefano Simonazzi, Andrea Molinari, Andrea Zugarini

arXiv:2610.37058v1cs.LGcs.AIcs.CL

TL;DR

DPO preference data often depends on external LLM judges, while delta learning proposes using a single pair of models without a judge. This paper studies that approach across model families and scales, finding systematic generator-dependent biases and underperformance that generator diversity alone partially mitigates.

  • Problem

    Preference data often uses LLM judges whose rankings can be unstable, while evidence about how single-pair delta learning generalizes across model families and scales is limited.

    Source: §1 · p. 1, §4 · p. 8

  • Method

    The paper theoretically analyzes and empirically compares single-pair and multi-model preference data, including whether generator diversity without an external judge mitigates single-pair effects.

    Source: §1 · p. 2, §3.4 · p. 7

  • Results

    Single-pair data underperforms across generator and training model families and sizes, with generator-dependent biases; increasing generator diversity improves performance over the single-pair baseline for most tested models.

    Source: §1 · p. 2, §4 · p. 8

  • Takeaways & Limitations

    Generator diversity is a critical factor in multi-model preference data, and increasing it provides a better judge-free baseline for constructing preference pairs from a model pool.

    Source: §1 · p. 2, §4 · p. 8

    The largest training models considered have 8 billion parameters, so whether the findings hold at larger scales remains to be validated.

    Source: Limitations · pp. 8–9

Abstract

from arXiv · show

Direct Preference Optimization (DPO) typically relies on preference pairs generated from diverse model pools and ranked by an external LLM judge. The recently proposed delta-learning hypothesis suggests that comparable supervision can instead be obtained from a single pair of models by treating the larger model's response as preferred. We investigate the limits of this approach across model families and scales. We show, both theoretically and empirically, that fixed model pairs introduce systematic, generator-dependent biases that are predictive of the preference label independent of response quality. Consequently, DPO trained on single-pair data consistently underperforms multi-model preference data, despite the larger generator producing higher-quality responses, and we trace this gap to training dynamics consistent with reward hacking on these biases. We further show that simply increasing the number of generators, even without an external judge, is enough to partially mitigate this effect. Our results highlight generator diversity as a critical factor in preference-data construction.

1 Introduction

DPO trains on chosen and rejected completions, often ranked by an LLM judge, but judge preferences can be unstable. This section asks whether the single-pair delta-learning approach generalizes across model families and scales, and reports systematic biases and performance gaps.

1 Introduction

  • LLM judges can favor superficial properties such as length, formatting, or judge-generator similarity, contributing to ranking instability.Prior work has explored bias mitigation, data-aware training, data cleaning, and alignment evaluation.Source: §1 · p. 1
  • The delta-learning hypothesis uses a larger model's completion as chosen and a smaller model's as rejected, without external supervision, but its generalization across families and scales remains uncertain.Earlier work reports comparable improvements when the quality delta is sufficient and noise is minimal.Source: §1 · pp. 1–2
  • The paper asks whether noise in single-pair supervision is occasional correctness error or systematic generator-dependent bias that persists across scales and dataset sizes.Source: §1 · p. 2
  • Single-pair DPO systematically underperforms multi-model preference data; the authors link the gap to generator-dependent spurious features and show generator diversity alone partially mitigates it.Their analysis combines theory, reward-accuracy training dynamics, and classification using stylistic features alone.Source: §1 · p. 2

2 Methodology

The methodology defines the DPO objective and formalizes how generator-dependent spurious features can predict preference labels, then describes the data, training, model, and benchmark choices used to study these effects.

2.1 Direct Preference Optimization

  • Length-normalized DPO optimizes chosen-versus-rejected completion probabilities using a policy model, reference model, and scaling hyperparameter β.The loss uses the logistic sigmoid and completion token lengths.Source: §2.1 · p. 2

2.2 Delta Learning Hypothesis Limits

  • The delta-learning hypothesis says training succeeds when a stronger teacher is better aligned with optimal parameters than a weaker teacher, provided spurious noise does not overwhelm correctness.The paper extends noise to systemic dataset-wide biases that can harm stability and downstream performance even when the stronger teacher is better.Source: §2.2 · p. 2
  • With a fixed generator pair, a spurious feature’s propensity gap predicts how accurately a feature-only classifier can recover preference labels.The relationship is monotonic: a larger gap corresponds to higher classifier accuracy.Source: §2.2 · pp. 2–3
  • Fixed-pair propensity gaps persist within prompt domains, so aggregate cancellation does not prevent domain-specific surface features from separating chosen and rejected completions.The paper notes that such domain-specific stylistic propensities can yield a very strong classifier.Source: §2.2 · p. 3
  • Pooling can attenuate bias when generator identity varies within domains, making the propensity gap vary across examples so no single feature threshold consistently predicts labels.The stated condition is that the spurious feature is uncorrelated with the features guiding selection; population-level correlations can leave residual bias.Source: §2.2 · p. 3
  • Pooling does not necessarily remove bias when desired and spurious features correlate across the model population, and the paper leaves this confound’s full characterization to future work.For example, stronger models may write longer answers on average.Source: §2.2 · pp. 3–4

2.3.1 Training

  • The study adapts a multi-model preference dataset, regenerates completions with models of varying sizes and families, and assigns chosen/rejected labels strictly by generator size.Training uses 1,000 steps with batch size 128; hyperparameters are selected using downstream benchmark performance on a separate evaluation set.Source: §2.3.1 · p. 4

2.3.2 Models and Evaluation

  • The evaluation design seeks capable, diverse models and benchmarks spanning the original dataset’s relevant capabilities and a broader range of skills.Generator selection tests preliminary QWEN3 models, robustness across families and scales, and larger pools; benchmarks include TruthfulQA, ARC-Challenge, IFEval, Math-500, GSM8K, Human-Eval, and MBPP.Source: §2.3.2 · p. 4

3 Results

Across model families and sizes, DPO trained on single-pair data shows lower downstream performance and greater instability than multi-model preference data. The analysis links this gap to generator-dependent shortcuts, while greater generator diversity partially mitigates it without an external judge.

3.1 Delta Learning doesn’t match LLM-as-a-judge

  • Across model families and sizes, multi-model preference data generally improves benchmark performance, while the QWEN3-4B/QWEN3-0.6B pair often degrades or lags the original dataset.The single pair degrades performance on about half the benchmarks; on the rest, it is unchanged or substantially behind the original dataset.Source: §3 · pp. 4–5, Figure 1 · p. 5, §3.1 · p. 5
  • Across four additional generator pairs, scores vary slightly, but the average remains much lower than with the original dataset.The authors interpret this as evidence of an intrinsic problem in the generated completions, rather than an effect of one specific pair.Source: Figure 2 · p. 5, §3.1 · p. 5

3.2 Does data-model alignment matter?

  • Training on the original dataset yields strong improvements across benchmarks, whereas training with a single model pair is unstable and degrades on average.This comparison tests whether the performance gap is simply due to a mismatch between the prompt distribution and the tested models.Source: §3.2 · pp. 5–6, Table 1 · p. 6

3.3 Spurious Features Analysis

  • With fewer model pairs, spurious features are more likely to emerge; empirically, these biases are especially pronounced on some tasks.The analysis examines generated data and training dynamics to investigate this mechanism.Source: §3.3 · p. 6

3.3.1 Training Dynamics

  • Reward accuracy differs by approximately 27% between single-pair and multi-model settings, consistent with reward hacking that hinders training improvement.With prompt distribution and underlying response-quality differences held fixed, reward metrics can reveal whether training exploits generator-dependent signals instead of learning the intended preference.Source: §3.3.1 · p. 6, Figure 3 · p. 6

3.3.2 Length Bias

  • Length disparity offers a shortcut for preference learning in Qwen and Gemma pairs, but this pattern does not hold for Olmo2 pairs.The Olmo2 results motivate examining less obvious biases.Source: §3.3.2 · p. 6, Table 2 · p. 6

3.3.3 Stylistic Bias

  • Style features separate chosen from rejected completions: logistic regression reaches 83% and 100% AUROC across the two datasets.The first two principal components explain 51.8% and 57.1% of variance, respectively; directly intervening on such features is impractical when they vary across domains and cannot be enumerated in advance.Source: §3.3.3 · p. 7, Figure 4 · p. 7

3.4 Increasing Generators Diversity

  • Increasing generator diversity improves average scores for three of four trained models, although the gains remain below judge-supervised data and are not uniform.Without an external judge, the method samples from nine models grouped into performance tiers; the reported results include persistent IFEval gains, while Mistral’s average does not improve.Source: §3.4 · pp. 7–8

4 Conclusion

The two-model approach consistently underperforms across generator and training model families and sizes, while generator-dependent biases can dominate preference signals. Increasing generator diversity without an external judge improves performance for most tested models.

4 Conclusion

  • The study compares the standard multi-model, judge-ranked strategy with deterministic preferences based on relative size within a two-model family.The conclusion characterizes generator diversity as critical to the effectiveness of multi-model preference data.Source: §4 · p. 8
  • The two-model approach consistently underperforms across generator and training model families and sizes.The paper identifies systematic generator-dependent biases as consistent with this degradation.Source: §4 · p. 8
  • Figure 5 compares relative benchmark score changes for multiple-model data with a judge, average single pairs, and multiple-model data using the delta heuristic.Dotted lines indicate average performance.Source: Figure 5 · p. 8
  • Chosen and rejected completions from a single pair can, in some settings, be separated almost perfectly using stylistic features alone.These generator-dependent biases can dominate the preference signal and hinder useful learning.Source: §4 · p. 8
  • Generator diversity without an external judge improves performance over the single-pair baseline for most tested models.Because parameter size no longer predicts completion quality across families, diverse pairs use coarse benchmark-derived tiers.Source: §4 · p. 8

Limitations

The largest training models considered have 8 billion parameters, and validating whether the findings hold at larger scales remains future work.

Limitations

  • The largest training models considered have 8 billion parameters, leaving validation at larger scales for future work.The theoretical results are agnostic to model scale, and the authors expect the mechanism to hold at larger scales.Source: Limitations · pp. 8–9

A Empirical Analysis of Style Bias

The style-bias analysis represents completions with formatting features and tests whether chosen and rejected responses are linearly separable using those features alone.

A Empirical Analysis of Style Bias

B Evaluation Details

The evaluation uses code equivalent to lm-evaluation-harness for the considered benchmarks and reports benchmark-specific settings for reproducibility.

B Evaluation Details

  • The evaluation code is equivalent to lm-evaluation-harness for the benchmarks considered.Table 4 reports the exact settings for each benchmark to facilitate reproducibility.Source: B Evaluation Details · p. 11

C Additional Results

This section maps the appendix’s full evaluation results to Tables 5–8, covering initial experiments, pair ablations, and comparisons using multiple generator pairs.

C Additional Results

  • Tables 5–8 provide the exact evaluation results for experiments discussed in the main paper, spanning initial results, pair ablations, and multiple-pair comparisons.Table 5 reports QWEN3-4B and QWEN3-0.6B results; Table 6 ablates the completion-generating pairs.Source: C Additional Results · p. 11
  • Table 7 reports OLMO2-1B-SFT experiments, addressing whether the findings reflect alignment between the data and the model it was designed for.The passage says these results rule out that possibility.Source: C Additional Results · p. 11
  • Table 8 compares the original dataset, the average across individual size-heuristic pairs, and benchmark-selected multiple pairs.Multiple-pair results are divided into FEW PAIRS, using previously considered models, and MORE PAIRS, adding other models.Source: C Additional Results · p. 11

D Data Generation

The data-generation section outlines how preference datasets are constructed and specifies shared generation settings. Truncated completions are discarded to avoid truncation bias.

D Data Generation

  • The section provides an overview of data generation and explains how the datasets are constructed.It introduces the generation settings in subsection D.1.Source: D Data Generation · p. 11

D.1 Settings

  • All data are generated using vLLM with top_p=0.9, temperature=0.2, thinking disabled, and max_new_tokens=2048.Other generation parameters use their defaults.Source: §D.1 · p. 11
  • The same generation parameters are used across models, and truncated completions are discarded to avoid truncation bias.Source: §D.1 · p. 11

D.2 Models Generator

For Section 3.4, the authors construct preference pairs from model generators grouped by capability. They compare a size-based few-pair pool with benchmark-informed groups and sample across groups to diversify generators.

D.2 Models Generator

  • For FEW PAIRS, chosen completions come from QWEN3-32B, OLMO2-32B-INSTRUCT, or GEMMA2-27B-IT, while rejected completions come from four smaller models.The rejected set is QWEN3-0.6B, OLMO2-1B-INSTRUCT, GEMMA2-9B-IT, and OLMO2-7B-INSTRUCT.Source: §D.2 · p. 11
  • The authors expand the generator pool because parameter count does not always track completion quality, and use Math, Instruction-Following, and Coding benchmarks as quality proxies.They note that GEMMA2-2B-IT significantly outperforms QWEN1.5-4B-CHAT despite having half the parameters.Source: §D.2 · p. 11
  • Models are partitioned into candidate groups, with Ck denoting chosen generators and Rk denoting rejected generators.The groups represent capability tiers, including maximum, high-quality subtle, and moderate deltas.Source: §D.2 · pp. 11–12
  • Table 3 describes the detection criteria for the 11-dimensional style representation, and Table 4 overviews the evaluation benchmarks.Table 4 benchmarks use multi-turn chat; CoT and few-shot examples are included as conversation turns where specified.Source: Table 3 · p. 12, Table 4 · p. 12
  • For each prompt, the procedure uniformly samples a group and then chosen and rejected generators from that group, diversifying the data to make reliance on spurious features harder.Source: §D.2 · p. 12

E More Training Details

The training details describe the software, hardware, fixed optimization setup, and learning-rate searches. Generated-completion settings required extending the search toward lower learning rates because higher rates often degraded performance.

E More Training Details

  • All experiments use olmo2-1b-prefmix as the baseline, replace its completions while keeping the prompt distribution fixed, and train for 1,000 steps with an effective batch size of 128.The pipeline uses Transformers TRL, DeepSpeed ZeRO, and one node with 8 H100 GPUs; the shared configuration supports comparisons across settings.Source: E More Training Details · p. 12
  • Tables 5–8 report benchmark deltas for initial generator pairs, pair ablations, OLMO2-1B-SFT, and multiple-pair settings; Table 9 evaluates generators across Math, Instruction-Following, and Code benchmarks.Source: Table 5 · p. 13, Table 6 · p. 13, Table 7 · p. 14
  • For generated completions, the learning-rate grid is {5×10−7, 1×10−7, 5×10−8, 1×10−8}, after higher rates often degraded performance.The authors report that 1×10−6 sometimes pushed performance below the SFT baseline; the original dataset grid was {1 × 10−6, 5 × 10−7, 1 × 10−7}.Source: E More Training Details · pp. 12, 15

Related papers

Loading 2610.37058v1…