PaperlayerView exact revision on arXiv

Source-linked AI summary

The Delta Learning Hypothesis: Preference Tuning on Weak Data can Yield Strong Gains

Scott Geng, Hamish Ivison, Chun-Liang Li, Maarten Sap, Jerry Li, Ranjay Krishna, Pang Wei Koh

arXiv:2507.06187v1cs.AI

TL;DR

Strong supervision can be costly or unavailable, raising the question of how models might improve beyond the capabilities shown in their training data. The paper tests preference tuning on pairs of weak responses, using their relative quality difference as the learning signal. Its simple small-model recipe matches T¨ulu 3 on an 11-benchmark suite, and a logistic-regression proof analyzes how weak teachers’ performance gaps can help a stronger student.

  • Problem

    Strong data can be costly to collect or unavailable for tasks beyond current human expertise, limiting the supervision available for training language models.

    Source: §1 · p. 2

  • Method

    The paper studies preference tuning on paired responses whose relative quality differs, and tests a small-model recipe that pairs chosen responses with outputs from an even smaller model.

    Source: §1 · p. 2

  • Results

    The small-model recipe matches T¨ulu 3 on an 11-benchmark suite despite using much less supervision, and logistic regression shows that the performance gap between two weak teachers can provide directionally correct signal for improving a stronger student.

    Source: §1 · p. 2

  • Takeaways & Limitations

    Pairing weak data to expose informative deltas may offer a more scalable alternative when strong supervision is limited, and simple model-size differences can capture such deltas.

    Source: §1 · p. 2

    Not all positive deltas drive learning, and what makes a delta informative beyond its magnitude—and how task and tuning algorithm affect the dynamics—remains open.

    Source: §5.1 · p. 8, §8 · p. 15

Abstract

from arXiv · show

Improvements in language models are often driven by improving the quality of the data we train them on, which can be limiting when strong supervision is scarce. In this work, we show that paired preference data consisting of individually weak data points can enable gains beyond the strength of each individual data point. We formulate the delta learning hypothesis to explain this phenomenon, positing that the relative quality delta between points suffices to drive learning via preference tuning--even when supervised finetuning on the weak data hurts. We validate our hypothesis in controlled experiments and at scale, where we post-train 8B models on preference data generated by pairing a small 3B model's responses with outputs from an even smaller 1.5B model to create a meaningful delta. Strikingly, on a standard 11-benchmark evaluation suite (MATH, MMLU, etc.), our simple recipe matches the performance of Tulu 3, a state-of-the-art open model tuned from the same base model while relying on much stronger supervisors (e.g., GPT-4o). Thus, delta learning enables simpler and cheaper open recipes for state-of-the-art post-training. To better understand delta learning, we prove in logistic regression that the performance gap between two weak teacher models provides useful signal for improving a stronger student. Overall, our work shows that models can learn surprisingly well from paired data that might typically be considered weak.

1 Introduction

Strong supervision can be costly or unavailable, motivating the question of whether models can improve beyond the capabilities shown in their training data. The paper proposes learning from quality differences between paired weak responses and tests this idea empirically, at scale, and theoretically.

1 Introduction

  • Strong data can be costly to collect or unavailable for tasks beyond current human expertise, limiting the supervision available for training.The paper frames this as a motivation for asking how models might exceed capabilities demonstrated in their training data.Source: §1 · p. 2
  • Paired weak responses can improve a stronger language model beyond each sample’s quality, even when supervised finetuning on those responses degrades performance.The paper calls the relative quality difference between paired responses the “delta” and proposes it as a source of preference-tuning supervision.Source: §1 · p. 2
  • On an 11-benchmark suite, a recipe using a small model’s chosen responses and an even smaller model’s responses matches T¨ulu 3’s performance.The approach uses much less supervision than the T¨ulu 3 recipe, which relies on stronger supervisors such as GPT-4o.Source: §1 · p. 2
  • A logistic-regression proof shows that the performance gap between two weak teachers can provide directionally correct signal for improving a stronger student.The result applies even when both teachers individually provide misleading supervision.Source: §1 · p. 2
  • The authors argue that simple heuristics such as model-size differences can expose useful deltas and that pairing weak data may offer a more scalable alternative when strong supervision is limited.They suggest targeted corruptions of existing data or lightweight human edits as possible ways to create pairs.Source: §1 · pp. 2–3

2 A Warm-up Case Study

The warm-up case study tests whether weak preference data can improve Llama models when the chosen responses come from models weaker than Llama 3. Preference tuning improves performance overall, whereas supervised finetuning on the same chosen responses hurts it.

Data

  • ULTRAFEEDBACK-WEAK contains preference pairs whose chosen responses come from models weaker than Llama-3.2-3B-Instruct but remain higher quality than the rejected responses.The dataset is formed by filtering ULTRAFEEDBACK to exclude responses from models with Arena ELO scores near or above Llama-3.2-3B-Instruct.Source: §2 · p. 3

Training and evaluation

  • The study compares DPO on preference pairs with SFT on chosen responses, evaluating Llama-3.2-3B-Instruct and Llama-3.1-8B-Instruct on eight standard benchmarks.The benchmarks cover knowledge recall, mathematical reasoning, instruction following, truthfulness, general reasoning, and coding.Source: §2 · p. 3

Results

  • Preference tuning on ULTRAFEEDBACK-WEAK improves overall performance across benchmarks, while SFT on the chosen responses significantly hurts performance.The authors suggest SFT may hurt because it trains models to imitate weaker outputs.Source: §2 · p. 3

3 The Delta Learning Hypothesis

The delta learning hypothesis proposes that preference tuning can improve a model beyond the quality of a chosen response when the pair contains an informative relative-quality difference. Controlled experiments test this idea with both a measurable style feature and semantic responses from a weaker model.

3 The Delta Learning Hypothesis

  • The hypothesis holds that preference tuning can extrapolate beyond a chosen response’s utility when it is no better than the model’s current capability but outranks its paired response.SFT on the chosen response under this condition hurts or does not help the model.Source: §3 · p. 3

3.1 Controlled Experiment: Stylistic Delta in Number of Bold Sections

  • When chosen responses contain fewer than the baseline 5.9 bold sections, SFT decreases section generation, but preference tuning can increase it using the pairwise delta.Positive-delta tuning can extrapolate beyond the chosen response; negative- and zero-delta controls do not yield gains.Source: Table 2 · p. 4, §3.1 · p. 4

3.2 Controlled Experiment: Semantic Delta from a Weaker Model

  • The semantic experiment pairs each model’s greedy responses with outputs from a smaller sibling, creating a noisy positive delta because some weaker-model responses can be better on individual prompts.Llama-3.1-8B-Instruct is tuned to prefer its own responses over Llama-3.2-3B-Instruct outputs and compared with SFT on its own responses.Source: §3.2 · pp. 4–5
  • Pairing self-generated responses with weaker responses creates a positive delta and yields a 0.4-point average gain; SFT on self-generated greedy responses reduces average performance by 1.2 points, while reversed preferences worsen overall performance by 0.7 points.Source: §3.2 · p. 5

4 Post-training Language Models with Delta Learning

The paper adapts the T¨ulu 3 preference-tuning recipe by replacing strong-model generation and judging with responses from small models paired by model size. Its best weak-data setup matches T¨ulu 3, while other weak datasets also yield gains.

4 Post-training Language Models with Delta Learning

  • The paper tests its simplified approach in realistic 8B post-training, aiming to reduce reliance on strong-model distillation while preserving performance.It keeps the T¨ulu-3-8B-SFT starting checkpoint and initial prompts fixed to isolate the recipe changes.Source: §4 · p. 5, §4.2 · p. 6

SOTA Open

  • The figure contrasts T¨ulu 3’s GPT-4o selection from strong-model responses with the proposed use of one small chosen-response generator and an even smaller paired model.The latter relies on an implicit quality delta to drive downstream learning.Source: Figure 2 · p. 6

4.1 A State-of-the-Art Existing Setup: The T¨ulu 3 Recipe

  • T¨ulu 3 constructs preference data from 271k prompts using strong-model responses and GPT-4o to select chosen and rejected outputs.Its recipe incurs substantial GPT-4o annotation cost and assumes supervisors stronger than the 8B model being trained.Source: SOTA Open · p. 6

4.2 Our Simple Recipe: Constructing Preference Data without Strong Supervision

  • The simplified recipe pairs chosen responses from one small model with responses from the next-smallest model in the same family, using model size as a noisy quality proxy instead of GPT-4o annotations.The recipe generates chosen responses with a single small model and removes strong-model use from response generation and preference annotation.Source: §4.2 · p. 6

4.3 Matching T¨ulu with Delta Learning

  • The Qwen-2.5-3B-Instruct recipe achieves a +0.4 point average gain over T¨ulu 3 preference data, matching its performance despite using weaker supervision.The result is reported for tuning T¨ulu-3-8B-SFT with the simple weak preference data recipe.Source: Table 4 · p. 7, §4.3 · p. 7
  • A dataset using Qwen-2.5-1.5B-Instruct chosen responses, 11.4 points worse than T¨ulu-3-8B-SFT on average, still yields a +2.1 point average gain.The recipe also produces gains with either Llama or Qwen models as response generators.Source: §4.3 · p. 7

5 Analysis

The analysis studies how preference-tuning performance relates to quality delta and chosen-response quality, and tests the model-size heuristic and base-model choice. Delta predicts performance up to a saturation point, while preference tuning can gain from chosen responses no stronger than the base model.

5 Analysis

  • The analysis examines delta magnitude, chosen-response quality, the model-size preference heuristic, and the choice of T¨ulu-3-8B-SFT as the base model.Unless otherwise noted, models are tuned from T¨ulu-3-8B-SFT with DPO and evaluated on the same 11 benchmarks as Section 4.Source: §5 · p. 7

5.1 How does the magnitude of the chosen-rejected quality delta affect learning?

  • ∆≈0.55 marks the point beyond which increasing the chosen–rejected quality delta yields no additional downstream gains; T¨ulu 3 data follows the same trend.Both T¨ulu 3 data and Qwen-2.5-3B-Instruct-generated weak data have deltas above this threshold.Source: Figure 3 · p. 8, §5.1 · p. 8
  • The Qwen-2.5-1.5B chosen-response dataset gains less than its delta predicts; the authors conjecture this is because 1.5B is substantially weaker than the base.Source: §5.1 · p. 8
  • Positive delta is not sufficient: pairing Qwen 72B responses with 32B or 14B responses hurts performance, and what makes a delta effective remains open.The authors hypothesize that downweighting good behaviors may be harmful when both responses are much stronger than the base model.Source: §5.1 · p. 8

5.2 How does the chosen response’s absolute quality affect learning?

  • SFT gains occur only with responses from models stronger than T¨ulu-3-8B-SFT, and gains increase monotonically with chosen-response quality.Source: §5.2 · p. 9
  • Preference tuning yields gains with chosen responses of any quality, but gains are smaller for Qwen 1.5B responses and saturate once quality reaches the tuned model’s level.Further improvements in chosen-response quality do not significantly improve performance after that point.Source: §5.2 · p. 9

5.3 Ablations

  • The model-size preference heuristic agrees with GPT-4o preferences 80.5% of the time, and tuning with either label source yields comparable performance except on AlpacaEval 2.Source: §5.3 · p. 10
  • Using Qwen-2.5-3B and 1.5B responses to tune OLMo-2-7B-SFT matches the original OLMo 2 preference recipe, with +0.2 points average.The original recipe requires far stronger supervision.Source: §5.3 · p. 10

6 Delta Learning in Logistic Regression, Provably

The paper analyzes preference tuning in binary logistic regression, where a student learns to prefer pseudo-labels from a stronger teacher over those from a weaker teacher. It proves that, under conditions on the teacher-performance gap and other errors, this can improve the student with high probability.

6.1 Problem Setup and Notation

  • The analysis models realizable binary classification with isotropic Gaussian inputs and defines teacher strength by alignment with the ground-truth parameters.Preference pairs choose the stronger teacher’s pseudo-label over the weaker teacher’s, with no other assumptions on teacher or student strength.Source: §6.1 · pp. 10–11

6.2 Delta Learning Succeeds with High Probability

  • In high dimensions, most teacher pairs with a performance gap improve the student, even when its initial accuracy exceeds both teachers’.The result applies under the theorem’s conditions on the useful signal and harmful error alignment.Source: §6.2 · pp. 12–13
  • Improvement grows with the teachers’ performance delta but shrinks as the student’s initial performance rises.The expected improvement is on the order of Θ(κ^2); language-model gains can saturate as response-quality deltas grow.Source: §6.2 · p. 12
  • For an 80%-accurate student and 70%/60%-accurate teachers, d > 2000 suffices for at least 90% of such pairs to improve it.The example samples teacher pairs conditional on their accuracies.Source: §6.2 · p. 13
  • The proof reduces exact-gradient training to movement along a ray that improves alignment, then extends the result to mini-batch SGD using concentration analysis.With batch size B = Θ(d) and suitable learning-rate and step choices, SGD achieves at least Θ(κ^2) improvement.Source: §6.2 · pp. 13–14
  • The useful signal from the teacher gap must dominate harmful signals from errors that align with and amplify the student’s existing errors.The theorem’s condition allows learning even when the chosen pseudo-label is sometimes wrong, provided it is more correct on average than the rejected label.Source: §6.1 · p. 11, §6.2 · pp. 12–13

7 Related Work

The related-work discussion places delta learning alongside preference-tuning methods and weak-to-strong generalization. It distinguishes this work by formalizing, validating, and theoretically analyzing learning from relative differences in weak data.

Learning from preference feedback

  • Preference tuning evolved from human-ranked reward modeling and PPO to direct policy updates and strong LLM judges, while chosen responses often remain stronger than the learner’s current policy.The paper argues that learning can succeed even when the chosen response is weak.Source: §7 · p. 14

Weak-to-strong generalization

  • Yao et al. (2024) study wrong-over-wrong preferences, while Zhu et al. (2024) use distributional changes from aligning a smaller model to align a larger one.The paper frames both as preliminary evidence that preference tuning can exceed supervision quality, then formalizes and analyzes the delta-learning hypothesis.Source: §7 · p. 14
  • Prior weak-to-strong work elicited behavior from base models or iteratively improved training data; this paper instead uses relative differences in weak data to guide generalization.The paper identifies Yao et al. (2024) and Zhu et al. (2024) as the closest related works in this spirit.Source: §7 · p. 14

8 Conclusion

The conclusion says models can learn from deltas between paired weak data points, but emphasizes that delta usefulness varies. It leaves the conditions for informative deltas and their scalability and transfer unresolved.

8 Conclusion

  • Models can learn from the delta between paired weak data points.The conclusion also characterizes delta magnitude and chosen-response absolute quality as factors that drive learning.Source: §8 · p. 14
  • Not all deltas drive gains, and gains saturate as chosen-response quality improves.Source: §8 · pp. 14–15
  • The conclusion leaves open what makes a delta informative, how to scale delta-based learning, and how dynamics depend on task or tuning algorithm.Source: §8 · p. 15

9 Reproducibility

The paper provides reproducibility details across training, evaluation, compute, and experiment setups. It also limits the scope of its empirical findings to a narrow set of algorithms, models, and evaluated behaviors.

9 Reproducibility

  • The paper documents training and evaluation setups, compute usage, and experiment-specific details, including hyperparameters and dataset creation.Source: §9 · p. 15

A Limitations

  • The empirical study centers on DPO and a few base models: Llama 3, Tulu 3, and OLMo 2.The authors identify other preference algorithms, larger scales, different base models, and new tasks as future directions.Source: A Limitations · p. 19
  • The evaluation does not cover multilingual capabilities or domain-specific use cases such as scientific writing.Source: A Limitations · p. 19

B Evaluation Details

The evaluation uses a consistent OLMES setup across a core benchmark suite, adds three benchmarks for post-training, and evaluates post-trained models on six safety benchmarks.

B Evaluation Details

  • All models use OLMES with the same evaluation configurations across benchmarks.The passage names prompts, metrics, and few-shot examples as examples of shared configurations.Source: B Evaluation Details · p. 19
  • The core suite covers MMLU, MATH, GSM8k, IFEval, AlpacaEval 2, TruthfulQA, BigBenchHard, and Codex HumanEval+.These measure knowledge recall, mathematical reasoning, instruction following, truthfulness, general reasoning, and coding.Source: B Evaluation Details · pp. 19–20
  • Safety evaluation covers refusal of unsafe requests and resistance to jailbreaking prompts across six benchmarks.The benchmarks are XSTest, HarmBench, WildGuardTest, Do-Anything-Now, JailbreakTrigger, and WildJailbreakTest.Source: Table A1 · p. 20
  • Post-training evaluations add PopQA, DROP, and Codex HumanEval to maintain consistency with Tulu 3’s full evaluation suite..Source: B Evaluation Details · p. 20

C Additional Safety Evaluations for Post-trained Models

The post-training safety evaluation finds small average safety declines relative to the base SFT model, while the appendix also reports pilot, stylistic-example, and AlpacaEval 2 analyses.

C Additional Safety Evaluations for Post-trained Models

  • Weak-preference models and Tulu 3 tend to slightly reduce average safety versus the base SFT model, though delta-learning models generally degrade less than Tulu-3-8B-DPO.The Qwen-2.5-3B-over-1.5B model refuses more often than Tulu-3-8B-DPO but is easier to jailbreak; the authors leave effective safety prompt and delta curation open.Source: C Additional Safety Evaluations for… · p. 20

D.1 Pilot Study on ULTRAFEEDBACK-WEAK

  • The ULTRAFEEDBACK-WEAK pilot results are reported as consistent with the earlier finding that preference tuning helps while SFT on weak chosen responses hurts.The appendix table caption describes DPO gains and SFT degradation on weak preferred responses.Source: §D.1 · p. 21, Table A2 · p. 21

D.2 Controlled Experiment: Stylistic Delta in Number of Bold Sections

  • Figure A1 provides examples of generations before and after DPO training in the controlled experiment on the number of bold sections..Source: §D.2 · p. 21

D.3 Ablation Study: Model Size Heuristic, AlpacaEval 2 Discrepancy

  • GPT-4o-preference training yields outputs about 200 characters longer than model-size-heuristic training, which may explain its low AlpacaEval 2 score under length correction.The authors suggest longer-response judge bias may raise output length, which AlpacaEval 2 then penalizes.Source: §D.3 · p. 21

D.4 Additional Ablation Study: Preference Tuning Algorithm

The appendix reports SimPO gains on weak preference data, describes dataset construction and response deltas, and provides statistics and proof preliminaries for the logistic-regression analysis.

D.4 Additional Ablation Study: Preference Tuning Algorithm

  • SimPO tuning on weak preference data gains 5.2 average-performance points over the base SFT model, about 1 point less than DPO.The comparison holds data and base model fixed in the ablation.Source: §D.4 · p. 21, Table A3 · p. 22

E Weak Preference Dataset Details

E.1 Dataset Creation Details

  • After removing about 6,000 duplicated prompts from the Tulu 3 dataset, 264,806 preference pairs remain for each weak dataset.The setup uses the same model to generate chosen responses for every prompt.Source: §E.1 · p. 22

E.2 Qualitative Examples

  • Manual inspection found no universal advantage axis, but Qwen 3B sometimes answers verifiable prompts correctly, gives more detail, or provides chain-of-thought where Qwen 1.5B does not.The authors describe these as non-exhaustive examples and call for further study of semantic deltas and downstream behavior.Source: §E.2 · p. 22

E.3 Quantitative Statistics

  • Chosen responses in the weak data are generally shorter and more vocabulary-diverse than rejected responses, while Tulu 3 responses have largely similar statistics.The authors hypothesize that weak-model repetitions can make rejected responses longer; chosen and rejected embedding cosine similarity is relatively small across datasets.Source: §E.3 · p. 23

F Delta Learning in Logistic Regression

F.1 Additional Preliminaries

  • The proof preliminaries bound preference-gradient covariance using isotropic-Gaussian data assumptions and state a vector Bernstein-Freedman concentration bound..Source: §F.1 · p. 24

F.2 Full Proof of Theorem 6.1

This section presents the proof that early-stopped delta learning improves cosine alignment under Condition C1, including its high-dimensional and stochastic-SGD analysis, alongside the training, experiment, and evaluation setups. It also reports compute costs and provides qualitative examples of weak-model preference pairs and DPO behavior.

F.2 Full Proof of Theorem 6.1

  • Under Condition C1 (κ > 0), early-stopped SGD improves cosine alignment by Θ(κ²), with total failure probability at most δ.The proof derives a deterministic gain from population updates and controls stochastic deviation of mini-batch SGD from those updates.Source: §F.2 · pp. 27–30

F.3 Full Proof of Corollary 6.2

  • In sufficiently high dimensions, random teacher components make Condition C1 hold with high probability because the orthogonal noise vanishes.For fixed failure probability δ, the passage says the condition is easily satisfied as dimension grows, but provides no recoverable numeric threshold here.Source: §F.3 · p. 31

G.1 Shared training and hyperparameter details

  • Shared training uses cosine annealing with 0.03 warmup, AdamW, length-normalized DPO, batch sizes 32 for DPO and 256 for SFT, and one DPO epoch.The setup also uses gradient checkpointing, DeepSpeed, FlashAttention2, and OpenRLHF; SFT sweeps epochs and learning rate.Source: §G.1 · p. 32

G.2 Pilot Study on ULTRAFEEDBACK-WEAK

  • The ULTRAFEEDBACK-WEAK pilot filters out responses from models above 1100 Arena ELO and evaluates on eight core benchmarks.Preference pairs use the highest-scoring response as preferred and a lower-scoring response as rejected; the benchmark list includes MMLU, MATH, GSM8k, and five others.Source: §G.2 · p. 32

G.3 Controlled Experiment: Stylistic Delta in Number of Bold Sections

  • The stylistic-delta experiment collects 16,384 Llama-3.2-3B-Instruct responses, using hard constraints to produce exactly a specified number of bolded sections.Regular expressions guarantee correctness, and hyperparameters are selected using a held-out validation set.Source: §G.3 · p. 33

G.4 Controlled Experiment: Semantic Delta from a Weaker Model

  • For the semantic-delta experiment, learning rates are swept below prior-work settings, and hyperparameters are selected using GSM8k accuracy while other evaluations remain held out.The setup evaluates on the eight core benchmarks; the passage reports that lower rates generally performed better in preliminary experiments for both DPO and SFT.Source: §G.4 · p. 33

G.5 Post-training with Weak Preference Data

  • Evaluation spans 11 benchmarks and reruns T¨ulu-3-8B-DPO with the same current codebase, slightly improving its numbers over the original report.For the best setup, the authors tuned DPO learning rate, β, dataset size, and five random seeds; training on a subset of the full dataset was typically slightly better.Source: §G.5 · p. 33

G.6 Analysis Experiments

  • Analysis uses 21 preference datasets drawn from a random 100k-prompt subset and evaluates on the same eleven benchmarks.DPO learning rate and β were tuned for delta-magnitude and chosen-response-quality experiments; SFT learning rate and epochs were tuned for chosen-response-quality experiments.Source: §G.6 · p. 34

H Compute Details

  • 4–6 hours on one H100 node are required to train an 8B model with DPO on 100k preference pairs, versus 2–4 hours for SFT on 100k data points.All models were trained on single H100 or A100 nodes.Source: H Compute Details · p. 34
  • DPO expands generated sections from 5 to 89 in one example, exceeding the chosen response’s 3 sections.This illustrates that DPO can increase the number of generated sections beyond the chosen response’s absolute quality.Source: Figure A1 · p. 35
  • Preference examples show 3B outputs can be correct, more detailed or structured, or justified where 1.5B outputs are wrong or terse.The examples include a math answer, a knowledge-seeking response, and a prompt admitting a brief answer.Source: Figure A2 · p. 36, Figure A3 · p. 37, Figure A4 · p. 38

Related papers

Loading 2507.06187v1…