PaperlayerView exact revision on arXiv

Source-linked AI summary

Smaller Models, Better Rejects: Preference Distillation Scaling

Rui Cai, Wenhui Zhu, Xiwen Chen, Jincheng Cao, Han Yu, Shayan Mohajer Hamidi, Zelin He, Qiyao Ma, Daiwei Chen, Xuanzhao Dong, Yuanda Xu, Jelena Markovic-Voronov, Kayhan Behdin, Zhengze Zhou, Ran He, Alborz Geramifard, Rohit Jain, Zhe Zhao

arXiv:2609.38987v1cs.LG

TL;DR

Preference distillation commonly treats a student’s own response as the rejected alternative, but those failures may not be the most useful negatives. The paper analyzes reject choice with a linearized DPO utility bound and tests interventions on reject sources and construction. Across students from 7B to 72B, smaller frozen models provide lower-cost rejects that train stronger students across code and math tasks.

  • Problem

    Preference distillation often uses the student’s own response as the reject, leaving open whether those failures provide the most useful negative supervision and whether reject generation must scale with the student.

    Source: §1 · p. 2

  • Method

    The paper derives a finite-horizon utility bound for DPO in a linearized feature model and uses it to motivate interventions on reject source, structure, and reference likelihood.

    Source: §1 · p. 3, §5 · p. 10

  • Results

    Across 7B–72B students, smaller frozen Base models generate rejects with less inference compute yet train stronger students than both Self constructions across code generation and mathematical reasoning.

    Source: §1 · p. 2, §5 · p. 10

  • Takeaways & Limitations

    The results suggest that effective rejects preserve task structure while limiting coupling to the reference policy, and that smaller frozen models can provide them at low cost.

    Source: §5 · p. 10

    The theoretical analysis uses fixed response features and varies the reject distribution while holding the prompt and teacher-response distributions and SeqKD reference shared.

    Source: §3.1 · p. 4

Abstract

from arXiv · show

Preference distillation typically treats a teacher response as preferred and the student's own response as rejected. This assumes that self-generated failures are the most informative negatives and that rejects must come from a model at least as large as the student, making generation costly at scale. We find neither assumption holds: across students from 7B to 72B, smaller frozen models generate rejects with less inference compute yet train stronger students than self-generated rejects, before and after sequence-level knowledge distillation, on code generation and mathematical reasoning. To explain this result, we derive a finite-horizon utility bound for Direct Preference Optimization in a linearized feature model. The bound characterizes favorable reject distributions and motivates three interventions. First, mixing rejects from smaller and student-scale models improves performance as the smaller model's share increases. Second, reassigning rejects to other prompts and shuffling their code tokens still outperform length-matched gibberish, showing that task structure contributes to reject utility. Third, selecting candidates with lower likelihood under the reference policy improves net transfer when higher-likelihood candidates provide less useful contrast. Lower-likelihood selections outperform higher-likelihood ones for every source. These results suggest that effective rejects preserve task structure while limiting coupling to the reference policy, and that smaller frozen models can provide them at low cost.

1 Introduction

The paper challenges the default of using student-generated failures as rejects, finding that smaller frozen models can provide stronger negatives at lower inference cost. It frames reject choice as inverse data design and examines how source, task structure, and reference likelihood affect utility.

1 Introduction

  • Across 7B–72B students, smaller Base rejects require less inference compute yet outperform both Self constructions across code generation and mathematical reasoning.The study includes vanilla Self and post-SeqKD Self as comparisons.Source: §1 · p. 2
  • A linearized feature model of DPO scores reject sources by task-directed movement and the share of that movement pointing the wrong way.The analysis motivates three reject-construction interventions.Source: §1 · pp. 2–3
  • For a 14B student, avg@4 rises from 59.73% to 63.28% as the Llama-1B Base share increases from 0% to 100%.Reassigned smaller Base rejects and even rejects retaining only code tokens outperform length-matched gibberish, indicating a contribution from task-characteristic structure.Source: §1 · p. 2, Figure 1 · p. 2
  • For every source, lower-likelihood selections outperform higher-likelihood selections under the SeqKD reference; the 1.5B source reaches 63.14% versus 61.88%.The passage reports that lower-likelihood rejects help when higher-likelihood candidates provide less useful contrast.Source: §1 · p. 3

1.1 Related Work

  • Prior preference-distillation work often ties response sources to a prescribed pipeline, while this paper varies reject-source identity and composition independently.The paper distinguishes its focus from work on pair selection, optimization changes, and complete-pair or chosen-response-source changes.Source: §1.1 · p. 3

2 Preference Distillation Setting

The setting holds the preference pipeline fixed while changing how rejected responses are constructed. It measures the resulting performance difference between reject constructions under a shared training protocol.

2 Preference Distillation Setting

  • The SeqKD policy initializes DPO and serves as its frozen reference, while rejects are constructed on a disjoint prompt set.The preference dataset pairs prompts and chosen responses with rejects produced by a construction q.Source: §2 · p. 3
  • DPO trains on preference triples containing a prompt, preferred response, and rejected response, minimizing the average loss over the constructed dataset.The reference-relative preference margin is scaled by β.Source: §2 · p. 4
  • The reject-source effect compares performance after two constructions under shared prompts, chosen responses, initialization, objective, optimizer, schedule, budget, and evaluation protocol.Only the reject construction changes between variants.Source: §2 · p. 4

3 Reject Source Selection as Inverse Data Design

The paper treats reject selection as inverse data design: choose a reject distribution that improves the student. A linearized feature model yields a finite-horizon utility bound and motivates interventions on source mixtures, prompt and token structure, and reference likelihood.

3.1 A linearized feature model of Reject Utility

  • The model linearizes DPO around initialization using fixed response features and a trainable coefficient vector; reject construction changes the distribution of feature differences between chosen and rejected responses.Each pair pushes the coefficient vector along its feature difference, with weight decaying as its margin grows.Source: §3.1 · p. 4

3.2 Feature Transfer and a Favorable Region

  • A reject’s utility is evaluated against a comparison distribution by how its transfer moves the initial update along the task-utility direction.Expected task reward defines utility, and the initial utility gradient gives the direction in which an initial update most improves it.Source: §3.2 · pp. 4–5
  • Transfer mass a_Q and adverse share c_Q determine net transfer κ_Q; a finite-horizon bound characterizes when a reject distribution yields a target gain.A larger adverse share requires larger transfer mass; the bound applies under shared regularity conditions, with bounded features sufficient.Source: §3.2 · p. 5

3.3 Construction Interventions and Hypotheses

  • For source mixtures, net transfer is affine in the mixing weights, motivating tests of whether utility rises with the smaller Base share.The analysis holds the common reference, utility, and comparison distribution fixed.Source: §3.3 · p. 5
  • Prompt reassignment preserves prompt-independent reject content, while lexical permutation retains only order-invariant structure, motivating comparison with length-matched gibberish.The account uses an additive decomposition into domain and prompt-pair features.Source: §3.3 · p. 5
  • Reference-likelihood reselection repels from or attracts toward the reference; repulsion raises net transfer when higher-likelihood candidates transfer less.The hypothesis is that near misses may be reference-likely yet provide less contrast, motivating repulsion-versus-attraction tests within fixed candidate banks.Source: §3.3 · p. 6

4 Experiments

Across Qwen2.5 students from 7B to 72B, smaller frozen Base models provide stronger, lower-cost rejects than Self constructions; interventions show that task structure and lower reference likelihood relate to reject utility.

4.1 Experimental Settings

  • The experiments use a two-stage pipeline: SeqKD on teacher responses, followed by DPO on disjoint preference prompts, with reject construction varied across DPO runs.A smaller Base source is a frozen vanilla model with strictly fewer parameters than the student; vanilla Self and post-SeqKD Self are the comparison sources.Source: §4.1 · p. 6

4.2 RQ1: How Should Reject Generation Scale with the Student?

  • Smaller Base sources outperform both Self constructions across the code scaling matrix and improve code and math results, while costing 0.9%–50.1% of same-scale vanilla Self.For a 14B student, Qwen2.5-3B-Instruct rejects improve avg@4 over vanilla Self by 3.1 points on code and 1.9 points on math.Source: §4.2 · p. 6, Figure 2 · p. 7
  • Increasing smaller Base reject share raises 14B code avg@4: Llama-1B rises from 59.73% to 63.28%, and Qwen2.5-3B rises from 59.58% to 62.63%.The intervention varies reject-source proportions on a matched training population.Source: §4.2 · p. 7, Figure 2 · p. 7

4.3 RQ2: Where Does the Smaller Base Gain Reside?

  • Prompt-reassigned Base rejects outperform length-matched gibberish at every student scale, showing that some gain survives without the original prompt pairing.For the 7B student, reassignment improves avg@4 over gibberish by 1.28 points; the incremental effect of restoring alignment varies by student and source.Source: §4.3 · p. 7, Figure 3 · p. 8
  • Lexical permutation beats gibberish at every scale, and preserving only the generated code-token inventory improves 14B avg@4 by 0.73 points.The permutation destroys token order, Python syntax, coherent programs, and prompt correspondence.Source: §4.3 · p. 8, Figure 3 · p. 8

4.4 RQ3: What Makes a Reject Distribution Useful?

  • Higher reject likelihood under the SeqKD reference is associated with lower downstream avg@4 at every student scale; the 7B correlation is −0.97.Smaller Base sources combine lower reference coupling with higher utility, while vanilla and post-SeqKD Self combine higher coupling with lower utility.Source: §4.4 · p. 8, Figure 4 · p. 9
  • Repelled selection yields the strongest endpoint for every source, including 63.14% avg@4 versus 61.88% for Attracted selection with the 1.5B source.The candidate-bank intervention holds generator, candidates, student initialization, and DPO protocol fixed while changing selection by reference likelihood.Source: §4.4 · p. 9, Figure 4 · p. 9

5 Conclusion

The conclusion argues that reject generation need not scale with the student. A linearized DPO bound and three interventions support smaller frozen models as low-cost sources of useful rejects.

5 Conclusion

  • Across students from 7B to 72B, smaller frozen Base models provide stronger rejects than either Self construction on code and math while using less inference compute.The paper concludes that reject generation therefore need not scale with the student.Source: §5 · p. 10
  • A finite-horizon bound in a linearized DPO feature model motivates interventions showing gains from smaller-Base mixtures, task-structured rejects, and lower-reference-likelihood selection.Reassigned rejects and rejects retaining only code tokens outperform length-matched gibberish, while lower-likelihood candidates outperform higher-likelihood ones for every source.Source: §5 · p. 10
  • The paper’s supported conclusion is that effective rejects preserve task structure while limiting coupling to the reference policy.Smaller frozen models provide both properties at low cost.Source: §5 · p. 10

A Experimental Details

The experimental details cover separate code-generation and mathematical-reasoning training lines, their evaluation benchmarks, and code-line training settings.

A Experimental Details

  • The main scaling study evaluates Qwen2.5 students at 7B, 14B, 32B, and 72B on separate verifiable code-generation and mathematical-reasoning lines.The code line uses KoDCode and the math line uses OpenR1-Math.Source: A Experimental Details · p. 14
  • The code preference stage uses 24,248 disjoint prompts, each paired with a verified teacher response and a reject verified incorrect by execution.SeqKD trains on 75,752 prompts with execution-verified teacher solutions; variants share the preference prompts and chosen responses.Source: A Experimental Details · p. 14
  • The math line uses 33,524 SeqKD prompts and 12,288 OpenR1-Math preference prompts, retaining rejects only when an answer verifier marks them incorrect.Source: A Experimental Details · p. 14
  • Code evaluation includes 5,000 held-out KoDCode problems and four external benchmarks; math evaluation uses MATH-500 and AIME with avg@4 over four sampled responses.The external code average weights benchmarks by problem count and is described as a descriptive summary.Source: A Experimental Details · p. 14
  • Code-line runs use full-parameter training with AdamW, cosine scheduling, 10% linear warmup, bf16, gradient checkpointing, and DeepSpeed ZeRO-3.Continued-SFT uses the same settings as DPO.Source: Table 1 · p. 14, A Experimental Details · p. 14

B Reject Generation Cost Analysis

The cost analysis compares output-length-adjusted parameter-token proxies with same-scale vanilla Self and checks their direction against measured H100 runtimes.

B Reject Generation Cost Analysis

  • The proxies account for source-dependent response length; total cost includes prompt prefill, and both quantities are compute proxies rather than measured FLOPs.The 0.5B source averages 466.7 reject tokens, compared with 240–305 tokens for the other Qwen2.5 sources.Source: B Reject Generation Cost Analysis · p. 15
  • Across strict-subscale configurations, total-cost proxies are 0.9%–50.1% of same-scale vanilla Self, corresponding to 2.0×–108.3× reductions.Relative to post-SeqKD Self, the range is 0.9%–55.1%, or 1.8×–108.3× reductions.Source: B Reject Generation Cost Analysis · p. 15
  • Matched H100 generation runs take 1.22, 0.53, 0.64, and 0.84 hours for the 0.5B, 1.5B, 3B, and 7B sources, versus 2.37 hours for the 14B generator.The measured runtime follows the direction of the proxy.Source: B Reject Generation Cost Analysis · p. 15

C Complete Evaluation Results

The evaluation reports results across code and mathematical reasoning, tests cross-family reject sources, and examines how source mixing, prompt assignment, lexical structure, and reference coupling relate to reject utility.

C Complete Evaluation Results

C.1 Cross-Family Reject Sources

  • Llama rejects outperform Self in all nine code configurations and eight of nine MATH-500 configurations, with the exception of Llama-1B for the 72B student.For that exception, MATH-500 is 91.7% versus 91.8%.Source: §C.1 · p. 20, Figure 5 · p. 20

D Construction Intervention Details

  • Across construction interventions, student, prompts, chosen responses, SeqKD initialization, objective, optimizer budget, and evaluation protocol are held fixed while reject construction changes.The interventions cover randomized source assignment, prompt reassignment and lexical permutation, and reference-likelihood reselection.Source: D Construction Intervention Details · p. 21

D.1 Randomized Source Assignment

  • 14B avg@4 rises from 59.730 to 63.280 with Llama-3.2-1B-Instruct rejects and from 59.575 to 62.630 with Qwen2.5-3B-Instruct rejects as Base share increases from 0% to 100%.For the 72B student, avg@4 rises from 65.850 to 67.530 with Qwen2.5-7B-Instruct rejects, passing through 66.685 at 50%.Source: §D.1 · p. 21, Table 6 · p. 21

D.2 Prompt Reassignment and Lexical Permutation

  • Lexical permutation beats gibberish at all four scales and recovers 40.0%–92.5% of the prompt-reassigned gain, indicating that code-domain lexical structure contributes to Base reject utility.Prompt reassignment retains real Base rejects but assigns each to another prompt; the retained rejects still fail at least one new prompt test.Source: §D.2 · pp. 21–22, Table 7 · p. 21, Table 8 · p. 22

D.3 Reference Coupling of Natural Sources

  • The negative association between reference likelihood and downstream utility persists after prompt standardization and every leave-one-source-out analysis, organizing the observed Base-versus-Self boundary.Measurements use the same 24,248 prompts per student scale.Source: §D.3 · p. 22

D.4 Reference Likelihood Reselection

Reference-likelihood reselection changes reject utility across sources, while additional DPO diagnostics examine whether optimization summaries recover the same source ordering.

D.4 Reference Likelihood Reselection

  • Repelled selection improves on Native for all five sources, while Attracted selection reduces utility for all four smaller Base sources; coupling and utility are strongly associated across the 15 arms.The within-generator-centered Pearson and Spearman associations are 0.868 and 0.936, respectively.Source: §D.4 · p. 22, Table 11 · p. 23

E.1 DPO Diagnostic Definitions

  • The rejected-score field’s first-step component lowers rejected likelihood, and its projection norm is intended only for within-student comparisons under the same tangent space.The student-local analysis uses shared prompts and two independent CountSketch projections.Source: §E.1 · p. 24, §E.2 · p. 24

E.2 Reject-Score Fields

  • Post-SeqKD Self has the weakest aggregate reject field at both measured scales, but vanilla Self overlaps the Base field range while achieving lower downstream utility, so field strength alone does not explain the source gap.Comparisons are within student and normalization view.Source: §E.2 · pp. 24–25, Table 12 · p. 25

E.3 Fixed-Pair Dynamics and RC-DPO

The section reports source-dependent fixed-pair and RC-DPO diagnostics, then develops a linearized feature account of how chosen–rejected contrasts shape DPO updates and transfer.

E.3 Fixed-Pair Dynamics and RC-DPO

  • RC-DPO raises fixed-bank Pathway III for raw sources but lowers it for Self, while held-out changes lack a common positive direction and objective calibration widens rather than removes the source gap.The corresponding source-by-objective effects and held-out endpoints are reported in Tables 14 and 15.Source: §E.3 · pp. 25, 28, Table 14 · p. 28, Table 15 · p. 28

F.1 Illustrative Code Example with Two Features

  • In the two-feature code example, one reject reinforces maximum selection, while another cancels that shared feature and corrects an erroneous zero lower bound.This illustrates how one response pair can share a feature while contrasting on another.Source: §F.1 · p. 29

F.2.1 Policy Construction

  • The model starts from the SeqKD reference policy, keeps the reference and response features fixed, and trains only the coefficient vector.The construction assumes a finite normalizer in the parameter region considered.Source: §F.2.1 · pp. 29–30

F.2.2 Connection to Local Linearization

  • With initial network score gradients as fixed features, the constructed policy matches the network’s response probabilities and log-probability gradients at initialization; its later approximation depends on higher-order terms.The expansion is pointwise, and a uniform remainder requires corresponding uniform derivative bounds.Source: §F.2.2 · p. 30

F.2.3 Shared Features and Response Contrasts

  • A chosen–rejected feature difference determines which coordinates change their probability ratio; shared coordinates leave the ratio unchanged.Identical feature values in a coordinate yield a zero difference there.Source: §F.2.3 · p. 31

F.2.4 DPO Margin and Gradient

  • At initialization, every pair has zero margin and weight 1/2, and the DPO update follows the feature difference with a weight that decreases as the margin increases.A positive margin means the chosen-to-rejected probability ratio has increased relative to the reference.Source: §F.2.4 · p. 31

F.2.5 The Reject-Dependent Update Field

  • Reject construction changes the DPO update through the induced distribution of feature contrasts: only its mean enters at initialization, while later updates depend on the broader distribution.This statement holds with the prompt, chosen-response, and feature distributions fixed.Source: §F.2.5 · p. 32

F.3 Vector DPO Dynamics and the Finite-Horizon Bound

The section bounds finite-horizon DPO utility change by separating initial transfer from trajectory and curvature corrections. The bound guarantees a positive utility gain when the transfer term exceeds the correction, under the stated conditions.

F.3 Vector DPO Dynamics and the Finite-Horizon Bound

  • The comparison assumes shared M, L, and L_U bounds, a nonzero initial utility gradient, positive step sizes, and finite response sets or uniform feature bounds.The conditions also specify H update steps and total step size τ_H.Source: §F.3 · pp. 32–33
  • The initial transfer is extracted by projecting the update onto the initial utility gradient and expressing it through the transfer score k.Splitting k into positive and negative parts gives κ_Q = a_Q(1 − 2c_Q).Source: §F.3 · p. 33
  • The same bound applies to Q0 and controls the complete vector remainder, including directions orthogonal to the initial utility gradient.Source: §F.3 · p. 33
  • The bound guarantees positive utility gain whenever K_Hκ_Q > E_H, while the correction must remain small enough not to dominate.The correction is a worst case over compared constructions; the bound is informative only while τ_H is small enough that E_H does not dominate.Source: §F.3 · pp. 33–34

F.4 Intervention Identities and Implementation

The section formalizes how source assignment, prompt structure, and reference-likelihood reselection affect the transfer term, then relates that transfer score to utility-aligned gradient directions. It distinguishes preserved feature components from source-dependent trajectory and curvature corrections.

F.4 Intervention Identities and Implementation

  • The gain lower bound is affine in randomized source weights under uniform M, L, and L_U, while the endpoint also includes source-dependent trajectory and curvature terms.Source: §F.4 · p. 34
  • Independent prompt reassignment preserves the domain-only feature component and its contribution to κ_Q, while changing the prompt-dependent component; lexical permutation preserves order-invariant lexical content.Source: §F.4 · p. 34
  • Reselection uses centered, length-normalized reference log-likelihood scores; positive γ favors lower-likelihood candidates, and negative averaged covariance means stronger repulsion increases leading transfer.At γ = 0, selection is native; shared Gumbel variables couple sampling randomness across arms.Source: §F.4 · p. 34

F.5 Connection to Gradient Interference

  • Unlike pairwise gradient-interference analysis, the transfer score projects changed negative supervision onto the aggregate reward-relevant utility-gradient direction over evaluation prompts.Source: §F.5 · p. 34
Loading 2609.38987v1…