DistiLLM-2: A Contrastive Approach Boosts the Distillation of LLMs

• Because LLM inference requires extensive computation, compressing models while maintaining performance is important for practical deployment.Source: §1 · p. 1
• Prior distillation work separately explored loss design and data curation, while often overlooking their synergy; Li et al. (2024b)’s direct DPO adaptation was prone to reward hacking.Student-generated outputs were proposed to address mismatches that can arise when training only on fixed teacher-generated outputs (Agarwal et al., 2024; Lin et al., 2020; Xu et al., 2024b).Source: §1 · p. 1
• DISTILLM-2 builds on DistiLLM with CALD, a contrastive approach that assigns distinct loss functions to student- and teacher-generated responses to exploit loss–data synergy.The approach is motivated by analysis of forward and reverse KL and SKL during training.Source: §1 · p. 2
• DISTILLM-2 further introduces optimized dataset-curation strategies and curriculum-based adaptive-loss mechanisms as enhancements to CALD.The paper presents these enhancements as practical guidelines for the contrastive approach.Source: §1 · p. 2
• DISTILLM-2 achieves state-of-the-art performance for small language models across instruction-following, mathematical reasoning, and code generation, with applications in preference alignment and vision-language models.The paper also describes preference alignment with better reference models.Source: §1 · p. 2

2.1. Related Work

• Prior work progressed from using student-generated outputs for distillation to on-policy divergences, policy-gradient methods, speculative decoding, and DistiLLM’s skew KL with adaptive off-policy training.DistiLLM is described as achieving state-of-the-art performance and greater efficiency.Source: §2.1 · p. 2

2.2. Preliminary: KD in LLMs and DistiLLM

• KL is the conventional tractable KD loss, but forward and reverse KL can cause mode-averaging and mode-collapse, respectively; skew KL improved convergence speed and performance over recent baselines.The cited baselines are MiniLLM and GKD.Source: §2.2 · p. 2
• Adaptive off-policy distillation reuses student-generated outputs through a replay buffer, balancing efficiency and efficacy while addressing costly, low-quality on-policy data.The approach was also used in subsequent LLM preference-alignment work.Source: §2.2 · p. 2
• Batch collection of student-generated outputs before each epoch reduces sample-gathering cost with minimal impact on student performance while remaining compatible with vLLM.This preserves the adaptive off-policy approach’s core philosophy while increasing generation efficiency.Source: §2.2 · pp. 2–3

3.1.1. MOTIVATION

• DPKD (Li et al., 2024b) can reward hack by excessively suppressing student responses, yielding student-response NLL 91.25 rather than fitting teacher responses at NLL 20.29.The paper attributes this failure to substituting the teacher for DPO’s reference model when the teacher probability of student responses is inherently small.Source: §3.1.1 · p. 3

3.1.2. CONTRASTIVE APPROACH FOR LLM DISTILLATION

• CALD’s SKL-on-teacher and SRKL-on-student design converges faster and performs better than using either loss alone, with higher efficacy than using KL or RKL as its backbone.The method assigns SKL to teacher responses, where teacher probability is high, and SRKL to student responses, where it is generally low.Source: Figure 1 · p. 4, §3.1.2 · p. 4

3.2. Optimal Data Curation for Contrastive Approach

• Teacher generations are preferred for SKL, while student generations are preferred for SRKL: speculative-response performance peaks at ε = 1.0 and ε = 0.0, respectively.Even stronger LLMs’ higher-quality responses did not outperform the teacher’s for SKL; speculative responses can weaken CALD’s complementary effects, while student responses help reduce training-inference mismatch.Source: §3.2 · pp. 5–6

3.3. Curriculum-based Adaptive Learning

• Curriculum-based α adapts to sample difficulty, while a gradually increasing SRKL coefficient β significantly improves student performance.Larger α favors stability and convergence, whereas smaller α permits more knowledge acquisition; optimal α also varies by setup and training epoch. Gradually increasing β shifts emphasis toward feedback from student responses, reducing training-inference mismatch when teacher imitation is difficult.Source: §3.3 · pp. 6–7, Table 5 · p. 7

4.1. General Instruction-Following

• General instruction-following training uses 50k UltraChat200k prompts with teacher- and student-generated responses, evaluated on AlpacaEval, Evol-Instruct, and UltraFeedback.Evaluation uses GPT-4o or GPT-4o-mini as LLM-as-a-Judge; teachers are Qwen2-7B, Mistral-7B, and Gemma-2-9B, paired respectively with Qwen2-1.5B, Danube2-1.8B, and Gemma-2-2B students.Source: §4.1 · p. 7
• DISTILLM-2 performs best in most instruction-following setups, except Danube2-1.8B on Evol-Instruct, and beats second-best methods on average for all three students.The average margins are +2.34% for Qwen2-1.5B, +1.95% for Danube2-1.8B, and +4.53% for Gemma2-2B.Source: §4.1 · p. 7

4.2. Mathematical Reasoning

• The math experiments pair Qwen2-Math-7B-Inst and Qwen2.5-Math-7B-Inst teachers with their respective 1.5B students and use MetaMathQA training data.The passage specifies 50k randomly selected MetaMathQA samples and supervised fine-tuning for one epoch on the entire dataset.Source: §4.2 · p. 7
• DISTILLM-2 outperforms recent baselines on GSM8K and MATH in both Qwen2 and Qwen2.5 setups, and the Qwen2.5 student surpasses the Qwen2 teacher on MATH.The passage describes the Qwen2.5 student's average performance as competitive and the MATH result as a challenging milestone.Source: §4.2 · p. 7

4.3. Code Generation

• Code-generation experiments use WizardCoder prompts, Qwen2.5-Coder-7B-Inst and DeepSeek-Coder-6.7B-Inst teachers, and their respective 1.5B and 1.3B students.Students are trained for two epochs and evaluated on HumanEval and MBPP.Source: §4.3 · pp. 7–8
• DISTILLM-2 consistently outperforms GKD and DistiLLM on HumanEval and MBPP; GKD scores higher than DistiLLM but remains below DISTILLM-2.The reported outcome is attributed to integrating SKL or SRKL with harmonized response types.Source: Table 4 · p. 7, §4.3 · p. 8

Component Analysis

• Adding the contrastive approach, increasing β, and using curriculum-based α updates yields incremental gains that bring DistiLLM’s performance in line with DistiLLM-2.Source: §5 · p. 8

Training Size

• Across instruction-following, math reasoning, and code generation, DistiLLM-2 consistently outperforms GKD and DistiLLM across varying training data sizes.Source: §5 · p. 8

Capacity Gap

• With Qwen1.5-0.5B as student, DistiLLM-2 improves monotonically and outperforms baselines as Qwen-1.5-Chat teacher size rises from 1.8B to 7B and 14B.The previous version struggled with capacity gaps, particularly with 7B and 14B teachers.Source: §5 · p. 8, Table 8 · p. 8

6.1. Additional Results for DISTILLM-2 + DPO

• Replacing SFT with distillation before DPO improved overall preference-alignment performance: DISTILLM-2 more than doubled WR for Qwen2-1.5B and showed a similar improvement for Gemma2-2B.The study evaluated the policy LLMs after DPO fine-tuning.Source: §6.1 · pp. 8–9

6.2. Expansion to Vision-Language Models

• On VLMs, DISTILLM-2 outperformed GKD and DistiLLM by +2.01% and +4.61% on average, respectively, using LLaVA-1.5-7B as teacher and TinyLLaVA-1.4B as student.The evaluation used 83K RLAIF-V-Dataset prompts and the OK-VQA and TextVQA benchmarks.Source: §6.2 · p. 9

6.3. Restoring the Performance of Quantized LLMs

• DISTILLM-2 achieved the best average performance among KD baselines when restoring 4-bit quantized LLMs with LoRA, while adding only a few trainable parameters.The approach replaces regular SFT with KD and can be applied to original and compressed model pairs for efficient mobile deployment.Source: §6.3 · p. 9

6.4. Inference Speedup of Speculative Decoding

• DISTILLM-2 draft models achieved greater speculative-decoding speedup than SFT- and DistiLLM-trained drafters for both Phi3.5-mini and Phi3-medium verifiers, indicating stronger token-level distribution alignment than other baselines, including Zhou et al. (2024b).The evaluation used Phi3.5-mini and Phi3-medium as verifiers and Llama-68m as the drafter.Source: Table 10 · p. 9, §6.4 · p. 9

7. Conclusion

• DISTILLM-2 achieves state-of-the-art performance with improved sample efficiency, reducing reliance on expensive preference-labeled data.The reported experiments span instruction following, mathematical reasoning, and code generation.Source: §7 · p. 9
• The framework combines contrastive loss, curated data, and curriculum-based learning, differentiating teacher and student outputs to strengthen alignment and generalization.Source: §7 · p. 9
• High-quality explanations and contrastive objectives further enhance reasoning ability and robustness in language and vision-language models.Source: §7 · p. 9

High-performed reference model for RLHF

• DistiLLM-2 can replace vanilla SFT before PPO with a reward model or DPO using chosen/rejected pairs; Table 6 tests DPO, while broader preference-method use is proposed.The authors also suggest applicability to IPO and SimPO.Source: Impact Statement · p. 10

Extension to multi-modal LLMs

• Experiments demonstrate potential adaptability to VLMs, and the authors suggest applying DistiLLM-2 across MLLMs built on LLMs (Dai et al. 2023; Tang et al. 2024).Source: Impact Statement · p. 10

Recovering the compressed LLMs

• DistiLLM-2 substantially improves quantized LLM performance, making these models highly competitive with their uncompressed counterparts; the authors suggest broader use with compression techniques.Source: Impact Statement · p. 10

Enhancing Inference speed of LLMs

• DistiLLM-2 improves draft–target alignment to enhance speculative decoding (Chen et al., 2023; Leviathan et al., 2023), potentially benefiting systems that combine diverse models.Source: Impact Statement · p. 10

• The authors propose exploring integration of DeepSeek-R1’s reasoning ability (Guo et al., 2025) with DistiLLM-2 to improve results for sLMs.They encourage future work on broader implications across diverse domains.Source: Impact Statement · p. 10

A. Additional Related Works

• Most LLM KD work uses logits, unlike BERT-based distillation focused on intermediate layers, because LLMs have many parameters.The passage contrasts the focus of small BERT-based models with that of LLMs.Source: A. Additional Related Works · p. 15
• Gu et al. (2024) proposed policy-gradient KD to address high variance in RL-based methods, while Agarwal et al. (2024) proposed on-policy SGO with RKLD and JSD objectives.Source: A. Additional Related Works · p. 15
• Later LLM KD methods include Li et al. (2025), who use top-k logits and logit differences, and Zhang et al. (2024), whose DSKD unifies model output spaces.Source: A. Additional Related Works · p. 15
• Wu et al. (2025) proposed adaptive KL to balance early-stage KL and reverse-KL behavior, but did not consider the data perspective of LLM distillation.Source: A. Additional Related Works · p. 15
• Actor-critic RLHF aligns language models with human preferences but can be unstable and memory-intensive because policy and reward models must be on device simultaneously.The passage cites Christiano et al. (2017), Stiennon et al. (2020), Bai et al. (2022), and Ouyang et al. (2022) for actor-critic RLHF.Source: A. Additional Related Works · p. 15
• DPO (Rafailov et al., 2023) and SLiC-HF (Zhao et al., 2023) learn contrastive preferences offline with closed-form losses, without a critic or reward model.The passage also cites Rafailov et al. (2023), Azar et al. (2024), and Ko et al. (2025) as offline contrastive preference algorithms.Source: A. Additional Related Works · p. 15
• Azar et al. (2024) argued that unregularized policies can overfit deterministic preferences and introduced IPO to optimize offline preference probabilities with regularization.Source: A. Additional Related Works · p. 15

B.1. Mathematical Explanation on Behavior of KL and RKL

• KL and SKL pull student probability upward where teacher probability is high, while RKL and SRKL push it downward where teacher probability is near zero.The explanation uses their f-divergence generators and is stated to apply to a more general setup than Wu et al. (2025), whose explanation is limited to unimodal Gaussians.Source: §B.1 · p. 15

B.2. Derivation for Remark 1

• Equation 5 can be rewritten as Equation 6, which behaves similarly to the contrastive approach in DPO (Rafailov et al., 2023).The derivation uses the definitions of SKL and SRKL and linearity of expectation for independent teacher and student outputs.Source: §B.2 · p. 16

B.3. First-order Approximation for Mercator Series

• The first-order Mercator approximation expresses S(R)KL as a closed-form function of α and teacher and student probabilities, enabling per-sample α selection.Because approximation error is a concern when either probability is much smaller than the other, the implementation allocates α by mini-batch and clips it between 0.01 and 0.1.Source: §B.3 · p. 16

C. Detailed Experimental Setup

• DISTILLM-2 and baselines GKD and SKD are implemented using the trl framework.The appendix introduces separate dataset, training, and evaluation setup descriptions.Source: C. Detailed Experimental Setup · p. 16

C.1. Dataset Description

• The experiments cover instruction-following, mathematical reasoning, and code generation, including UltraChat200k, MetaMathQA, GSM8K, and WizardCoder.The datasets include filtered dialogue, bootstrapped math problems, grade-school multi-step problems, and evolved code instructions.Source: §C.1 · p. 17
• Visual-language evaluation and preference data include RLAIF-V-Dataset, OK-VQA, and TextVQA.RLAIF-V-Dataset contains multimodal preference pairs, OK-VQA requires external knowledge, and TextVQA tests reasoning over text in images.Source: §C.1 · p. 18

C.2. Training Details

• Training uses LoRA, the largest batch size that fits on four NVIDIA A100 80GB GPUs, and an effective batch size of 128.Students are first trained on ground-truth responses with SFT and then undergo KD; experiments omit pretraining-corpus language-modeling loss.Source: §C.2 · p. 18, Table 11 · p. 18

C.3. Evaluation

• Instruction-following responses were sampled on one NVIDIA A100 80GB GPU at temperature 0.8, top-p 0.95, and a 512-token maximum.The LLM-as-a-Judge comparison used temperature 0.7 and a pairwise prompt shown in Figure 4.Source: §C.3 · p. 18, Figure 4 · p. 19
• For AlpacaEval, responses were compared with released text-davinci-003 responses; Evol-Instruct and UltraFeedback used internally produced gpt-3.5-turbo responses.Source: §C.3 · p. 18
• Instruction-following pairwise comparison results were averaged across both response orders to avoid position bias.Source: §C.3 · p. 18
• Mathematical reasoning and code-generation responses were sampled greedily on one NVIDIA A100 80GB GPU with a 1,024-token maximum.Code generation was evaluated using the EvalPlus framework.Source: §C.3 · p. 18

D.1. Comparison on On-policy Setup

• The on-policy comparison includes ImitKD (Lin et al., 2020), MiniLLM, GKD, and adaptive on-policy DistiLLM and DISTILLM-2.It follows Ko et al. (2024)’s setup, training GPT-2 on databricks-dolly-15k.Source: §D.1 · p. 19
• On databricks-dolly-15k test-set ROUGE-L, the “batched” on-policy setup avoids severe performance degradation despite its significant efficiency.The paper says this shares the takeaway of Ko et al. (2024)’s adaptive off-policy approach.Source: §D.1 · p. 19, Table 12 · p. 19
• DPKD (Li et al., 2024b) performs much worse than its reported values, which the authors attribute to reward hacking.They therefore exclude DPKD from the main baselines.Source: §D.1 · p. 19
• Except in the on-policy setup, DPO performs worse than DISTILLM-2 but better than DPKD.Source: §D.1 · p. 19

D.2. LLM distillation with Inst models

• The instruction-tuned comparison uses Gemma-2-2B-it as student and Gemma-2-9B-SimPO as teacher, training the student for one epoch of 200 iterations.The other experimental settings match those used for the base model.Source: §D.2 · p. 19
• Inst-model students outperform base models (+SFT), and DISTILLM-2 performs best among the student models on UltraFeedback, even exceeding its teacher.The results support the effectiveness of distillation with recent state-of-the-art models.Source: §D.2 · p. 19, Table 13 · p. 19
• The authors attribute DISTILLM-2’s strong instruction-tuned results to the contrastive approach’s fast convergence, even with limited training iterations.Source: §D.2 · p. 19