Survey on Knowledge Distillation for Large Language Models: Methods, Evaluation, and Application

1 INTRODUCTION

• LLMs’ growing parameter sizes bring high inference costs and memory demands that make practical deployment challenging.The paper presents model compression as a way to reduce parameters while preserving performance.Source: §1 · p. 2
• Knowledge distillation trains a smaller model to mimic a larger model’s outputs, representations, or loss behavior, aiming to improve inference speed without substantial performance sacrifice.The student is trained to match original labels as well as teacher behavior.Source: §1 · p. 2
• Prior reviews cover broad knowledge-distillation topics [37] and visual-task techniques [141], but LLMs raise distinct evaluation and real-world application needs.Other cited work examines evaluation metrics [4] and teacher–student architectures [48].Source: §1 · p. 3
• The survey organizes LLM distillation across methods, evaluation, and application, distinguishing white-box logits- and hint-based methods from black-box API-based methods.It also evaluates robustness benchmarks and discusses method relationships, application scenarios, and future research directions.Source: §1 · p. 3

2 OVERVIEW OF KNOWLEDGE DISTILLATION

• The overview frames knowledge distillation through its optimization objectives and distinguishes logits-based, hint-based, and prompt-driven approaches.The listed prompt-driven approaches include in-context learning, chain-of-thought, and instruction following.Source: §2 · p. 3, §2.3 · p. 4, §2.4 · p. 4

2.1 Logits-based KD

• Logits-based KD transfers teacher behavior by aligning student and teacher logits, with temperature controlling logit smoothness and KLD replaceable by other divergence losses.The survey names Reverse Kullback–Leibler and Jensen–Shannon distillation as alternatives.Source: §2.1 · pp. 3–4

2.2 Hint-based KD

• Hint-based KD matches intermediate student and teacher features, using a mapping function to align their dimensions and a metric such as mean squared error.This transfers information about both results and the processes leading to them.Source: §2.2 · p. 4

2.3 In-Context Learning

• In-context learning presents task descriptions or examples as natural-language demonstrations, and the student predicts outputs generated by the LLM.The prompt includes a new query whose answer is left blank for prediction.Source: §2.3 · p. 4

2.4 Chain-of-Thought

• Chain-of-thought distillation adds intermediate reasoning steps to prompts, requiring the student to emulate the teacher’s rationales as well as its labels.This contrasts with in-context learning’s simple input-output pairs.Source: §2.4 · p. 4

2.5 Instruction Following

• Instruction-following distillation generates task-specific instructions with an LLM, then fine-tunes the student on that instruction dataset.Instruction tuning is described as supporting generalization to unseen tasks expressed in instructional formats without explicit examples.Source: §2.5 · p. 5

3 KNOWLEDGE DISTILLATION IN LARGE LANGUAGE MODELS

• Transformer distillation has shifted from BERT-focused work toward billion-parameter models, with methods grouped as white-box or black-box KD.The section focuses on white-box KD, which uses teacher internals during training.Source: §3 · p. 5

3.1 White-box Knowledge Distillation

• White-box KD transfers either teacher logits or intermediate-layer features, with feature matching intended to capture both final outputs and the processes behind them.The section compares these approaches by evaluation tasks and robustness.Source: §3.1 · p. 5

3.1.1 Logits-based KD.

• MixKD [75] interpolates example pairs to augment task-specific data, while ReAugKD adds retrieval-based teacher labels and a relationship loss; both report gains over baselines.MixKD reports gains across six datasets; ReAugKD reports superior performance on six datasets with latency overhead below 3% of baseline.Source: §3.1.1 · p. 6
• MINILLM uses reverse KLD during free-running generation, with sampling and normalization steps to stabilize training; GKD [3] lets students generate sequences and supports alternative divergences.MINILLM outperformed standard KD across five datasets; GKD reported average relative gains of 2.1 times for abstracts, 1.7 times for machine translation, and 1.9 times for arithmetic reasoning.Source: §3.1.1 · pp. 6–7
• Sequence-level f-DISTILL [151] uses step-wise decomposition to compute word-level losses, and its variants outperformed distribution-matching KD methods across four datasets.The reported results also favored symmetric over asymmetric distillation loss.Source: §3.1.1 · p. 7

3.1.2 Hint-based KD.

• Feature-based methods transfer embeddings, hidden layers, and prediction-layer information; PKD [122] uses last-layer or skip-layer strategies and outperformed standard KD across seven datasets and four tasks.MetaDistill [168] uses feedback from a fixed teacher, while AD-KD [155] filters token-attribution dimensions and distills multiple teacher decisions; both report benchmark gains.Source: §3.1.2 · pp. 7–8
• XtremeDistil [91] reports up to 35-fold parameter compression and 51-fold lower batch inference delay while retaining 95% of multilingual NER performance; MiniLM [143] retained over 99% accuracy using half the teacher’s parameters and compute.TinyBERT [55] reports 7.5–9.4× faster inference while its BERT_base teacher exceeded 96.8% on GLUE; TED uses task-aware filters, and MobileBERT and HomoBERT adjust model width or prune neurons.Source: §3.1.2 · pp. 8–9

3.2 Robustness Evaluation of White-box KD

• The evaluation tested five distillation algorithms across four models, measuring adversarial robustness with ASR and OOD robustness with F1 on the specified datasets.The authors frame robustness as important for safety-sensitive applications.Source: §3.2 · p. 9
• For GPT-2, MINILLM led adversarial and OOD results at 340M and beat alternatives on Flipkart and DDXPlus across sizes; OPT favored token-level teacher-distribution KD overall.For OPT, MINILLM also exceeded teacher performance on Flipkart across model sizes.Source: §3.2 · p. 9, Table 2 · p. 10, Table 3 · p. 10
• SeqKD performed comparatively better for LLaMA, whereas JS performed relatively better for LLaMA2, showing that algorithm effectiveness varied across these similar-sized, similarly structured models.Source: §3.2 · p. 11, Table 4 · p. 11

• Hint-based KD aligns intermediate layers and can convey richer information than logits-based alignment, leading to better results.Layer-to-layer distillation requires careful teacher–student layer mapping and a deep understanding of model architecture.Source: §3.3 · p. 11
• Logits-based KD aligns teacher and student output distributions.Source: §3.3 · p. 11
• Both methods use substantial GPU memory because intermediate-feature activations are retained during forward propagation, even without teacher backpropagation.Reducing training costs and time is identified as an important research direction.Source: §3.3 · p. 11

3.4 Black-box Knowledge Distillation

• Black-box KD uses teacher predictions rather than internal data, which is unavailable for many closed-source LLMs.The surveyed methods commonly draw on in-context learning, chain-of-thought, and instruction-following capabilities.Source: §3.4 · p. 11

3.4.1 In-Context Learning.

• Multitask-ICT reduces model size by 93% while retaining 91.4% of the teacher’s performance, with higher computational costs than Meta-ICT.Meta-ICT adapts to unseen tasks, whereas Multitask-ICT treats target tasks as training tasks and uses their examples for distillation.Source: §3.4.1 · p. 12

3.4.2 Chain-of-Thought.

• CoT distillation trains students on teacher-generated reasoning; reported approaches include explanation-based training and multi-solution or consistency-based methods.Step-by-step distillation reduced required training samples by over 50% in reported experiments; other surveyed work reports gains on reasoning benchmarks.Source: §3.4.2 · pp. 12–13

3.4.3 Instruction Following.

• Instruction-following distillation uses instruction, input, and output datasets, with some approaches adding teacher feedback or input-diversity augmentation.Adversarial distillation reached open-generation capability comparable to ChatGPT using 70,000 training samples; UniversalNER instead focuses on input diversity.Source: §3.4.3 · pp. 14–15
• Self-instruction expands a manually seeded task pool by generating instructions and input-output examples, then filtering low-quality or duplicate tasks.The cited process can be repeated until a large task pool is obtained.Source: §3.4.3 · p. 14

• CoT distillation using CQA and SVAMP outperforms distillation using ANLI and e-SNLI on Flipkart and DDXPlus for both LLaMA and OPT.Table 8 presents evaluation results for LLaMA and LLaMA2.Source: §3.5 · p. 16, Table 8 · p. 16
• For GPT-2 models at 120M and 340M parameters, ANLI- and e-SNLI-based interpretations produce better results, but their explanatory power diminishes as model size increases.Source: §3.5 · p. 16
• For OPT models, ANLI- and e-SNLI-generated explanations are suboptimal, while commonsense and mathematical data are more conducive to CoT distillation.Source: §3.5 · p. 16

• Black-box KD is memory-efficient because the teacher generates explanations or instruction pairs while only the student participates in training.Source: §3.6 · p. 16
• Many current methods rely on closed-source teachers, and generating additional data can be costly.Source: §3.6 · p. 16
• Closed-source or unavailable data-generation techniques challenge fair evaluation of black-box distillation algorithms.Source: §3.6 · p. 16

3.7 Others

• Multimodal distillation remains nascent and focuses primarily on improving instruction-following as language models are adapted to visual information.The motivation is that text-focused LLMs cannot comprehend visual information directly.Source: §3.7.1 · p. 16, §3.7 · p. 16

3.7.1 Muiti-Modal Large Language Models.

• CoMD’s second-stage multimodal competitive distillation uses bidirectional feedback for instruction adjustment, evaluation, and augmentation; evaluations show it surpasses existing models in inference tasks and zero-shot settings.Evaluations include ScienceQA [80], SEED-Bench [66], and LLaVA Test Set [78].Source: §3.7.1 · p. 17
• Localized commonsense distillation outperforms passing generated reference expressions to baseline LLMs, while VPD improves counting, spatial understanding, and combinatorial reasoning.VPD samples, executes, and verifies candidate programs, then converts correct programs into language descriptions of inference steps.Source: §3.7.1 · p. 17

4.1 Healthcare

• HuatuoGPT [162] combines ChatGPT-distilled data with physician insights; Chatdoctor exceeds ChatGPT in accuracy and F1, and 13B PMC-LLaMA [153] surpasses ChatGPT on public medical QA benchmarks.Chatdoctor is fine-tuned on 100K patient physiological dialogues; PMC-LLaMA incorporates 4.8M biomedical papers and 30K medical textbooks.Source: §4.1 · pp. 17–18

4.2 Education

• Education examples include DARWIN [156] for scientific discovery, WizardMath [81] for mathematical reasoning, and K2 [29] with GeoBench, the first geoscience benchmark.WizardMath significantly outperforms other open-source LLMs on GSM8k [27] and MATH [42].Source: §4.2 · p. 18

4.3 Law

• Legal models use supervised fine-tuning and retrieval of relevant legal articles, while ChatLaw [28] combines vector and keyword retrieval with self-attention to address hallucination concerns.The described approaches target professional legal skills, reference accuracy, coherence, and problem-solving.Source: §4.3 · p. 18

5.1 Unified Evaluation Benchmark

• GLUE [137], MMLU [41], BIG Bench [119], and HELM [76] cover varied evaluation tasks, but a unified standard for distillation remains an open research direction.Specialized benchmarks include TyDiQA [26] for multilingual knowledge and MGSM [115] for multilingual mathematical reasoning.Source: §5.1 · pp. 18–19

5.2 Advanced Algorithms

• Current methods target specific student capabilities; the survey proposes exploring white-box LLM distillation to integrate multiple capabilities and more advanced MLLM distillation for multiple modalities.The survey notes that MLLM distillation development lags behind LLM distillation.Source: §5.2 · p. 19

5.3 Interpretability

• Distillation fidelity does not consistently track generalization or accuracy, while more faithful students show better calibration; LLM distillation also faces interpretability questions about CoT capability and instruction-data needs.The survey argues that interpretability can support evaluation and improve reliability and predictability in production.Source: §5.3 · p. 19

6 CONCLUSION

• The survey systematically examines LLM knowledge distillation from the perspectives of methods, evaluation, and application.Source: §6 · p. 19
• Many algorithms still rely on frameworks designed for smaller models, while large-model compression remains challenging and calls for efficient methods that preserve universality and generalization.Source: §6 · p. 20