Source-linked AI summary
A Survey on Knowledge Distillation of Large Language Models
Xiaohan Xu, Ming Li, Chongyang Tao, Tao Shen, Reynold Cheng, Jinyang Li, Can Xu, Dacheng Tao, Tianyi Zhou
TL;DR
LLMs lack up-to-date knowledge and can produce factual inaccuracies, while continual fine-tuning can compromise previously acquired abilities. This survey synthesizes KD algorithms, skills, applications, and data augmentation, emphasizing enriched training data as a vital tool for effective distillation and identifying future research opportunities.
Problem
LLMs can lack up-to-date knowledge and produce factual inaccuracies, while continual fine-tuning on particular datasets can cause catastrophic forgetting of earlier knowledge and problem-solving abilities.
Method
The survey organizes LLM knowledge distillation across algorithmic techniques, skill enhancement, vertical applications, and the interaction between data augmentation and distillation.
Results
Data augmentation and synthesis within KD emerge as vital tools for improving distillation, revealing a synergy between enriched training data and effective model distillation.
Takeaways & Limitations
KD can democratize access to advanced LLM capabilities while motivating research on efficiency, transparency, ethics, trustworthiness, weak-to-strong generalization, self-alignment, and multimodal LLMs.
Takeaways & Limitations
Feature-knowledge distillation is unsuitable for black-box LLMs with inaccessible internal parameters, and students distilled from white-box teachers may underperform those from more powerful black-box teachers.
Abstract
from arXiv · showhide
In the era of Large Language Models (LLMs), Knowledge Distillation (KD) emerges as a pivotal methodology for transferring advanced capabilities from leading proprietary LLMs, such as GPT-4, to their open-source counterparts like LLaMA and Mistral. Additionally, as open-source LLMs flourish, KD plays a crucial role in both compressing these models, and facilitating their self-improvement by employing themselves as teachers. This paper presents a comprehensive survey of KD's role within the realm of LLM, highlighting its critical function in imparting advanced knowledge to smaller models and its utility in model compression and self-improvement. Our survey is meticulously structured around three foundational pillars: \textit{algorithm}, \textit{skill}, and \textit{verticalization} -- providing a comprehensive examination of KD mechanisms, the enhancement of specific cognitive abilities, and their practical implications across diverse fields. Crucially, the survey navigates the intricate interplay between data augmentation (DA) and KD, illustrating how DA emerges as a powerful paradigm within the KD framework to bolster LLMs' performance. By leveraging DA to generate context-rich, skill-specific training data, KD transcends traditional boundaries, enabling open-source models to approximate the contextual adeptness, ethical alignment, and deep semantic insights characteristic of their proprietary counterparts. This work aims to provide an insightful guide for researchers and practitioners, offering a detailed overview of current methodologies in KD and proposing future research directions. Importantly, we firmly advocate for compliance with the legal terms that regulate the use of LLMs, ensuring ethical and lawful application of KD of LLMs. An associated Github repository is available at https://github.com/Tebmer/Awesome-Knowledge-Distillation-of-LLMs.
1 INTRODUCTION
The survey presents knowledge distillation as a response to the accessibility, cost, privacy, and capability trade-offs between proprietary and open-source LLMs. It organizes KD around capability enhancement, compression, self-improvement, and data-informed knowledge transfer.
- Motivation: Proprietary LLMs offer advanced emergent abilities, alignment, and task specialization, but their accessibility, cost, and privacy constraints limit use.Open-source models are more accessible, adaptable, and customizable, yet generally have smaller scale, fewer resources, and weaker performance on instruction-heavy or specialized tasks.
- Knowledge Distillation: Knowledge distillation transfers capabilities from advanced teacher LLMs such as GPT-4 or Gemini to open-source student models.The survey frames this transfer as a way to narrow the performance gap while reducing computational requirements.
- Knowledge Distillation: KD in LLMs has three roles: enhancing capabilities, compressing models for efficiency, and enabling self-improvement through self-generated knowledge.These roles are summarized in Figure 1 as the survey’s central framing of KD’s changing function.
- Survey Contribution: The survey is motivated by demand for accessible, cost-effective, adaptable AI and synthesizes methodologies, challenges, breakthroughs, and future research directions.Its organization covers foundational KD, teacher-knowledge elicitation, distillation methods, and open problems.
2 OVERVIEW
The overview distinguishes traditional compression from LLM-era knowledge elicitation and transfer, then organizes the survey around algorithms, skills, and vertical applications. It emphasizes data augmentation as a mechanism for producing context-rich, targeted distillation data.
- Comparing Traditional Recipe: Traditional KD transfers knowledge from large teacher networks to smaller students for deployment under computational and resource constraints.LLM-era KD shifts beyond architecture compression toward eliciting and transferring knowledge from models whose internal parameters may be inaccessible.
- LLM-era Distillation: Prompts elicit specific knowledge and capabilities, including natural language understanding, reasoning, problem-solving, reasoning patterns, preference alignment, and value alignment.This broadens transfer beyond simple output replication toward more abstract cognitive qualities.
- Data Augmentation: Data augmentation within KD generates novel, context-rich training data tailored to particular domains and skills rather than merely enlarging datasets mechanically.The survey describes DA as a symbiotic component of KD that uses seed knowledge to produce explicit skill- or domain-focused data.
- Data Augmentation: DA is presented as a force multiplier that can help distilled models acquire and refine capabilities with fewer datasets and computational resources.The survey connects this use of DA with more efficient, sustainable, and accessible distillation.
- Survey Scope: The taxonomy covers KD algorithms, skill distillation, and verticalization distillation across domains including law, healthcare, finance, and science.KD algorithms provide technical foundations for skill and verticalization distillation, while the survey guides readers through related methodologies and opportunities.
- Scope Boundary: The survey acknowledges that its rapidly evolving subject means it may not include every relevant study or development.It nevertheless aims to present foundational paradigms and impacts across applications.
3 KNOWLEDGE DISTILLATION ALGORITHMS
The survey divides LLM knowledge distillation into two principal steps: eliciting knowledge from teacher models and injecting that knowledge into student models.
- Pipeline Structure: The distillation pipeline is abstracted into ‘Knowledge,’ which elicits information from teacher LLMs, and ‘Distillation,’ which injects it into student models.The two steps correspond to Eq.1 and Eq.2, respectively.
3.1 Knowledge
Knowledge elicitation from teacher LLMs is organized into labeling, expansion, data curation, feature, feedback, and self-knowledge, each providing a distinct route for obtaining distillation knowledge. These methods range from generating outputs and synthetic datasets to extracting internal representations, evaluating student generations, and refining self-generated outputs.
- 3.1.1 Labeling: Labeling uses teacher LLMs to generate outputs for existing inputs under instructions or demonstrations, providing controllable seed knowledge across tasks.Inputs can come from NLP datasets, while instructions such as chain-of-thought prompts can elicit richer signals.
- 3.1.2 Expansion: Expansion uses in-context learning to generate both new inputs and outputs from demonstrations, producing larger and more varied datasets.Its formulation samples x from teacher-conditioned demonstrations and y from the teacher given x and an instruction.
- 3.1.2 Expansion: Expansion quality and diversity depend heavily on the teacher model and seed demonstrations, risking bias amplification and homogeneous generations.The method can expand instruction-following data, but seed-data biases may be inadvertently amplified.
- 3.1.4 Feature: Feature methods extract teacher output distributions or intermediate features, offering transparent knowledge transfer but requiring access to internal model information.Applications include token-level distributions, task-aware layer-wise hidden-state alignment, self-generated sequences, and quantization.
- 3.1.5 Feedback: Feedback methods evaluate student generations with teacher assessments, corrections, preferences, or expansions, enabling response refinement and preference-model distillation.Self-knowledge instead uses the student as both teacher and student, iteratively filtering, rewarding, evaluating, or improving its own outputs without an external powerful teacher.
- 3.1.6 Self-Knowledge: Self-knowledge uses a model’s own generated outputs as material for iterative improvement, potentially surpassing the ceiling of conventional teacher-student methods.The additional process may be governed by external tools or by the student itself.
3.2 Distillation
Distillation transfers teacher knowledge into student models through supervised fine-tuning, divergence or similarity objectives, reinforcement learning, and ranking optimization. These methods respectively imitate teacher sequences, align distributions or representations, learn preference patterns, or directly optimize preference rankings.
- Supervised Fine-Tuning: Supervised Fine-Tuning (SFT) distills black-box LLMs by maximizing the likelihood of sequences generated by teacher models.SFT aligns student predictions with teacher outputs and is described as simple and effective.
- Divergence and Similarity: Divergence-based methods minimize differences between teacher and student probability distributions, while similarity-based methods align their hidden states or features.Similarity objectives transform teacher and student feature maps into comparable shapes before matching them.
- Divergence and Similarity: Forward KL encourages mode covering but may assign mass to low-probability teacher tokens, whereas reverse KL favors prominent modes and can reduce diversity.The survey links mode-covering behavior to hallucinations and low-quality generations, while describing reverse KL as potentially producing more accurate outputs.
- Reinforcement Learning: Reinforcement-learning distillation trains a reward model from teacher feedback, then optimizes the student for reward while constraining divergence from a reference policy.The framework transfers both explicit teacher content and teacher preference patterns; using the teacher directly as reward model can perform better but costs more computation.
- Ranking Optimization: Ranking optimization offers a stable, computationally efficient alternative to reinforcement learning by directly increasing preferred responses’ relative likelihood from fixed preference data.DPO reduces reinforcement-learning reward maximization with a KL constraint to single-stage policy training, while PRO handles rankings of arbitrary length.
4 SKILL DISTILLATION
Skill distillation transfers selected capabilities exhibited by LLMs, including context following, alignment, agent abilities, NLP task specialization, and multi-modality. Context following concerns comprehending and responding effectively to input information, while alignment concerns the model’s behavior relative to intended criteria.
- 4 SKILL DISTILLATION: The survey organizes skill distillation around Context Following, Alignment, Agent, NLP Task Specialization, and Multi-Modality.Context Following addresses the student’s ability to comprehend and respond effectively to input information.
4.1 Context Following
Context-following distillation transfers instruction, dialogue, reasoning, and retrieval-related capabilities from teacher LLMs into smaller student models. The survey emphasizes data construction and teacher-generated reasoning traces as central routes to these capabilities.
- Instruction Following: Instruction-following distillation constructs prompt-response data for supervised fine-tuning, using human curation, task transformation, or teacher LLM generation.GPT-series models are commonly used to generate prompt-response pairs for student training.
- Basic Instructions: 52K task-agnostic instructions expanded from 175 seed tasks provide broad general instruction coverage in Self-Instruct and related work.Filtering and post-processing remove redundant or similar instructions.
- Complex Instructions: 250k complex instructions produced through four evolution rounds enabled WizardLM to achieve a 7.9% higher win rate than ChatGPT on high-difficulty test instructions.Evol-Instruct increases both instruction difficulty and topic diversity.
- Human Instructions: Natural conversation data can improve multi-turn dialogue abilities, but students may imitate teacher style without fully capturing its reasoning process.Vicuna is trained exclusively on ShareGPT conversations, while self-chat methods expand dialogue datasets from seed questions.
- System Instructions: System prompts that request explanation traces expose teacher reasoning and improve smaller models’ instruction following on reasoning-intensive tasks.Orca 2 additionally trains students to select effective solution strategies.
4.2 Alignment
Alignment distillation extends beyond matching teacher answers to learning reflection, preference, helpfulness, harmlessness, and broader human-value criteria. The surveyed approaches use language-model feedback, preference data, and reasoning-oriented supervision to shape student behavior.
- Thinking Patterns: Direct response imitation can reproduce teacher style without transferring the teacher’s reasoning process, motivating distillation of richer thinking patterns.Reflection-based methods fit both final responses and revision or feedback chains.
- Preference: RLAIF combines AI- and human-labeled preferences to optimize helpfulness while minimizing potential harm.Other approaches derive preference comparisons or heuristic guidelines from language-model outputs.
- Preference: RLAIF is reported to match or sometimes surpass RLHF and can also improve supervised fine-tuning performance.Directly prompting an LLM for reward scores may outperform training a reward model from LLM preferences.
- Human Values: Human-value alignment is commonly framed through the HHH criteria: helpful, harmless, and honest.Distillation methods use AI-generated labels, explicit principles, and helpful-versus-harmful generations to guide students.
4.3 Agent
Agent-oriented distillation equips compact models with tool use, task decomposition, and embodied interaction capabilities. The surveyed methods construct supervised datasets or trajectories from teacher models and apply them to student training.
- Tool Use: Tool-use distillation addresses compact models’ difficulty with large numerical values and intricate calculations by training on tool-related demonstrations.ToolAlpaca creates 3938 instances spanning more than 400 real-world APIs across 50 categories.
- Task Decomposition: Agent distillation trains smaller models on trajectories that decompose high-level goals into actionable steps for interactive environments.GPT-4-generated trajectories consistently enhance performance, while AgentTuning uses the AgentInstruct dataset.
- Embodied Multimodal Agents: Embodied multimodal agents can be trained with supervision from pretrained vision-language models, combined with model distillation and hindsight experience replay.The supervision targets language understanding and action within the agent’s operating environment.
4.4 NLP Task Specialization
Knowledge distillation supports NLP task specialization by transferring labels, explanations, generated examples, rankings, and evaluation criteria into task-specific students. Applications span NLU, NLG, information retrieval, recommender systems, evaluation, and code-related tasks.
- NLU: LLM knowledge supports NLU students through information augmentation, data augmentation, semantic representations, labels, and explanations.AugGPT applies teacher knowledge to general and clinical text classification under limited annotation and privacy constraints.
- NLG: GPTSUMM uses over 4 million GPT-3.5-generated paragraph-summary pairs to train a student summarization model.The survey notes a discrepancy between automatic metrics favoring smaller fine-tuned models and human preferences for LLM summaries.
- Information Retrieval: Information-retrieval distillation generates pseudo-queries, transfers retrieval-augmented query understanding, and distills listwise rankings into pointwise rerankers.RankGPT uses GPT-4 to permute candidate passages before student training with ranking losses.
- NLG Evaluation: Distilled LLM evaluators provide more efficient, flexible, and customizable assessment of generated text.PandaLM performs pairwise comparisons, while Prometheus distills customized scoring rubrics from GPT-4.
- Code: WaveCoder distills GPT-3.5-turbo across four universal code-related tasks and demonstrates superior generalization across different code-related tasks compared with other open-source models.It selects a diverse coreset, generates task definitions and outputs, and filters data with teacher evaluation.
4.5 Multi-Modality
Multi-modality distillation extends LLM knowledge transfer beyond language by using teacher models to generate instruction data and distill skills across images, audio, video, and embodied environments.
- Multimodal distillation addresses scarce instruction-following data while transferring commonsense and world knowledge from teacher LLMs.
- Vision-Language: LLaVA converts images into textual descriptions and distills GPT-4 to generate context-specific data, producing the LLaVA-Instruct-150k dataset.
- Vision-Language: GPT-4 distillation supports referential question answering, localized image-region inference, and text-rich image instruction tuning.
- Multiple Modalities: Audio and video methods typically transform multimodal content into text that teacher LLMs can process before distilling instruction-response pairs.
- Others: EMMA distills GPT-4 skills in a parallel textual world, while Silkie distills GPT-4V preferences for embodied or multimodal applications.
5 DOMAIN-SPECIFIED VERTICAL DISTILLATION
Verticalization distillation adapts LLMs to specialized domains by combining domain data, instruction tuning, and teacher-generated or distilled knowledge across law, medicine, finance, and science.
- The verticalization literature customizes distilled LLMs for law, healthcare, finance, science, and other specialized fields.
- Law: Legal applications require retrieval, understanding, reasoning, and generation while handling specialized terminology, subtle interpretations, and evolving legislation.
- Law: LawyerLLaMA combines Chinese legal-corpus pretraining with judicial-exam fine-tuning and ChatGPT-generated consultation responses.
- Medical & Healthcare: Medical distillation augments instruction-following data with advanced teacher LLMs to better align models with practical healthcare interactions.
- Medical & Healthcare: MedAlpaca reconstructs existing data with GPT-3.5-Turbo, while PMC-LLaMA combines continual pretraining with domain-specific multitask supervised fine-tuning and GPT-4 augmentation.
- Finance: XuanYuan uses self-instruct and self-QA over structured and unstructured data to generate finance-domain instruction data.
- Science — Mathematics: WizardMath’s RLEIF method outperforms open-source models on GSM8k and MATH and surpasses ChatGPT-3.5 and Minerva.
- Science: InstructMol combines molecular structures with natural-language instructions through two-stage tuning, improving molecule-related tasks and narrowing the gap with specialized models.
6 OPEN PROBLEMS
Open problems include selecting useful distillation data, reducing cost, combining teachers, extracting richer teacher knowledge, preserving prior capabilities, and improving trustworthiness and self-alignment.
- Further Data Selection: The amount and quality of data required for distillation, including filtering low-quality examples, remain open questions.
- Reduce the Distillation Cost: Model compression and efficient fine-tuning are central strategies for reducing the resource requirements of distillation.
- Multi-Teacher Distillation: Multi-teacher distillation asks whether knowledge from differently trained teachers, such as GPT2 and LLaMA, can be combined in one student.
- Explore Richer Knowledge from Teacher LLMs: Beyond hard-label imitation, researchers seek richer teacher knowledge through feedback, feature knowledge, and combined elicitation methods.
- Overcoming Catastrophic Forgetting During Distillation: Continual fine-tuning can cause catastrophic forgetting, compromising previously acquired knowledge and problem-solving abilities.
- Overcoming Catastrophic Forgetting During Distillation: Effective retention of original capabilities across domains and skills remains challenging, including when distillation lacks access to old data.
- Trustworthy Knowledge Distillation: Trustworthy distillation must address truthfulness, safety, fairness, robustness, privacy, and machine ethics.
- Weak-to-strong Distillation: Weak-to-strong and self-alignment approaches explore whether weaker models or students can provide supervision beyond conventional teacher-provided preferences.
7 CONCLUSION AND DISCUSSION
The survey presents knowledge distillation as a route to more accessible LLM capabilities and emphasizes algorithmic, skill-based, and vertical applications. It identifies data augmentation, efficiency, transparency, ethics, multimodality, weak-to-strong generalization, and self-alignment as important directions.
- The survey organizes LLM knowledge distillation around algorithms, skill enhancement, and verticalization.
- Data augmentation and synthesis emerge as important tools within KD because enriched training data supports effective distillation.
- Future research should pursue efficiency, transparency, ethics, and trustworthiness as model architectures and training methods evolve.
- Multimodal LLMs, weak-to-strong generalization, and self-alignment are identified as promising directions for improving distilled models.
- Applying KD to models such as ChatGPT or Llama requires compliance with model providers’ legal terms.