PaperlayerView exact revision on arXiv

Source-linked AI summary

A Comprehensive Survey on Knowledge Distillation

Amir M. Mansourian, Rozhan Ahmadi, Masoud Ghafouri, Amir Mohammad Babaei, Elaheh Badali Golezani, Zeynab Yasamani Ghamchi, Vida Ramezanian, Alireza Taherian, Kimia Dinashi, Amirali Miri, Shohreh Kasaei

arXiv:2503.12067v2cs.CV

TL;DR

Large models pose runtime, memory, annotation, and privacy challenges, particularly for edge deployment and settings with limited labels. This paper surveys knowledge distillation across sources, schemes, algorithms, modalities, and applications, emphasizing recent areas including foundation models, LLMs, diffusion models, and 3D data. It concludes that KD supports lightweight models and broad knowledge-transfer applications, while identifying source selection, scheme choice, and teacher-student capacity gaps as challenges.

  • Problem

    Large models create substantial runtime and memory demands, while some tasks also lack sufficient labels or involve data-privacy constraints.

  • Method

    The paper presents a comprehensive survey organizing KD methods by distillation sources, schemes, algorithms, modalities, and applications.

  • Results

    The survey covers recent KD developments across feature-based, adaptive, contrastive, foundation-model, LLM, diffusion-model, and 3D applications.

  • Takeaways & Limitations

    KD transfers knowledge from high-capacity teachers to lightweight students and supports applications including point-cloud segmentation, cross-modal learning, and efficient generation.

  • Takeaways & Limitations

    KD remains challenged by choosing suitable knowledge sources and distillation schemes, and by capacity gaps between teacher and student models.

Abstract

from arXiv · show

Deep Neural Networks (DNNs) have achieved notable performance in the fields of computer vision and natural language processing with various applications in both academia and industry. However, with recent advancements in DNNs and transformer models with a tremendous number of parameters, deploying these large models on edge devices causes serious issues such as high runtime and memory consumption. This is especially concerning with the recent large-scale foundation models, Vision-Language Models (VLMs), and Large Language Models (LLMs). Knowledge Distillation (KD) is one of the prominent techniques proposed to address the aforementioned problems using a teacher-student architecture. More specifically, a lightweight student model is trained using additional knowledge from a cumbersome teacher model. In this work, a comprehensive survey of knowledge distillation methods is proposed. This includes reviewing KD from different aspects: distillation sources, distillation schemes, distillation algorithms, distillation by modalities, applications of distillation, and comparison among existing methods. In contrast to most existing surveys, which are either outdated or simply update former surveys, this work proposes a comprehensive survey with a new point of view and representation structure that categorizes and investigates the most recent methods in knowledge distillation. This survey considers various critically important subcategories, including KD for diffusion models, 3D inputs, foundational models, transformers, and LLMs. Furthermore, existing challenges in KD and possible future research directions are discussed. Github page of the project: https://github.com/IPL-Sharif/KD_Survey

1 Introduction

This survey organizes knowledge distillation across sources, algorithms, schemes, modalities, and applications, addressing recent developments involving feature distillation, foundation models, LLMs, diffusion models, and 3D data. It also compares prominent methods and discusses current challenges and future directions.

  • Motivation: Large models create runtime and memory challenges for deployment on edge devices, motivating approaches such as pruning, quantization, low-rank factorization, and knowledge distillation.These challenges are especially relevant for large language and foundation models.
  • Motivation: Knowledge distillation transfers knowledge from a larger teacher to a lightweight student and can also support knowledge transfer across tasks or modalities with limited labels.Data-free distillation can generate synthetic data when privacy concerns prevent sensitive-data storage.
  • Survey scope: Compared with earlier surveys, it addresses overlooked feature-based methods, foundation models, LLMs, and 3D inputs while providing quantitative comparisons, challenges, and future directions.Earlier surveys are described as missing recent advances in these areas.
  • Survey scope: The survey classifies distillation sources as logit-based, feature-based, and similarity-based, with particular attention to recent feature-based methods.The authors emphasize feature-based distillation because of its importance and widespread application.
  • Survey scope: It reviews attention-based, adversarial, multi-teacher, cross-modal, graph-based, adaptive, and contrastive distillation algorithms.Adaptive and contrastive distillation are presented as recent categories not covered in previous works.
  • Survey scope: The survey covers offline, online, and self-distillation schemes and adds modality-based coverage of video, speech, text, multi-view, and 3D data.Its application coverage includes self-supervised learning, foundation models, transformers, diffusion models, visual recognition, and LLMs.

2 Sources

The survey organizes knowledge distillation by knowledge source into logit-based, feature-based, and similarity-based methods. It reviews how these sources, transformations, and relations transfer information from teacher to student networks.

  • Knowledge-source taxonomy: Knowledge sources are classified into logits, intermediate features, and similarities among channels, instances, classes, or other features.Logits are final-layer outputs, features are intermediate outputs, and similarities encode higher-order relationships.
  • Logit-based distillation: Logit distillation trains the student to mimic the teacher’s final predictions using a loss that measures differences between logits.For classification, soft-label distributions apply softmax with a temperature factor, and KL divergence can measure distribution differences.
  • Logit-based distillation: Logit-based methods normalize, soften, or smooth logits, but their predefined-class outputs restrict them largely to supervised learning.Final-layer matching also omits intermediate teacher representations and can be difficult when teacher and student architectures differ.
  • Feature-based distillation: Feature distillation aligns intermediate teacher and student representations with similarity functions and transformations that match spatial sizes or channel counts.Intermediate features provide multi-level, step-by-step information, especially when a model’s output is a representation rather than logits.
  • Feature-based distillation: Feature-based research includes attention transfer, neuron-selectivity matching, activation-boundary transfer, cross-layer matching, and learned feature transformations.Recent methods also explore diffusion-based denoising, augmented student features, frequency-domain distillation, and decomposed magnitude-versus-angular alignment.

3 Schemes

The survey organizes teacher-student distillation into offline, online, and self-distillation schemes, distinguished by teacher availability, training synchronization, and network identity. It also summarizes each scheme’s practical uses, limitations, and recent algorithmic developments.

  • 3.1 Offline Distillation: Offline distillation pre-trains a large teacher, then transfers its knowledge while keeping the teacher’s weights frozen during student training.This is the most common distillation approach.
  • 3.1 Offline Distillation: Offline distillation is preferred when a large pre-trained teacher exists, but loading it creates memory and computational overhead in resource-constrained environments.A fixed teacher can also constrain student performance and transfer teacher biases.
  • 3.1 Offline Distillation: In LLM distillation, offline distillation remains the only practical approach when the proprietary teacher is accessible only through an API.The survey presents this as a setting-specific reason for continued use of offline distillation.
  • 3.2 Online Distillation: Online distillation trains teacher and student simultaneously, avoiding dependence on a large pre-trained model and enabling virtual teachers formed from groups of students.Early methods used ensembles of multi-branch logits to create an on-the-fly teacher.
  • 3.3 Self-distillation: Self-distillation uses identical teacher and student networks, with methods transferring knowledge across training stages, depths, or classifiers within the same network.Recent work includes deeper-to-shallower distillation and other diverse self-distillation algorithms.

4 Algorithms

The survey organizes KD algorithms around mechanisms for transferring and refining teacher knowledge, including attention, adversarial, multi-teacher, cross-modal, and contrastive approaches. These methods target selective information transfer, robustness, modality gaps, and richer feature or relational alignment.

  • Attention-based distillation: Attention distillation transfers teacher attention maps so students focus on crucial feature regions rather than exactly matching predictions.Attention mechanisms reflect neuron selectivity and highlight important image regions for student learning.
  • Adversarial distillation: Adversarial distillation uses generative networks or discriminators to synthesize data, refine teacher guidance, or align teacher and student representations.Discriminators can align logits or intermediate features, while adversarial frameworks also support GAN and diffusion-model compression.
  • Adversarial robust distillation: Adversarial Robust Distillation combines distillation with adversarial training to reduce prediction discrepancies under adversarial perturbations.Later methods refine teacher reliability, balance natural and adversarial distributions, manage data imbalance, and sample diverse adversarial examples.
  • Cross-modal and contrastive distillation: Cross-modal distillation transfers knowledge from well-annotated modalities to less-annotated ones while allowing inference without the additional modality.Contrastive distillation complements this landscape through feature alignment, relational knowledge preservation, and local-global information integration.

5 Modalities

Although KD is predominantly studied for computer vision and image inputs, it has also been applied to 3D or multi-view data, text, speech, and video. The survey organizes these methods by modality-specific tasks.

  • Modalities: KD methods extend beyond image-based computer vision to 3D or multi-view data, text, speech, and video.The survey categorizes methods for each modality according to their respective tasks.

5.1 3D Input

The survey reviews KD across 3D tasks including object detection, classification, segmentation, depth estimation, representation learning, and shape-related problems. Methods address geometric, sparsity, annotation, and cross-modal challenges using point, voxel, image, LiDAR, and BEV representations.

  • 3D task coverage: KD is applied to 3D object detection, semantic segmentation, shape generation, and shape classification across point clouds, voxel grids, and multi-angle images.The survey groups 3D methods by task domain, data type, and distillation source.
  • 3D object detection: LiDAR-based detection methods address sparsity, occlusion, geometric structure, weather-induced domain gaps, and varying point density through feature, logit, and refinement strategies.Examples include local geometric modeling, feature reduction, mutual refinement, and density- or shape-based instance alignment.
  • 3D object detection: Cross-modal detection transfers spatial, depth, and geometric knowledge from LiDAR or fused teachers to camera, radar, or multi-camera students, often through BEV representations.Shared BEV spaces align heterogeneous teacher and student detectors while avoiding LiDAR at inference in camera-based systems.
  • 3D semantic segmentation: KD improves point-cloud segmentation accuracy and robustness by transferring knowledge from high-capacity teachers to lightweight students.Point-to-voxel KD transfers teacher outputs at fine-grained point and coarse voxel levels, while multi-to-single KD combines feature, logit, and instance-aware similarity distillation.
  • 3D semantic segmentation and representation learning: Self-distillation, self-supervision, and cross-modal transfer address expensive dense annotations, domain differences, and costly 3D pre-training.Methods use graph consistency, vision foundation models, 2D semantic knowledge, and image-LiDAR contrastive alignment for segmentation and representation learning.

5.2 Multi-view Input

Multi-view KD transfers and refines information across perspectives and modalities for 3D perception and broader applications. The surveyed methods target representation alignment, incomplete views, temporal structure, computational complexity, and scarce depth supervision.

  • Overview: Multi-view learning improves accuracy and robustness by combining perspectives, but modality diversity and cross-modal knowledge transfer create alignment challenges.These challenges motivate KD methods for sharing and refining knowledge across views and modalities.
  • 3D object detection: Multi-view 3D detection uses fusion teachers, simulated or camera-based students, cross-task distillation, cross-modal feature and output transfer, and adversarial training.LiDAR-based teachers commonly provide depth and geometric information to students operating with less costly sensors.
  • Challenges: Cross-modal KD can suffer from feature distribution mismatches, while FSD removes the need for pre-trained teachers and complex strategies.This limitation defines an important boundary for transferring knowledge between heterogeneous modalities.
  • 3D object detection: BEV-based methods align image and LiDAR features, transfer foreground and depth knowledge, and regulate temporal or cross-modal similarities to improve multi-view detection.Several frameworks use BEV as a shared representation, while others reduce redundancy through cross-correlation or encode temporal relations across frames.
  • Broader applications: Multi-view distillation also supports shape recognition, robotic manipulation, BEV detection, stereo depth reconstruction, segmentation, action recognition, and medical diagnosis.Applications include transferring multi-camera teacher policies to single-view students and probabilistically transferring self-supervised depth knowledge.

5.3 Text Input

Knowledge distillation transfers capabilities from large language and NLP models to smaller students, reducing computational costs while supporting performance across diverse text tasks.

  • Overview: KD transfers large models’ representations, probabilities, or predictions to lightweight students for efficient NLP deployment.The surveyed applications include neural machine translation, question answering, text generation, and understanding tasks.
  • Neural Machine Translation: Comparable translation quality with reduced computational costs makes KD useful for efficient and scalable neural machine translation.
  • Question Answering: In question answering, students learn answer predictions and intermediate representations such as attention distributions from larger teachers.This transfers both output behavior and internal processing signals while minimizing computational costs.
  • Text Generation: Text generation distillation transfers vocabulary distributions and language patterns, with some methods targeting contextual understanding, fluency, and diversity.The cited approaches include BERT-based efficient generation and distillation combined with GANs.
  • Other Text Tasks: KD also supports event detection, document retrieval, text recognition, and named entity recognition by transferring task-specific capabilities to smaller models.
  • Language Understanding: Across NLU, sentiment analysis, and text classification, students mimic teacher probabilities, decision patterns, contextual understanding, or feature representations.

5.4 Speech Input

Speech-input distillation compresses complex acoustic, generative, and multimodal models for efficient deployment while transferring capabilities across recognition, enhancement, synthesis, and understanding tasks.

  • Overview: KD addresses the deployment cost of complex speech models on resource-limited devices by transferring knowledge to smaller, faster students.
  • Speech Recognition: In speech recognition, KD supports rapid inference with minimal accuracy loss and can improve adaptation to poor acoustic or expert domains.Medical transcription is cited as an example of an expert domain.
  • Speech Enhancement: KD enables speech enhancement through denoising and model compression, while preserving high-fidelity speech representations.
  • Recognition and Security: For speaker recognition, verification, and spoofing detection, KD reduces complexity while preserving identity or anti-spoofing capabilities.The surveyed applications include noisy, multilingual, cross-domain, and adversarial settings.
  • Translation and Understanding: Speech translation and spoken-language understanding use KD to transfer information from high-resource multilingual teachers to lighter models for low-resource settings.Speech translation is also described as maintaining low latency for real-time applications.
  • Other Speech Tasks: KD compresses models for speech synthesis, separation, audio captioning, retrieval, and conversational question answering while retaining task-relevant capabilities.

5.5 Video Input

Video knowledge distillation targets the computational burden of video models, transferring knowledge across diverse video tasks and between video, image, text, and audio modalities.

  • Video Applications: KD is increasingly applied to video action recognition, classification, segmentation, retrieval, forecasting, and masked video representation learning.
  • Organization: Table 8 summarizes video KD methods according to their sources of distillation.
  • Efficiency Motivation: For video quality estimation and attention prediction, KD lets smaller students learn from complex teachers while reducing computational cost with minimal performance reduction.
  • Cross-Modal Distillation: Video distillation supports cross-modal transfer in video-to-image, video-to-text, text-to-video, audio-to-video, and video-to-audio directions.
  • Cross-Modal Distillation: A spatio-temporal graph model uses KD for video captioning to improve robustness against spurious correlations.

6 Applications

The survey organizes KD applications across LLMs, foundation models, and vision-language systems, emphasizing compression, modality transfer, and trade-offs between teacher access and student generalization.

  • Application Scope: KD applications span LLMs, foundation models, vision transformers, self-supervised learning, diffusion models, and visual recognition.
  • Large Language Models: LLM distillation addresses deployment challenges caused by large parameter counts and many attention blocks by compressing models into smaller language models.
  • Large Language Models: White-box LLM KD transfers logits or intermediate features and generally provides more effective knowledge transfer through access to teacher internals.
  • Large Language Models: Black-box KD enables distillation from closed-source models through in-context learning, instruction following, or chain-of-thought rationales.
  • Distillation Trade-offs: White-box methods are computationally expensive during student training, while black-box methods may have lower generalizability because they rely on teacher-generated data.
  • Foundation Models: Foundation-model KD transfers multimodal or general-purpose knowledge from systems such as CLIP, ChatGPT, DALL-E, and SAM into smaller task-specific models.
  • Foundation Models: Figures 9 and 10 organize foundation models by type and downstream task, and identify architectural points where data is extracted for distillation.
  • Vision-Language Models: For VLMs such as CLIP, KD adapts joint visual-textual representations into compact or task-specific students, reducing computational overhead.

6.3 Distillation in Self-supervised learning

Knowledge distillation supports self-supervised learning through teacher-student representation matching, while diffusion-model distillation reduces iterative denoising costs through several complementary strategies.

  • Self-supervised learning: Self-distillation underpins prominent self-supervised learning methods by training student representations to match those of teacher networks.DINO and DINOv2 use global and local image patches, exponential-moving-average teacher updates, and representation similarity objectives.
  • Self-supervised learning: DINO and DINOv2 apply local-to-global teacher-student matching, making the approach suitable for segmentation tasks.The teacher and student process global and local image patches, with the teacher updated using EMA.
  • Self-supervised learning: BINGO groups images into sample bags and minimizes inter- and intra-sample losses across epochs.It first uses a pretrained self-supervised method to form the bags, then samples from them during training.
  • Self-supervised learning: Distilling pretrained self-supervised networks into smaller networks narrows the supervised-versus-self-supervised performance gap as model size increases.The cited work also examines the effect of adding an MLP layer after the distilled representation.
  • Diffusion distillation: Diffusion distillation addresses expensive hundreds- or thousands-step denoising by transferring teacher outputs and representations to smaller students.The distilled model aims to preserve generation quality with far fewer sampling steps.
  • Diffusion distillation: Sampling-process distillation merges denoising steps or matches a one-step student to the full teacher, while feature methods align internal representations.Adversarial, data-free, and consistency-based methods provide additional routes to acceleration.
  • Diffusion distillation: Diffusion distillation trades acceleration and flexibility against training cost, stability, calibration, or sample-quality preservation.Cross-modal and universal approaches broaden applicability across generation tasks.

6.5 Visual Recognition Distillation

Visual-recognition distillation methods are organized by transferred knowledge and teacher-student scheme, covering feature, logit, similarity, and combined approaches across offline, online, and self-distillation.

  • Organization: Visual-recognition methods are categorized by knowledge source and distillation scheme.The source categories are feature-based, logit-based, similarity-based, and combinations; schemes include offline, online, and self-distillation.
  • Distillation schemes: Self-distillation is included as a distinct scheme within the visual-recognition method landscape.The summary explicitly lists a self-distillation category among the reported approaches.
  • Distillation schemes: Offline distillation is represented across feature, logit, similarity, and combined-source methods.The tabulated examples primarily identify offline schemes, with additional self-distillation entries.
  • Knowledge sources: Feature-based, logit-based, and similarity-based methods appear alongside feature-plus-logit and feature-plus-similarity combinations.The listed methods span multiple source combinations in the visual-recognition summary.

7 Performance Comparison

The survey compares distillation methods across classification, semantic segmentation, and LLM settings, but emphasizes that methodological and reporting differences make broad performance claims difficult.

  • Compared tasks and datasets: Performance comparisons cover CIFAR-100 and ImageNet classification, plus PascalVOC and Cityscapes semantic segmentation.The survey uses these widely employed datasets for comparison.
  • LLM comparison: LLM comparisons distinguish black-box and white-box methods and report compression rate with improvement over the teacher model.Because datasets and models vary substantially, the survey reports more commonly used settings to improve comparability.
  • Compared tasks and datasets: Classification tables report knowledge source, teacher and student architectures, and post-distillation accuracy.They commonly distinguish teacher-student pairs with similar versus different architectures.
  • Compared tasks and datasets: Segmentation results are taken from original papers when available and otherwise from the survey authors’ implementations.Author-generated results are marked with an asterisk.

8 Discussion

The discussion structures KD methods across five dimensions and offers constraint-aware selection guidance, while identifying comparison, knowledge extraction, architecture, capacity, and legal challenges alongside future directions.

  • Survey structure: The survey categorizes methods by distillation sources, schemes, algorithms, modalities, and applications according to each method’s central innovation.Cross-references are provided when methods span multiple categories.
  • Selection guidance: Table 17 maps task constraints to suitable KD sources, schemes, and algorithms as practical design guidance.The authors state that the optimal choice depends on the goals, constraints, and context of each use case.
  • Comparison limits: The survey’s quantitative comparisons remain difficult because hyperparameters, teacher weights, datasets, models, and metrics vary across studies.Reported tables are presented for meaningful comparison, but the survey avoids declaring universal winners.
  • Practical considerations: Using commercial models as teachers may raise copyright concerns in research and deployment.The survey identifies this as a legal limitation requiring careful consideration.
  • Challenges: Knowledge extraction is challenging because logits are unavailable without labels and foundation-model knowledge may require alternative sources.The passage identifies choosing an appropriate knowledge source as a central distillation challenge.
  • Challenges: The suitable distillation scheme depends on context: offline, online, self-distillation, and multi-teacher approaches each impose different requirements.Multi-teacher methods require more memory and GPUs, while training foundation-model or LLM teachers alongside students is infeasible because of model size.
  • Challenges: Large teacher-student capacity gaps and inaccessible intermediate layers in black-box LLM distillation make transferred knowledge difficult for students to absorb.Teacher and student receptive fields can differ substantially when model sizes diverge.
  • Future directions: Future directions emphasize feature-based and adaptive distillation, LLM reasoning transfer, and applications beyond compression.The discussion also identifies privacy, security, intellectual-property, and environmental considerations.

9 Conclusion

The survey organizes knowledge distillation across sources, schemes, algorithms, modalities, applications, and performance comparisons. It emphasizes recent developments including adaptive and contrastive algorithms, 3D inputs, self-supervised learning, diffusion models, foundation models, and LLMs.

  • The survey reviews knowledge distillation from sources, schemes, algorithms, modalities, applications, and performance comparisons.
  • Adaptive and contrastive distillation are presented as increasingly important algorithms.
  • The survey connects knowledge distillation with 3D inputs, particularly point clouds.
  • Applications covered include self-supervised learning, diffusion models, foundation models, and large language models.
  • Self-supervised methods primarily use self-distillation, diffusion models reduce generation steps, and large language models are distilled into smaller versions.
Loading 2503.12067v2…