PaperlayerView exact revision on arXiv

Source-linked AI summary

Knowledge Distillation: A Survey

Jianping Gou, Baosheng Yu, Stephen John Maybank, Dacheng Tao

arXiv:2006.05525v7cs.LGstat.ML

TL;DR

Deploying high-capacity deep models on resource-limited devices remains challenging because of their computational and storage demands. This survey organizes knowledge distillation methods, architectures, applications, and comparisons, concluding that distillation improves lightweight models across diverse deep-model settings.

  • Problem

    Deploying large deep models on mobile and embedded devices is challenging because of their computational complexity and storage requirements.

  • Method

    The paper comprehensively surveys knowledge distillation through knowledge types, training schemes, teacher-student architectures, algorithms, performance comparisons, and applications.

  • Results

    Knowledge distillation improves lightweight student models across different deep models, with online and self-distillation also improving deep-model performance.

  • Takeaways & Limitations

    Knowledge distillation supports model compression across diverse deep models and transfers teacher knowledge to improve lightweight models.

  • Takeaways & Limitations

    Online distillation methods generally struggle to address high-capacity teachers, while adaptive teacher-student architecture remains an open research topic.

Abstract

from arXiv · show

In recent years, deep neural networks have been successful in both industry and academia, especially for computer vision tasks. The great success of deep learning is mainly due to its scalability to encode large-scale data and to maneuver billions of model parameters. However, it is a challenge to deploy these cumbersome deep models on devices with limited resources, e.g., mobile phones and embedded devices, not only because of the high computational complexity but also the large storage requirements. To this end, a variety of model compression and acceleration techniques have been developed. As a representative type of model compression and acceleration, knowledge distillation effectively learns a small student model from a large teacher model. It has received rapid increasing attention from the community. This paper provides a comprehensive survey of knowledge distillation from the perspectives of knowledge categories, training schemes, teacher-student architecture, distillation algorithms, performance comparison and applications. Furthermore, challenges in knowledge distillation are briefly reviewed and comments on future research are discussed and forwarded.

1 Introduction

Deep learning achieves strong performance but is difficult to deploy on resource-limited devices, motivating knowledge distillation as a lightweight model-compression approach. This survey organizes knowledge distillation by knowledge types, training schemes, architectures, algorithms, comparisons, applications, challenges, and future directions.

  • Motivation: Deep learning has enabled major successes across computer vision, reinforcement learning, and natural language processing.
  • Motivation: Deploying large deep models on mobile and embedded devices is challenging because of limited computational capacity and memory.
  • Knowledge Distillation: Knowledge distillation transfers information from a large model or ensemble into a small model without a significant accuracy drop.
  • Open Questions: Despite practical success, knowledge distillation has limited theoretical and empirical understanding, although existing work identifies conditions influencing student learning and distillation success.
  • Survey Scope: The survey reviews knowledge distillation from perspectives including knowledge categories, training schemes, teacher-student architectures, algorithms, performance comparisons, applications, challenges, and future research.

2 Knowledge

Knowledge distillation categorizes transferable knowledge into response-based, feature-based, and relation-based forms. These approaches respectively mimic teacher outputs, intermediate representations, or relationships among layers and data samples, while feature-based transfer still faces layer-selection and representation-matching challenges.

  • Response-based knowledge: Response-based knowledge distillation mimics the teacher’s final neural response, typically using logits or soft targets, and is simple yet effective for model compression.Soft targets are controlled by a temperature factor T and contain informative dark knowledge; the student also uses a cross-entropy loss with ground-truth labels.
  • Response-based knowledge: Response-based distillation can support varied predictions, including logits with bounding-box offsets in object detection, but usually lacks intermediate-level supervision.Its reliance on the last layer limits access to teacher information from intermediate layers.
  • Feature-based knowledge: Feature-based knowledge transfers intermediate feature maps by directly or indirectly matching teacher and student representations, extending supervision beyond the final layer.Transformation functions may be applied when teacher and student feature maps have different shapes, and similarity functions compare the transformed representations.
  • Feature-based knowledge: Feature-based transfer provides useful student supervision, but selecting teacher hint layers and student guided layers and matching differently sized representations remain open challenges.The passage explicitly identifies layer selection and size differences between hint and guided layers as unresolved issues.
  • Relation-based knowledge: Relation-based knowledge distillation captures relationships among feature maps, layers, or data samples rather than transferring only outputs from individual teacher layers.Examples include flow-of-solution-process matrices based on inner products between feature maps and instance relationship graphs; relation knowledge also includes structured data knowledge and privileged input information.

3 Distillation Schemes

Knowledge distillation training schemes are categorized as offline, online, or self-distillation according to whether the teacher is updated with the student. Offline distillation uses a pre-trained teacher, online distillation trains teacher and student together, and self-distillation uses the same network for both roles.

  • Training-scheme taxonomy: Distillation schemes comprise offline, online, and self-distillation, classified by whether the teacher model is updated simultaneously with the student.These schemes are presented as the main training categories for teacher and student models.
  • Offline distillation: Offline distillation transfers logits or intermediate features from a pre-trained teacher to guide student training in a two-stage process.The teacher is trained first, then extracts knowledge for student optimization during distillation.
  • Offline distillation: Offline methods are simple to implement and can cache knowledge from teachers trained with different software packages or on different machines.Their main advantage is implementation simplicity, while student training is usually efficient under teacher guidance.
  • Offline distillation: Offline distillation uses one-way transfer and two-phase training but retains the expensive high-capacity teacher, capacity gap, and student reliance on the teacher.The teacher’s complex training cost cannot be avoided, even though student training is typically efficient.
  • Online distillation: Online distillation is a one-phase end-to-end scheme with efficient parallel computing, but existing methods often fail to address high-capacity teachers.It was proposed especially when a large-capacity, high-performance teacher is unavailable.
  • Self-distillation: Self-distillation uses the same network as teacher and student and can be regarded as a special case of online distillation.Methods include distilling knowledge from deeper network sections into shallower sections.

4 Teacher-Student Architecture

Teacher-student architecture determines how knowledge is acquired and transferred, yet fixed model setups can create a capacity gap. The section reviews relationships between teacher and student structures and motivates adaptive architectures that jointly consider student design and knowledge transfer.

  • Architecture relationship: Teacher-student architecture provides the generic carrier for knowledge transfer, while network design influences the quality of knowledge acquisition and distillation.The survey notes that teacher and student design determines how effectively knowledge is transferred.
  • Architecture relationship: Pre-fixed teacher and student sizes and structures can cause a model capacity gap, while the specific architectural rationale and design remain insufficiently studied.The section frames its discussion around the relationship between teacher and student structures.
  • Architecture relationship: Distillation typically transfers knowledge from deeper and wider teachers to shallower and thinner students because neural-network complexity mainly arises from depth and width.Students are commonly simplified teacher versions with fewer layers and fewer channels in each layer.
  • Architecture adaptation: Teacher assistants and residual learning have been introduced to reduce the training gap between teacher and student models.The assistant structure learns the residual error in residual-learning approaches.
  • Architecture adaptation: Adaptive teacher-student learning is necessary to match a small student to a large teacher, motivating joint neural architecture search over student structure and knowledge transfer.The search is conducted under guidance from the teacher model.

5 Distillation Algorithms · 5.1 Adversarial Distillation

The section reviews knowledge-distillation algorithms that match teacher and student responses, features, or representation distributions. Adversarial distillation extends this transfer through GAN-based data generation, discriminators, online joint optimization, and GAN compression.

  • 5 Distillation Algorithms: Distillation algorithms directly match teacher and student responses, features, or representation distributions in feature space.These provide basic mechanisms for transferring knowledge between teacher and student models.
  • 5.1 Adversarial Distillation: Adversarial distillation addresses the difficulty of imperfect teacher learning and limited student capacity by using adversarial learning to improve student imitation.The motivation arises because teachers may not perfectly learn the true data distribution and students cannot accurately mimic larger teachers.
  • 5.1 Adversarial Distillation: Adversarial distillation methods using GANs fall into three categories based on generators, discriminators, and online teacher-student optimization.Figure 10 summarizes generator-based data production, discriminator-based imitation, and discriminator-enhanced online distillation.
  • 5.1 Adversarial Distillation: The first category trains an adversarial generator to produce synthetic data used directly as training data or to augment the training dataset.The teacher may also serve as the discriminator in this GAN-based arrangement.
  • 5.1 Adversarial Distillation: Generator-based distillation uses generated samples G(z) and a distillation loss LG, such as cross entropy or KL divergence, to match predicted and ground-truth probability distributions.G(z) denotes samples generated from random input vector z, while LG forces distributional agreement.
  • 5.1 Adversarial Distillation: The second category introduces discriminators that distinguish student from teacher samples using logits or features, with variants using unlabeled data, multiple discriminators, and intermediate supervision.This discriminator-based setup is intended to make the student better match the teacher.
  • 5.1 Adversarial Distillation: The third category jointly optimizes teacher and student networks online, while knowledge distillation can also compress a larger GAN teacher into a smaller GAN student.In the online setting, teacher and student are optimized together during each iteration.
  • 5.1 Adversarial Distillation: Overall, GANs enhance student learning, joint GAN and KD can generate valuable data despite unusable or inaccessible data, and KD can compress GANs.These summarize the principal benefits identified for adversarial distillation.

5.2 Multi-Teacher Distillation

Multi-teacher distillation transfers diverse response- and feature-based knowledge from multiple teacher networks to train a versatile student model. Existing schemes average teacher responses, combine logits with intermediate features, select teachers per iteration, or add teacher branches, while effective knowledge integration remains an open problem.

  • Multi-Teacher Distillation: Multiple teacher architectures provide useful knowledge, with teachers used individually or integrally during student training and averaged responses serving as a simple supervision signal.The typical framework uses a large teacher model or ensemble, averaging responses from all teachers.
  • Multi-Teacher Distillation: Multi-teacher distillation commonly transfers logits and feature representations, including intermediate features that encourage dissimilarity among training samples.You et al. (2017) combined averaged logits with intermediate-layer features.
  • Multi-Teacher Distillation: Schemes assign response-based and feature-based knowledge to separate teachers, randomly select one teacher per iteration, or add teacher branches to mimic intermediate features.These approaches use two teachers, teacher-pool sampling, or additional student branches for feature transfer.
  • Multi-Teacher Distillation: Multi-teacher knowledge distillation provides rich knowledge and can tailor a versatile student model, but effectively integrating knowledge types from multiple teachers requires further study.The diversity of knowledge from different teachers is the stated source of this benefit.

5.3 Cross-Modal Distillation

Cross-modal distillation transfers knowledge between modalities when some modality data or labels are unavailable. It includes paired-sample and multimodal knowledge-transfer scenarios, but remains challenging when modality gaps prevent paired samples.

  • Cross-modal knowledge transfer is important because data or labels for some modalities may be unavailable during training or testing.
  • Gupta et al. transfer RGB teacher knowledge to a student using a new unlabeled modality, such as depth images or optical flow.The method uses unlabeled paired RGB-depth samples, with teacher features from RGB images supervising student training.
  • Knowledge distillation supports cross-modal applications including visual question answering and transfer from textual to visual modalities.Do et al. distill a trilinear-interaction image-question-answer teacher into a bilinear-interaction image-question student, while Passalis and Tefas use probabilistic distillation for textual-to-visual transfer.
  • Cross-modal distillation performs well in visual recognition, but modality gaps make knowledge transfer challenging when paired samples are unavailable.

5.4 Graph-Based Distillation

Graph-based distillation extends knowledge transfer beyond individual instances by modeling intra-data relationships or using graphs to control transfer. Its central challenge is constructing graphs that properly represent informative data structure.

  • Graph-Based Distillation: Graph-based methods explore intra-data relationships by using graphs as carriers of teacher knowledge or as mechanisms that control knowledge transfer.Most distillation algorithms transfer individual instance knowledge, whereas graph-based approaches model relationships within the data.
  • Graph-Based Distillation: Graphs transfer structure knowledge through logits, intermediate features, sample relations, activation similarities, or locality-preserving objectives.Examples include logits and representation graphs, high-dimensional sample relationships, multi-head graphs over CNN features, edge matching, and similarity matrices.
  • Graph-Based Distillation: Some methods instead use directed distillation graphs to model modality relationships and control transfer of privileged information from a source domain.Vertices represent modalities, while directed edges indicate connection strength between modalities.
  • Graph-Based Distillation: Properly constructing graphs that model data structure remains a challenging research problem.Graph-based distillation can transfer informative structural knowledge, but graph construction is not yet fully resolved.

5.5 Attention-Based Distillation · 5.6 Data-Free Distillation

Attention-based distillation transfers feature-embedding knowledge through attention maps or confidence rules. Data-free distillation addresses unavailable training data by generating or reconstructing transfer data, but producing high-quality, diverse data remains challenging.

  • 5.5 Attention-Based Distillation: Attention-based knowledge distillation uses attention mechanisms to improve student-network performance because attention reflects convolutional-neural-network neuron activations.The cited methods include Zagoruyko and Komodakis (2017), Huang and Wang (2017), Srinivas and Fleuret (2018), Crowley et al. (2018), and Song et al. (2018).
  • 5.5 Attention-Based Distillation: Attention transfer defines attention maps for neural-network feature embeddings and transfers feature-embedding knowledge through attention-map functions.This describes the core mechanism of attention transfer.
  • 5.5 Attention-Based Distillation: Song et al. (2018) proposed a different attentive distillation method that uses an attention mechanism to assign different confidence rules.Unlike attention maps, this method transfers knowledge through confidence rules.
  • 5.6 Data-Free Distillation: Data-free distillation methods address unavailable training data caused by privacy, legality, security, and confidentiality concerns by newly or synthetically generating data.As the term implies, these methods use no training data.
  • 5.6 Data-Free Distillation: A generic data-free-distillation framework transfers knowledge using newly generated or reconstructed data rather than an available training dataset.Figure 14 presents this generic framework.
  • 5.6 Data-Free Distillation: Transfer data can be generated by a GAN or reconstructed from teacher-network layer activations or layer spectral activations.DeepInversion is also identified among data-free knowledge-distillation methods, while few-shot distillation uses limited labelled data.
  • 5.6 Data-Free Distillation: Data-free distillation usually synthesizes data from feature representations of a pretrained teacher model, but generating high-quality, diverse data to improve generalizability remains challenging.The passage identifies this challenge as central despite data-free distillation’s potential when data is unavailable.

5.7 Quantized Distillation

Quantized distillation combines network quantization with knowledge distillation to transfer knowledge from high-precision or complex teachers to small, low-precision students. Recent work also explores self-distillation, in which the teacher shares the student’s parameters to improve quantized models.

  • Quantized Distillation: Network quantization converts high-precision networks, such as 32-bit floating point, into low-precision networks, such as 2-bit and 8-bit, reducing computation complexity.Knowledge distillation trains a small model to achieve performance comparable to a complex model.
  • Quantized Distillation: Quantized distillation transfers knowledge from a high-precision teacher network to a small, low-precision student network, including weight-quantized students.Mishra and Marr (2018) call their quantized KD method the “apprentice.”
  • Quantized Distillation: Self-distillation improves quantized deep models by using a teacher that shares the student’s model parameters.This training scheme is attributed to Boo et al. (2021).

5.8 Lifelong Distillation · 5.9 NAS-Based Distillation

Lifelong distillation uses knowledge distillation to preserve and transfer learned knowledge while addressing catastrophic forgetting across new tasks. NAS-based distillation searches for suitable student architectures and also uses distillation to improve neural architecture search efficiency.

  • 5.8 Lifelong Distillation: Lifelong learning accumulates previously learned knowledge and transfers it into future learning, while knowledge distillation helps preserve and transfer knowledge without catastrophic forgetting.The survey describes lifelong learning as including continual learning, continuous learning, and meta-learning.
  • 5.8 Lifelong Distillation: Recent lifelong-learning-based knowledge distillation variants include global distillation, knowledge distillation-based lifelong GAN, and multi-model distillation.
  • 5.8 Lifelong Distillation: Table 5 compares different knowledge distillation methods on CIFAR10, with ↑ denoting improvement over the corresponding baseline student model.
  • 5.8 Lifelong Distillation: These KD-based methods extract learned knowledge and teach student networks on new tasks.
  • 5.9 NAS-Based Distillation: Neural architecture search automatically identifies deep neural models and adaptively learns appropriate structures, addressing the importance of student architecture for successful knowledge transfer.
  • 5.9 NAS-Based Distillation: NAS-based distillation includes oracle-based and architecture-aware searches, while distillation also improves search efficiency through methods such as AdaNAS, TGSA, and one-shot NAS.The survey also mentions NAS with distilled architecture knowledge among these approaches.

6 Performance Comparison

This section compares typical knowledge distillation methods on CIFAR10 and CIFAR100 using classification accuracies derived from original papers. The comparison concludes that knowledge distillation effectively and efficiently compresses deep models while improving performance across varied models and training schemes.

  • Experimental setup: Typical knowledge distillation methods are compared on CIFAR10 and CIFAR100 image classification datasets.Both datasets contain 32 × 32 RGB images, with 50000 training images and 10000 testing images.
  • Experimental setup: Classification accuracies are directly derived from original papers and reported across knowledge types, distillation schemes, and teacher/student structures.Teacher and individually trained student accuracies are given in parentheses for comparison.
  • Observations: Knowledge distillation applies across different deep models and enables model compression for those models.The performance comparison summarizes these observations across the reported methods.
  • Observations: Online collaborative learning and self-knowledge distillation can significantly improve deep-model performance.The section cites multiple methods as examples of both online and self-knowledge distillation.
  • Conclusion: Overall, knowledge distillation is an effective and efficient technique for compressing deep models.This conclusion follows directly from the performance comparison of different knowledge distillation methods.

7 Applications

Knowledge distillation is widely applied to compress and accelerate models across visual recognition, NLP, and speech recognition. Across these domains, teacher-student transfer supports lightweight deployment while preserving effective task performance and handling diverse data modalities.

  • Visual recognition: In face recognition, distillation targets both efficient deployment and competitive recognition accuracy, including low-resolution settings where high-resolution teachers guide low-resolution students.Methods transfer information from informative teacher neurons or hint layers and use teacher weighting or recursive student initialization.
  • Visual recognition: Knowledge distillation supports efficient teacher-student learning across diverse visual recognition tasks, including image classification, face recognition, low-resolution recognition, and action recognition.Lightweight students are trained under high-capacity teachers, while flexible architectures exploit cross-modality, multi-domain, multi-task, and low-resolution data.
  • Natural language processing: Knowledge distillation produces lightweight language models for numerous NLP tasks, including neural machine translation and natural language understanding.Multiple or ensemble NMT teachers can supervise students, while distillation is also used to create lightweight BERT variations.
  • Natural language processing: Teacher-student transfer enables efficient multilingual language processing by transferring knowledge from multilingual models to smaller students.The survey identifies multilingual task solving as an effective use of teacher-student knowledge transfer.
  • Speech recognition: In speech recognition, lightweight students support real-time responses, limited-resource use, and high recognition accuracy, with RNN teachers preserving temporal knowledge from acoustic data.Sequence-level knowledge distillation is also identified as a speech-recognition application.

8 Conclusion and Discussion

The paper reviews knowledge distillation across knowledge, schemes, architectures, algorithms, performance, and applications, and identifies major challenges in knowledge quality, transfer, architecture design, and theoretical understanding. It also highlights opportunities for hybrid compression methods and broader applications of teacher-student knowledge transfer.

  • The survey comprehensively reviews knowledge distillation from perspectives including knowledge, distillation schemes, teacher-student architectures, algorithms, performance comparison, and applications.
  • Knowledge distillation requires extracting rich knowledge from the teacher and transferring it to guide student training, with challenges spanning knowledge quality, distillation types, architectures, and theory.
  • Most methods combine response-, feature-, and relation-based knowledge, motivating analysis of each type’s influence and their complementary interactions.
  • Teacher-student architecture design remains underinvestigated, although structural relationships and capacity gaps can significantly influence distillation performance.
  • Efficient lightweight deployment may require hybrid compression combining knowledge distillation with other techniques and determining their proper application order.
  • Knowledge distillation extends beyond model compression to privacy and security, adversarial attacks, cross-modalities, multiple domains, catastrophic forgetting, and traditional machine learning.
Loading 2006.05525v7…