Knowledge Distillation in Deep Learning and its Applications

I. INTRODUCTION

• Knowledge distillation addresses deployment difficulties caused by large deep-learning models on end devices.The paper distinguishes it from transfer learning: distillation trains a smaller model for the same task, whereas transfer learning initializes a model for a similar task [1],.Source: §I · p. 1
• The paper proposes a comprehensive survey of recent knowledge-distillation methods because earlier coverage was limited.The cited 2017 survey included other compression approaches and was not specific to knowledge distillation [3].Source: §I · p. 1
• The proposed distillation metric compares methods using model-size reduction and accuracy change.The paper identifies these as its main comparison criteria among possible measures such as processing time.Source: §I · p. 1
• The paper is organized around background, the metric, surveyed approaches, discussion and outlook, and conclusions.Source: §I · p. 1

II. BACKGROUND

• Distillation generally seeks smaller students that retain teacher accuracy, although some methods use same-size networks to accelerate training.The paper notes that students should outperform same-size baselines trained without teachers [1].Source: §II · pp. 1–2
• The metric compares distillation methods by student-size reduction and accuracy relative to the teacher, allowing designers to judge fit against system requirements.The paper says there is no universally best size reduction; the appropriate tradeoff depends on the system.Source: §II · p. 2
• Surveyed approaches include single-teacher-to-single-student, multiple-teacher-to-single-student, and student-ensemble designs.Source: §II · p. 2

III. DISTILLATION METRIC

• The distillation score uses student-to-teacher size and accuracy ratios, where smaller size ratio and accuracy ratio nearer 1 are desirable.The score is denoted DS; the student and teacher size and accuracy terms are defined in the paper.Source: §III · p. 2
• The weight α balances size against accuracy according to system requirements; this survey assigns both criteria equal weights of 0.5.Source: §III · p. 2
• When student accuracy exceeds teacher accuracy, the accuracy ratio is greater than 1 and can make the score negative, indicating a small student with better accuracy.The paper says this outcome is uncommon and may occur, for example, with an ensemble of student models.Source: §III · p. 2

IV. SURVEY

• The survey divides methods into those using teacher soft labels directly and those transforming soft labels before training students.Here, soft labels refer to outputs of hidden layers.Source: §IV · p. 2
• The survey compares student size reductions and accuracy against corresponding teachers, but some studies omit the details needed for those comparisons.Source: §IV · p. 2
• Distillation scores appear in Tables I and II, using each paper's best results to address differing datasets and method variants.Source: §IV · p. 2

A. Techniques Using Soft Labels to Train Student Models

• Gao et al. distilled a student backbone before fitting it on new data, reporting 80.5% verification accuracy on IJB-A after training the teacher on CASIA-WebFace.Their approach trains the backbone to mimic teacher feature extraction, then fixes the backbone while fitting the student on ground-truth data.Source: A. Techniques Using Soft Labels to… · p. 3
• Liu et al. distilled CNN logits into decision trees that scored 1–5% better than similar-depth trees trained without a deep-learning teacher.The student and teacher accuracies remained separated: 86.55% versus 99.25% on MNIST, and 73.42% versus 86.62% on connect-4.Source: A. Techniques Using Soft Labels to… · p. 3
• Furlanello et al.’s Born-Again Neural Networks trained successive students on labels and prior-student predictions, with CIFAR-100 error falling from 16.87% to 14.9% [14].Predictions from the student models were averaged.Source: A. Techniques Using Soft Labels to… · p. 3
• Quantized distillation and differentiable quantization each reduced model size by 52.87%; soft-label approaches also covered U-Net and speech-recognition students [16].Differentiable quantization avoids the full-precision student used to compute gradients in quantized distillation; Wang et al. report 78.99% size reduction, while Kurata and Audhkhasi report 55.07%.Source: A. Techniques Using Soft Labels to… · pp. 3–4
• Munim et al. trained Seq2Seq students using teacher soft labels and k-best hypotheses; student-small reduced WER by 22.2% against its same-size baseline.Student-mid reduced WER by 9.63% against its baseline.Source: A. Techniques Using Soft Labels to… · p. 4
• Jin et al. progressively trained students across selected teacher checkpoints and reported 70.85% accuracy on CIFAR-100 and 68.21% on ImageNet.The student was a 3.4-million-parameter MobileNetV2, compared with a 23.521-million-parameter ResNet-50 teacher.Source: A. Techniques Using Soft Labels to… · p. 4
• Heo et al. transferred teacher activation boundaries using an activation-transfer loss and connector function; their student reached 74.10% accuracy on MIT scenes.The teacher and student achieved 69.78% and 64.93%, respectively.Source: A. Techniques Using Soft Labels to… · pp. 4–5
• Munim et al. achieved the greatest size reduction at 89.88%, while Heo et al. had the strongest accuracy maintenance, with a 6.19% increase.The survey’s Table I comparison identifies Munim et al. as having the best distillation score as well.Source: A. Techniques Using Soft Labels to… · p. 5

B. Techniques Using Transformation of Soft Labels to Train Student Models

• Lopes et al. reconstructed training data from teacher activation metadata; layer-pairs spectral activation records performed best among the records tested [30].The student models used 50% fewer parameters than their teachers.Source: B. Techniques Using Transformation of… · p. 5
• Kimura et al. augmented 1.25K samples to 10K using inducing points, achieving 44.1% MNIST and 44.8% fashion-MNIST accuracy [32].Direct training with the same model achieved 37.9% and 39.3%, respectively.Source: B. Techniques Using Transformation of… · p. 5
• Nayak et al. synthesized training samples from the teacher’s SoftMax-space distribution, training students without the original dataset or metadata [34].The reported student models had about 40% fewer parameters than their teachers.Source: B. Techniques Using Transformation of… · p. 5
• Yim et al. trained students first to mimic matrices between teacher and student activations, then normally; training time fell 34.37% on CIFAR-10.The student’s accuracy was 92.28%, compared with the teacher’s 92%.Source: B. Techniques Using Transformation of… · p. 5
• Fukuda et al. trained a student on separate outputs from multiple teachers; the compact CNN reached 13.2% WER versus 15.1% for its baseline.In a separate experiment, its 11.2% WER was between the LSTM’s 11.7% and VGG’s 10.5%.Source: B. Techniques Using Transformation of… · pp. 5–6
• Pintea et al. mapped several teacher blocks to one student block via recurrence relations; their strongest size reduction was 94.09%, with a 5.307% accuracy-score reduction.They evaluated three student variants on CIFAR-10.Source: B. Techniques Using Transformation of… · p. 6
• Zhou et al. jointly trained booster and lightweight networks with shared backbone layers; He et al. compressed teacher knowledge, while Wu et al. averaged teachers and Liu et al. used instance-relation graphs.Zhou et al. also used a gradient-block scheme to prevent booster-specific parameters from updating during the joint objective’s backpropagation.Source: B. Techniques Using Transformation of… · pp. 6–7
• Pintea et al. achieved the best size reduction at 94.09% and best distillation score, while Kimura et al. had the largest accuracy increase at 10.526% [32].These are the survey’s reported Table II comparisons.Source: B. Techniques Using Transformation of… · p. 7

• Distillation scores across Tables I–II averaged 0.1804 with a standard deviation of 0.07835; scores below 0.2 generally indicated good size reduction and accuracy maintainability.Six scores were better than average, with most corresponding algorithms achieving more than 70% size reduction and less than 5% accuracy reduction.Source: §V · p. 7
• Low size reductions can yield relatively high distillation scores despite strong accuracy retention, as Wu et al. retained accuracy well but reduced size by 58.31%.Source: §V · p. 7
• Authors should report student size and accuracy changes against teachers, and compare students with same-sized models trained from scratch.Source: §V · p. 7
• Polino et al. [16] proposed reinforcement learning to optimize student models for memory and processing-time constraints.The paper notes that it is unclear how to develop students that meet those constraints.Source: §V · p. 7
• Only two surveyed works considered NLP, while transformer-based language-model distillation remains underinvestigated despite models having hundreds of millions of parameters.The paper calls for studying distillation on transformer language models to develop relatively small, high-performance models.Source: §V · p. 7
• Table II covers soft-label transformation approaches and reports the largest student size reduction and the dataset with the lowest accuracy reduction when multiple options exist.Its baselines match student-model size and are trained without teacher models.Source: Table II · p. 8

• Some surveyed methods reduce model size by more than 80%,, while others achieve around 50% reduction while maintaining teacher accuracy, [16].Source: §VI · p. 8
• Some distillation approaches produce students with better accuracy than their teachers [14],.Source: §VI · p. 8
• The proposed distillation metric compares algorithms using achieved student sizes and accuracy scores, alongside contexts such as limited data, interpretability, and combining transfer learning with distillation.Source: §VI · p. 8
• Designers are advised to start with simple methods using relatively few teacher and student layers, then progress to complex methods as domain-specific intuitions develop.The paper says this can help designers eliminate some methods without trying them and use the distillation metric to assess evaluations.Source: §VI · p. 8