Does Knowledge Distillation Really Work?

Samuel Stanton; Pavel Izmailov; Polina Kirichenko; Alexander A. Alemi; Andrew Gordon Wilson

Source-linked AI summary

Does Knowledge Distillation Really Work?

Samuel Stanton, Pavel Izmailov, Polina Kirichenko, Alexander A. Alemi, Andrew Gordon Wilson

arXiv:2106.05945v2Jun 10, 2021cs.LGstat.ML

TL;DR

The paper asks whether knowledge distillation truly transfers a teacher’s predictive knowledge to a smaller student, rather than merely improving generalization. It separates fidelity from generalization and examines dataset design and optimization as explanations for mismatch. Distillation often improves generalization, but students frequently fail to match teachers closely, and closer matching does not always improve generalization.

Problem
Knowledge distillation is commonly understood as transferring a large teacher’s knowledge to a small student, but the relationship between teacher matching and student generalization remains unclear.
Method
The paper distinguishes fidelity from generalization and systematically investigates distillation-data choice and optimization as explanations for low student fidelity.
Results
Knowledge distillation often improves student generalization, yet students can retain large predictive discrepancies from teachers even when they have sufficient capacity to match them.
Takeaways & Limitations
Improved student generalization does not guarantee faithful teacher matching, and fidelity can align with generalization for large teachers while behaving differently in self-distillation.
Takeaways & Limitations
The paper focuses on supervised classification with input space X and label space Y.

Abstract

from arXiv · show

Knowledge distillation is a popular technique for training a small student network to emulate a larger teacher model, such as an ensemble of networks. We show that while knowledge distillation can improve student generalization, it does not typically work as it is commonly understood: there often remains a surprisingly large discrepancy between the predictive distributions of the teacher and the student, even in cases when the student has the capacity to perfectly match the teacher. We identify difficulties in optimization as a key reason for why the student is unable to match the teacher. We also show how the details of the dataset used for distillation play a role in how closely the student matches the teacher -- and that more closely matching the teacher paradoxically does not always lead to better student generalization.

1 Introduction

Knowledge distillation often improves student generalization, but students can remain far from their teachers in predictive behavior even when they have sufficient capacity to match them. The paper separates fidelity from generalization and investigates dataset and optimization explanations for this gap.

1 Introduction: Students may produce very different predictions from teachers despite having the capacity to match them perfectly.This discrepancy occurs with modern architectures and motivates studying fidelity separately from generalization.
1 Introduction: When the teacher is a large model or ensemble, improved fidelity can align with improved student generalization.The paper contrasts this setting with self-distillation, where fidelity and generalization can behave differently.
1 Introduction: Fidelity measures how well a student matches teacher predictions, whereas generalization measures performance on unseen in-distribution data.The paper examines why high fidelity is difficult to obtain, including identifiability and optimization hypotheses.
1 Introduction: The study investigates how distillation-data design and optimization affect the student’s ability to match the teacher.These questions organize the paper’s analysis of low distillation fidelity.
1 Introduction: Knowledge distillation can improve generalization while transferring surprisingly limited knowledge from teacher to student.The paper’s central answer is yes for student generalization, but no if success means closely matching the teacher.

2 Related Work

Prior work established knowledge distillation as a way to compress ensembles and improve practical model properties. This paper differs by explicitly separating student fidelity from generalization and treating fidelity as a central object of study.

2 Related Work: Knowledge distillation has been applied to model efficiency, unsupervised domain adaptation, object detection, transparency, and adversarial robustness.These applications primarily emphasize practical student performance or capabilities.
2 Related Work: Earlier studies compressed ensembles of simple models into single networks, while recent work extends ensemble distillation to deep and tabular models.The cited literature includes both shallow teacher ensembles and modern deep-ensemble settings.
2 Related Work: This paper explicitly decouples student fidelity from generalization and systematically investigates why modern distillation often yields low fidelity.The broader literature has focused largely on improving generalization without making this distinction central.
2 Related Work: Related work considers practical distillation modifications, including data augmentation and presenting identical input images to teacher and student.Those approaches focus heavily on achieving the best student top-1 accuracy.

3 Preliminaries

The paper formalizes classifiers, teacher ensembles, the distillation objective, temperature scaling, and separate fidelity and generalization metrics. These preliminaries define fidelity as predictive agreement rather than merely high student accuracy.

3 Preliminaries: A classifier maps inputs to logits whose softmax values define a categorical predictive distribution over classes.The teacher and student are denoted by t and s, respectively.
3 Preliminaries: For an m-component teacher ensemble, component predictive distributions are averaged and represented through combined teacher logits.The main experiments use ensemble sizes m ∈ {1, 3, 5}, with larger ensembles examined in an appendix.
3 Preliminaries: The student minimizes a weighted combination of supervised cross-entropy and a teacher-student distillation loss.The main-text experiments set α = 0, removing the true-label objective to focus on fidelity.
3 Preliminaries: The distillation loss is cross-entropy between teacher and student predictive distributions, scaled by temperature τ, and becomes KL( p̂_t || p̂_s ) when τ = 1.Temperature controls label softness and how student capacity is allocated across teacher predictions.
3 Preliminaries: As τ approaches infinity, the distillation gradient approximately matches the gradient of a squared logit-distance objective.This connects high-temperature distillation to directly matching teacher and student logits.
3 Preliminaries: The distillation data denotes the student’s training data, which may overlap with or extend beyond the teacher’s training data.This terminology distinguishes the student’s data from the data originally used to train the teacher.
3 Preliminaries: Generalization is measured with top-1 accuracy, NLL, and ECE, while fidelity uses top-1 agreement and teacher-to-student KL divergence.The KL-based fidelity metric is sensitive to all labels, unlike top-1 agreement.
3 Preliminaries: Higher agreement with a teacher does not by itself establish better distillation when independently trained models have different generalization levels.Fidelity comparisons must distinguish direct teacher matching from generic agreement caused by predictive performance.

4 Knowledge Distillation Transfers Knowledge Poorly

Knowledge distillation often improves student generalization, but modern students can remain far from their teachers in predictive behavior. The paper finds low fidelity across architectures, datasets, and domains, with optimization difficulties central to the mismatch.

4.1 When is knowledge transfer successful?: Over 99% top-1 test agreement is achievable when an identical LeNet-5 student distills a LeNet-5 teacher on sufficiently extensive MNIST and EMNIST data.In this setting, self-distillation does not improve generalization beyond trial variance.
4.1 When is knowledge transfer successful?: ResNet-56 self-distillation on CIFAR-100 improves fidelity as the distillation dataset grows, but remains far below 99% test agreement.With small distillation datasets, the student can outperform the teacher, so increasing fidelity can decrease student generalization.
4.2 What can self-distillation tell us about knowledge distillation in general?: For large teacher ensembles, fidelity becomes positively correlated with generalization, yet a substantial fidelity gap remains even after adding 50k GAN samples.Higher fidelity therefore does not always produce better generalization when the fidelity improvement remains insufficient.
4 Knowledge Distillation Transfers Knowledge Poorly: Knowledge distillation often improves generalization but frequently transfers limited knowledge, making fidelity important for understanding and improving distillation.Fidelity also matters when the goal is to close the generalization gap between large teachers and smaller students or transfer interpretable behavior.
4.4 Possible causes of low distillation fidelity: Low fidelity is not primarily explained by student capacity or network architecture, and similar findings extend to ImageNet and IMDB sentiment classification.The evidence spans ResNet and VGG networks, larger-scale image data, and text classification.
4.4 Possible causes of low distillation fidelity: The paper distinguishes fidelity from generalization and identifies identifiability and optimization as complementary explanations for poor teacher-student agreement.Identifiability concerns insufficient distillation data, whereas optimization concerns failure to match the teacher even on the training data.

5 Identifiability: Are We Using the Right Distillation Dataset?

The paper tests whether poor fidelity reflects insufficient or inappropriate distillation data. Augmentation and data recycling can slightly improve fidelity, but their effects on generalization and fidelity diverge.

5.1 Should we do more data augmentation?: Data augmentation increases the support of the distillation distribution, providing a practical test of whether identifiability limits student fidelity.The evaluation measures both student fidelity and generalization across augmentation strategies.
5.1 Should we do more data augmentation?: MixUp and GAN4 provide the best generalization, whereas Mixup (τ = 4) provides the best fidelity among the tested policies.The baseline crops-and-flips policy with τ = 4 remains a strong comparator.
5.1 Should we do more data augmentation?: 86% test agreement is achieved by Mixup (τ = 4), while Baseline (τ = 4) achieves 84.5% test agreement.These modest gains over the baseline indicate that augmentation does not yield high fidelity.
5.1 Should we do more data augmentation?: Noise and OOD augmentation decrease performance on all metrics compared with the baseline, showing that augmentation effects extend beyond identifiability.The paper attributes this broader effect to regularization, though the supplied passage truncates the explanation.
5.1 Should we do more data augmentation?: Extensive augmentation slightly improves fidelity, but its benefit is much smaller than logit tempering, making insufficient teacher labels an unlikely primary obstacle.Logit tempering improves fidelity without modifying the data distribution support.
5.2 The data recycling hypothesis: Distilling on D0, the teacher’s training data, yields higher accuracy but worse ECE and fidelity than distilling on unseen D1.Using both D0 and D1 combines favorable attributes and produces the best result.
5.2 The data recycling hypothesis: The best test agreement can remain around 85%, so modifying the distillation dataset slightly improves fidelity without establishing that dataset choice is the main cause of poor fidelity.The results expose a trade-off between obtaining more teacher labels and maintaining train agreement.
5.2 The data recycling hypothesis: Adding examples outside the teacher’s training data makes teacher labels softer and substantially harder for students to match during training.This pattern appears across GAN augmentation, data recycling, and multiple augmentation policies.

6 Optimization: Does the Student Match the Teacher on Distillation Data?

Students often fail to match teachers even on distillation data, and optimization—not only dataset identifiability—limits fidelity. Initializing near the teacher can move optimization into the teacher’s basin, while shared initial weights do not improve function-space agreement.

More distillation data lowers train agreement: Larger distillation datasets improve test agreement but make train agreement harder to achieve, reaching only 95% with 50k synthetic images even at identical capacity.The same experiment reports test agreement below 80% for the largest dataset.
More distillation data lowers train agreement: Extensive augmentation further reduces train agreement because optimization struggles to match the teacher on the augmented distillation set.The authors frame this inability to solve the optimization problem as undermining distillation.
More distillation data lowers train agreement: Combined Augs has the lowest train agreement but better test agreement than policies with higher train agreement, revealing a train-test fidelity trade-off.Adding examples outside the teacher’s training data makes train matching difficult while providing more teacher labels for test matching.
Why is train agreement so low?: Training longer or switching from SGD to Adam does not substantially solve the optimization problem, and Adam reduces fidelity.SGD reaches 83.3% agreement after 5k epochs versus 78.95% after 300 epochs.
Why is train agreement so low?: When initialized far from the teacher (λ ≤0.25), students converge to suboptimal loss basins, whereas λ = 0.375 yields nearly 100% agreement in the teacher’s basin.The initialization is θs = λθt + (1 −λ)θr, with λ = 0 random and λ = 1 teacher initialization.
Why is train agreement so low?: Sharing the initial teacher weights makes activations slightly more similar but leaves function-space results indistinguishable from random initialization.Table 1 evaluates ResNet-20 self-distillation on CIFAR-100 using test-set metrics across 10 trials.
Why is train agreement so low?: The authors identify failure to converge to optimal student parameters as a root cause, while acknowledging that identifiability can also matter.Matching the teacher on all inputs requires at least matching it on the distillation data.

7 Discussion

The discussion separates student generalization from fidelity and emphasizes that optimization remains difficult even when capacity is sufficient. Distillation data quality creates a trade-off: broader data can aid solution identification while making optimization harder.

Key findings: Good student accuracy does not imply good distillation fidelity, including outside self-distillation.Models with the best generalization do not always achieve the best fidelity.
Key findings: For ensemble distillation, fidelity correlates with calibration, although the highest-fidelity student is not always the most accurate.The highest-fidelity student is reported as the best calibrated.
Key findings: Optimization is challenging even when the student has sufficient capacity to match the teacher on the distillation data.The difficulty concerns achieving fidelity, not merely representing the teacher’s solution.
Key findings: Enlarging the distillation dataset beyond the teacher’s training data eases identification of the correct solution but makes optimization harder.This establishes a trade-off between optimization complexity and distillation data quality.
Interpretation: Unlike standard deep learning, knowledge distillation directly aligns good fidelity with solving an exceptionally difficult optimization problem.Standard deep learning can often generalize without solving its multimodal training loss well.
Distillation data: The experiments vary distillation data using standard transformations, mixup, synthetic GAN images, random noise, and out-of-domain SVHN data.These procedures range from data close to the training distribution to data the teacher would not normally encounter.

A.2 Network architectures

The experiments span multiple architectures, datasets, and distillation settings to examine predictive distributions, teacher capacity, ensemble size, and temperature. Results indicate that low fidelity persists despite capacity and architecture choices, while temperature and ensemble size affect student performance and agreement.

Experimental setup: The study also evaluates VGG-16 image classifiers, LeNet networks for MNIST and EMNIST, ResNet-50 networks for ImageNet, and bidirectional LSTMs for IMDB.Synthetic-image experiments use SN-GAN generators and discriminators with three-stage ResNet architectures.
Experimental setup: Experiments use pre-activation ResNets with depths 20, 56, and 110, focusing primarily on CIFAR-100 and also evaluating MNIST and EMNIST.Teachers are trained for 200 epochs, while distilled students are trained for 300 epochs with temperature-scaled teacher-student cross-entropy.
Predictive distributions: Teacher predictive distributions vary with ensemble size, network depth, temperature, and whether examples come from CIFAR-100 train or test data.Figure 8 compares teacher logits across these factors to characterize what students are asked to emulate.
Fidelity mechanisms: The observed low student fidelity is not explained solely by the assumption that the student lacks capacity to emulate the teacher ensemble.The paper presents increasing student capacity as contrary evidence against the capacity explanation.
Temperature and ensemble size: If τ is too small, a student distilled from a 3-component ResNet56 ensemble is no better than one distilled from a single network, while larger τ enables soft-label matching.The gains in student performance and fidelity taper off fairly quickly as ensemble components are added.
Temperature and ensemble size: Student accuracy saturates as teacher ensemble size increases, whereas teacher-student agreement continues improving after accuracy saturation.This pattern is reported for ensembles of up to 12 ResNet56 components, with student accuracy showing little substantial improvement beyond 4 models.

B.3 Detailed results for distillation with heavy data augmentation

This appendix expands the heavy-augmentation distillation experiment across ensemble sizes and examines whether architecture or student capacity explains low fidelity. The results retain the paper’s broader finding that fidelity can remain limited despite additional capacity.

Detailed augmentation results: The detailed augmentation experiment reports generalization and fidelity metrics for ensemble sizes m = 1, 3, and 5, alongside an independent student baseline.The main text reported only m = 5, whereas the appendix adds smaller ensembles for comparison.
Detailed augmentation results: Independent students trained to mimic a different model often achieve nearly as much test agreement with the teacher as students explicitly trained to emulate it.The independent baseline is included to aid interpretation of fidelity metrics.
Student capacity: The appendix presents the capacity findings as evidence against student capacity being the primary explanation for low distillation fidelity.This conclusion is framed as contrary evidence to a possible capacity-based explanation.
Student capacity: Increasing student capacity provides some benefit to accuracy and fidelity when moving from 20 to 56, but further increases do not substantially improve fidelity.The comparison holds teacher capacity fixed while varying student capacity.

C.2 Architecture: is low fidelity an artifact of using ResNets?

VGG experiments test whether low distillation fidelity is specific to ResNet architectures. Their results generally reproduce the ResNet-56 pattern, including different augmentation policies favoring accuracy versus fidelity.

VGG-16 experiment: The VGG-16 experiment distills a 5-component teacher ensemble into a VGG-16 student using varying augmentation policies.The evaluation reports test accuracy, negative log-likelihood, and teacher-student agreement over three runs.
VGG-16 experiment: The VGG results are generally analogous to those obtained with ResNet-56 networks.This supports examining low fidelity beyond the ResNet architecture family.
Augmentation trade-offs: CombAug underperforms other policies on student accuracy but provides the best negative log-likelihood and nearly matches MixUp on teacher-student agreement.MixUp and ColorJit provide the best student accuracy in the VGG comparison.
Augmentation trade-offs: Regardless of augmentation strategy, test agreement does not exceed 85%.The result separates augmentation choices that improve generalization from those that improve distillation fidelity.

C.3 Dataset: does increasing the scale of the dataset increase fidelity?

Experiments across larger-scale and varied datasets examine whether dataset scale or other settings improve fidelity. Across datasets, larger teacher ensembles are more accurate and easier to distill, yet teacher-student disagreement remains substantial.

ImageNet results: For ImageNet, top-1 teacher-student agreement is again in the 80–90% range, while adding ensemble components improves both student accuracy and fidelity.The results also show observable accuracy and fidelity gaps between teacher and student.
Cross-dataset results: Teacher-student disagreement remains at least 10% of test points across the evaluated datasets.The table summarizes this behavior as consistent across all datasets.
Self-distillation: In self-distillation, taking α > 0 can improve student accuracy but does not consistently improve teacher-student agreement.When k > 0, α = 0.1 provides a slight benefit before larger α values negatively affect both accuracy and agreement.

C.4 Data domain: is low fidelity specific to image classification?

The findings extend beyond image classification to LSTM sentiment classifiers, where increasing teacher ensemble size stops improving student accuracy and agreement. Loss-weight experiments likewise show that distillation-loss weighting can improve accuracy without improving agreement, and often harms both.

C.4 Data domain: is low fidelity specific to image classification?: On IMDB sentiment analysis, student accuracy ceases to improve from 3 to 5 teacher ensemble components, alongside a similar lack of improvement in agreement.The teachers’ accuracy continues to improve as ensemble components increase.
C.4 Data domain: is low fidelity specific to image classification?: In self-distillation, α > 0 improves test accuracy but not test agreement.This result is reported for ResNet56 on CIFAR-100 with τ = 1.
C.4 Data domain: is low fidelity specific to image classification?: For teacher ensembles, α = 0.1 offers a slight benefit, while most tested weights deleteriously affect both accuracy and fidelity.The comparison varies the relative weights of LNLL and LKD.