Source-linked AI summary
On the Efficacy of Knowledge Distillation
Jang Hyun Cho, Bharath Hariharan
TL;DR
Knowledge distillation aims to help smaller networks learn from larger teachers, but the paper evaluates when teacher and student architectures make that transfer effective. It finds that larger, more accurate teachers often fail to improve students because of capacity mismatch, while early-stopping teacher training mitigates the problem across datasets and models.
Problem
Knowledge distillation seeks to reduce the computational burden of large visual-recognition networks by transferring information from larger teachers to smaller students.
Method
The paper evaluates knowledge distillation across student and teacher architectures, examines capacity mismatch and sequential distillation, and tests early-stopping strategies.
Results
Larger, more accurate teachers do not necessarily produce better students because low-capacity students often cannot mimic them; early-stopping teacher training improves distillation efficacy across datasets and models.
Takeaways & Limitations
Knowledge distillation is not universally effective when student capacity is too low, but stopping teacher training early can produce solutions more amenable to students.
Takeaways & Limitations
The paper hypothesizes that on challenging ImageNet, low-capacity students may be unable to minimize both the training loss and the knowledge-distillation loss.
Abstract
from arXiv · showhide
In this paper, we present a thorough evaluation of the efficacy of knowledge distillation and its dependence on student and teacher architectures. Starting with the observation that more accurate teachers often don't make good teachers, we attempt to tease apart the factors that affect knowledge distillation performance. We find crucially that larger models do not often make better teachers. We show that this is a consequence of mismatched capacity, and that small students are unable to mimic large teachers. We find typical ways of circumventing this (such as performing a sequence of knowledge distillation steps) to be ineffective. Finally, we show that this effect can be mitigated by stopping the teacher's training early. Our results generalize across datasets and models.
1. Introduction
Knowledge distillation is intended to make smaller models learn from larger teachers, but its generality across architectures had not been empirically established. This paper evaluates the factors governing distillation and finds that teacher accuracy and size do not reliably predict student performance.
- Knowledge distillation supervises a smaller student with a larger teacher’s soft probabilities in addition to class labels.The paper motivates these probabilities as containing more information than labels alone.
- Prior work assumed that conclusions from a few carefully chosen architectures would generalize across student and teacher architectures.The authors identify a missing empirical answer about whether any student can learn effectively from any teacher.
- Teacher accuracy is a poor predictor of student performance, because larger, more accurate teachers do not necessarily make better teachers.As teacher capacity and accuracy increase, students often struggle to emulate the teacher.
- Sequential knowledge distillation does not mitigate the difficulty of matching a high-capacity teacher.The paper reports that prior sequence-based solutions fail to resolve the student’s emulation problem.
- Early stopping regularizes teacher training and improves distillation efficacy across the evaluated settings.The authors stop teacher training early and stop distillation near convergence so the student can fit the training loss better.
2. Related Work
Related work treats knowledge distillation as a broadly applicable compression and accuracy-improvement technique, including repeated distillation and applications beyond image classification. The paper emphasizes that prior evidence was not sufficiently exhaustive and that these methods can depend strongly on student capacity.
- Knowledge distillation trains smaller, cheaper students to mimic larger teachers and has been applied to deep neural networks.The authors note that a thorough empirical evaluation was missing despite earlier experiments on small datasets.
- Prior research explored feature-map losses, bidirectional distillation, model averaging, and other changes intended to improve distillation quality.
- Sequential knowledge distillation repeatedly transfers knowledge between models and can be used as an alternative to ensembling.
- Sequential distillation underperforms an ensemble trained from scratch and can reduce a network’s ability to act as a teacher.The paper reports that these methods are especially ineffective when student capacity is limited or the dataset is complex.
- Knowledge distillation has also been used for sequence modeling, semi-supervised learning, domain adaptation, and multimodal learning.Its broad application range makes exhaustive evaluation more important.
3. Background: Knowledge distillation
Knowledge distillation transfers information from a teacher’s softened output distribution to a student alongside ordinary label supervision. The student combines classification and distillation losses, with temperature and weighting controlling the training objective.
- A teacher produces class scores that are converted into a probability distribution for each input image.
- Soft probabilities can reveal relationships among classes, such as proximity to a decision boundary, beyond the information in a single class label.
- Temperature scaling softens the teacher probabilities so that otherwise peaked distributions can carry more information.The temperature parameter satisfies τ > 1.
- The student produces its own softened distribution and is trained with a linear combination of cross-entropy and knowledge-distillation losses.
- Popular distillation settings use τ ∈ {3, 4, 5} and α = 0.9.
4. Methods
The experiments hold the student fixed while varying teacher capacity on CIFAR10 and ImageNet. They use standard SGD-based training and consistent knowledge-distillation hyperparameters across the experimental settings.
- The study evaluates multiple teachers of varying capacity while keeping the student architecture fixed on CIFAR10 and ImageNet.
- CIFAR10: CIFAR10 models are trained for 200 epochs with momentum 0.9, an initial learning rate of 0.1, and scheduled learning-rate drops.The experiments repeat each setting five times and report median, mean, and standard deviation.
- CIFAR10: CIFAR10 uses temperature τ = 4, loss weight α = 0.9, and attention-transfer weight β = 1000.
- ImageNet: ImageNet experiments use SGD with Nesterov momentum 0.9, initial learning rate 0.1, weight decay 1 × 10^-4, and learning-rate drops every 30 epochs.
- ImageNet: ImageNet uses the same τ = 4, α = 0.9, and β = 1000 hyperparameters and evaluates ResNet architectures.
5. Results
Across CIFAR10 and ImageNet, larger and more accurate teachers do not necessarily improve student performance because capacity mismatches make them difficult to mimic. Early-stopping distillation and teacher training improves results, whereas sequential distillation does not reliably solve the problem.
- Bigger models are not better teachers: As teachers become larger and more accurate, student accuracy rises and then falls on CIFAR10 and decreases on ImageNet.
- Analyzing student and teacher capacity: Higher KD error or loss for the largest teachers indicates that smaller students struggle to mimic them because of mismatched capacity.
- Distillation adversely affects training: Early-stopped knowledge distillation improves all tested teachers on ImageNet and can outperform similarly sized students trained from scratch.
- Distillation adversely affects training: Early-stopping does not remove the relationship between larger teachers and worse student mimicry, as test-set KD loss still increases with teacher size.
- The efficacy of repeated knowledge distillation: Sequential distillation is ineffective: step-wise large-to-medium-to-small distillation performs almost exactly like direct large-to-small distillation.
- Early-stopped teachers make better teachers: On ImageNet, sequential distillation is also ineffective, while early-stopped teachers and a fully trained small teacher achieve the strongest reported results.
- Early-stopped teachers make better teachers: Early-stopping teacher training benefits both WRN16-1 and WRN28-1 students compared with their optimal fully trained teachers.
- Early-stopped teachers make better teachers: The benefits of early stopping generalize across different student and teacher model families and also improve attention transfer.
6. Conclusion
The paper finds that knowledge distillation is limited by student capacity rather than being universally effective. Early stopping teacher training mitigates this mismatch, with benefits reported across datasets and transfer learning.
- Knowledge distillation cannot succeed when student capacity is too low to mimic the teacher.
- Stopping teacher training early produces a solution more amenable to the student.
- The approach benefits CIFAR10, ImageNet, and transfer learning on Places365.
- The authors conclude that further research is needed before distillation becomes a general practical approach.
On the Efficacy of Knowledge Distillation - Supplementary Materials
The supplementary materials report that early-stopped DenseNet teachers substantially improve WideResNet16-1 students compared with training from scratch.
- Early-stopped DenseNet teachers produce WideResNet16-1 students that perform better by a large margin than students trained from scratch.
- The supplement identifies the setup as WideResNet16-1 trained with different DenseNet teachers.
- The table’s top row provides the student trained from scratch as the comparison baseline.
7. More Results on CIFAR10
Additional CIFAR10 experiments vary teacher training schedules and show that early-stopped teachers consistently outperform fully trained teachers for the tested students.
- The supplementary CIFAR10 experiments test early-stopping across varying teacher widths.
8. Details on ImageNet Experiments
Supplementary ImageNet and CIFAR10 results reinforce the paper’s conclusions: distillation can harm limited-capacity students, while early-stopped teachers generally improve student performance across architectures and schedules.
- ImageNet experiments: Figure 7 compares student accuracy plots and illustrates the harming effect of distillation.
- CIFAR10 results: Early-stopped DenseNet teachers improve DenseNet40-12 students compared with students trained from scratch.
- CIFAR10 results: Across shrinking learning schedules, early-stopped teachers outperform fully trained teachers for WideResNet16-1 and WideResNet28-1.The tested schedules vary step size k ∈ {10, 15, 20, 25, 60} and total epochs N ∈ {35, 50, 65, 80, 200}.
- CIFAR10 results: The conclusions remain consistent when using attention transfer and when comparing teachers with increasing width.
- CIFAR10 results: Even shallower early-stopped teachers produce better WideResNet28-1 students in parameter-matched comparisons.