On the Efficacy of Knowledge Distillation

• Because large, deep recognition networks carry computational costs, work seeks similar accuracy in smaller models through efficient architectures, pruning, quantization, and distillation.Source: §1 · p. 1
• Knowledge distillation trains a smaller student with a larger teacher’s soft probabilities alongside class labels, which may convey information beyond labels.Source: §1 · p. 1
• Despite its apparent any-student/any-teacher generality, prior evaluations used few chosen architectures; isolated failures, including ImageNet [26], left its generality and remedies unclear.Source: §1 · p. 1
• Teacher accuracy is a poor predictor of student performance: larger, more accurate teachers do not necessarily teach better, as students often struggle to emulate them.The paper reports that students can retain high KL divergence from teacher logits during training.Source: §1 · p. 1
• Sequential distillation does not resolve the capacity problem; instead, early teacher stopping and ending distillation near convergence are presented as broadly effective ways to improve student fitting.The paper describes this as regularizing the teacher and helping the student fit the training loss better.Source: §1 · p. 2

2.1. Knowledge distillation

• Knowledge distillation builds on model compression, can be applied to deep networks [12], and was not yet thoroughly evaluated empirically.This paper uses the framework described by Hinton et al. [12].Source: §2.1 · p. 2
• Prior work explored feature-map losses [26] [15], bidirectional distillation [27], student-model averaging [21], and a more tolerant teacher loss [23].Source: §2.1 · p. 2
• Sequential distillation, including self-teacher sequences proposed by Furlanello et al., underperforms scratch-trained ensembling and can weaken teacher ability, especially for limited-capacity students or complex datasets.The paper reports that these methods depend strongly on student capacity and are ineffective in many cases.Source: §2.1 · p. 2
• Knowledge distillation has been applied to sequence modeling [16], semi-supervised learning [21], domain adaptation [19], and multimodal learning [9].The paper argues that this wide applicability makes exhaustive evaluation of distillation ideas more important.Source: §2.1 · p. 2

• Soft teacher probabilities can reveal relationships among classes, such as an image lying near a decision boundary, and student imitation can transfer this information beyond labels.The paper presents this as the key idea behind knowledge distillation.Source: §3 · p. 2
• For an input image, the teacher produces class scores that are converted into probabilities.Source: §3 · p. 2
• Because trained networks produce peaky probabilities, Hinton et al [12] propose softening them with temperature scaling [8], using a hyperparameter τ > 1.Source: §3 · p. 2
• The student produces a softened class-probability distribution and is trained with a linear combination of classification cross-entropy and the knowledge-distillation loss.Source: §3 · p. 2

Experimental design

• Across both datasets, the student is held constant while multiple teachers of varying capacity perform knowledge distillation.The experiments are conducted on CIFAR10 and ImageNet.Source: §4 · pp. 2–3

CIFAR10

• On CIFAR10, the study compares ResNet [11], WideResNet [25], and DenseNet [13] teachers with a fixed student across five repeated runs.Models train for 200 epochs using SGD with momentum 0.9, an initial learning rate of 0.1 dropped by 0.2 every 60 epochs, and standard augmentation; median, mean, and standard deviation are reported. Distillation uses τ = 4, α = 0.9, and β = 1000 for attention transfer, following popular choices [12], [26].Source: §4 · p. 3

ImageNet

• On ImageNet, the experiments use ResNet [11] and follow Zagoruyko et al. [26] closely.Training uses SGD with Nesterov momentum 0.9, initial learning rate 0.1, weight decay 1 × 10−4, and a 0.1 learning-rate drop every 30 epochs; distillation uses τ = 4, α = 0.9, and β = 1000 for attention transfer.Source: §4 · p. 3

5.1. Bigger models are not better teachers

• On CIFAR10, student accuracy rose and then fell as teacher size increased; on ImageNet, larger, more accurate teachers likewise reduced ResNet18 student accuracy.The trend held even as teacher accuracy continued to rise.Source: §5.1 · p. 3, Table 1 · p. 3, Figure 2 · p. 4

5.2. Analyzing student and teacher capacity

• Higher KD error or loss with the largest teachers coincided with the least accurate students, supporting a student–teacher capacity mismatch rather than an inability to improve accuracy despite mimicry.The paper interprets this as students being unable to find solutions in their limited capacity that correspond well to the largest teachers.Source: §5.2 · pp. 3–4

5.3. Distillation adversely affects training

• On ImageNet, distillation initially improved validation accuracy but hurt it late in training; early-stopped KD beat full KD for all three teachers and outperformed the 30.24% scratch baseline.Early-stopped KD also had lower training cross-entropy and higher KD loss, consistent with reducing the trade-off between those losses.Source: §5.3 · pp. 4–5, Table 3 · p. 5

5.4. The efficacy of repeated knowledge distillation

• Sequential KD sometimes left the final CIFAR student below scratch, and its ensembles did not beat equal-sized scratch ensembles; step-wise distillation also failed to help on CIFAR10 or ImageNet.The authors suggest that sequential students may be correlated, weakening their ensemble.Source: §5.4 · p. 5, Table 6 · p. 6

5.5. Early-stopped teachers make better teachers

• ≈29.01% was the best ImageNet result, from early-stopped ResNet34 and fully trained ResNet18 teachers; it gained ≈1.23 points over scratch and was ≈0.2% from the best known result [17].Early-stopped distillation with the largest fully trained teacher also outperformed most prior work at ≈29.45%.Source: Table 8 · p. 6, §5.5 · p. 7
• For CIFAR10 WRN students, all tested early-stopped teachers outperformed the optimal fully trained teachers, despite requiring only one third to one fourth of the total training epochs.The paper also reports that sequentially distilled teachers were not better teachers even when their accuracy improved.Source: §5.4 · p. 5, §5.5 · pp. 6–7, Table 5 · p. 6

5.6. Other factors impacting knowledge distillation

• Early-stopped teachers remained beneficial across DenseNet and Wide ResNet configurations, and transfer to Places-365 followed the CIFAR10 and ImageNet findings.The transfer comparison fine-tuned ImageNet students on Places-365 for 12 epochs.Source: §5.6 · pp. 7–8, Table 9 · p. 7, Figure 5 · p. 8
• On CIFAR10, early-stopped teachers consistently beat fully trained teachers across α and τ settings; at τ = 20, higher temperature improved ESKD but not Full KD.The early-stopped teacher still performed best in the high-temperature ImageNet experiment.Source: §5.6 · pp. 7–8, Table 10 · p. 7, Figure 6 · p. 8

6. Conclusion

• Knowledge distillation cannot succeed when student capacity is too low to successfully mimic the teacher.The paper presents this as its key finding from an exhaustive study of factors influencing distillation.Source: §6 · p. 8
• Stopping teacher training early can recover a solution more amenable for the student and mitigate the capacity limitation.Source: §6 · p. 8
• The early-stopping approach showed benefits on CIFAR10, ImageNet, and transfer learning on Places365.The authors say further research into distillation's nuances is necessary before it can succeed as a general and practical approach.Source: §6 · p. 8

7. More Results on CIFAR10

• Tables 17 and 18 report the efficacy of early-stopped teachers for WRN16-1 and WRN28-1 students trained from teachers with varying width factors.The experiments considered total epochs N ∈ {35, 50, 65, 80, 200} and learning-rate decay step sizes k ∈ {10, 15, 20, 25, 60}.Source: §7 · p. 10
• Table 19 shows that the conclusions are consistent with attention transfer combined with knowledge distillation (AT+KD).Source: §7 · p. 10
• Tables 11, 12, 13, and 16 present additional experiment settings, including different student–teacher pairs and learning methods.Source: §7 · p. 10

DenseNet teachers

• DenseNet40-12 students trained with early-stopped DenseNet teachers outperform those trained with fully trained teachers by a large margin.Some students trained with fully trained teachers fail to match the accuracy of the student trained from scratch.Source: Table 12 · p. 10

Cosine scheduling

• CIFAR10 students trained with early-stopped teachers perform better under cosine learning-rate decay.Source: Table 13 · p. 10

ImageNet teacher models

• Table 14 lists the scratch-trained fully trained and early-stopped models used as teachers in the ImageNet experiments.Source: Table 14 · p. 10, §8 · p. 10

ImageNet experiments

• Figure 7 shows that knowledge distillation harms students with limited capacity, and early-stopping mitigates the effect.The section also identifies Table 14 as the source of the fully trained and early-stopped teacher models used in the ImageNet experiments.Source: §8 · p. 10, Figure 7 · p. 11

Teacher depth

• Across WideResNet teacher depths, early-stopped teachers outperform fully trained teachers, while larger fully trained teachers do not produce better students.This finding holds for WideResNet16-1 students with WideResNet teachers and DenseNet40-12 students with WideResNet teachers.Source: Table 15 · p. 11, Table 16 · p. 11

Learning-schedule shrinking

• For WideResNet16-1 and WideResNet28-1 students, teachers trained with any of the tested shortened learning schedules outperform fully trained teachers.The tested step sizes were k ∈ {10, 15, 20, 25, 60}, with total epochs N ∈ {35, 50, 65, 80, 200}.Source: Table 17 · p. 12, Table 18 · p. 12

Other distillation and teacher comparisons

• The conclusions remain consistent with attention transfer and with WideResNet28-1 students trained using shallower teachers matched to larger teachers by parameter count.The shallower-teacher comparisons use WideResNet16-x teachers chosen to match the parameter counts of WRN28-3, WRN28-4, and WRN28-8.Source: Table 19 · p. 13, Table 20 · p. 13