Self-Distillation as Instance-Specific Label Smoothing

Zhilu Zhang; Mert R. Sabuncu

Source-linked AI summary

Self-Distillation as Instance-Specific Label Smoothing

Zhilu Zhang, Mert R. Sabuncu

arXiv:2006.05065v2Jun 9, 2020cs.LGstat.ML

TL;DR

The paper asks why multi-generational self-distillation improves generalization despite limited understanding of the mechanism. It interprets teacher-student training as amortized MAP estimation with instance-specific regularization, relates it to label smoothing, and finds that predictive diversity is important alongside uncertainty. It also proposes Beta smoothing, which often outperforms classical label smoothing.

Problem
The mechanism behind the generalization improvement from multi-generational self-distillation remains poorly understood.
Method
The paper interprets teacher-student training as amortized MAP estimation and proposes instance-specific Beta smoothing to promote predictive diversity without a separately trained teacher.
Results
Experiments associate improved self-distillation performance with increasing teacher-prediction diversity and report that Beta smoothing often outperforms classical label smoothing.
Takeaways & Limitations
Predictive diversity should be considered alongside predictive uncertainty when designing regularization for neural-network predictions.
Takeaways & Limitations
Directly regularizing confidence diversity can be unsatisfactory because mini-batch estimation is difficult and early promotion may harm learning.

Abstract

from arXiv · show

It has been recently demonstrated that multi-generational self-distillation can improve generalization. Despite this intriguing observation, reasons for the enhancement remain poorly understood. In this paper, we first demonstrate experimentally that the improved performance of multi-generational self-distillation is in part associated with the increasing diversity in teacher predictions. With this in mind, we offer a new interpretation for teacher-student training as amortized MAP estimation, such that teacher predictions enable instance-specific regularization. Our framework allows us to theoretically relate self-distillation to label smoothing, a commonly used technique that regularizes predictive uncertainty, and suggests the importance of predictive diversity in addition to predictive uncertainty. We present experimental results using multiple datasets and neural network architectures that, overall, demonstrate the utility of predictive diversity. Finally, we propose a novel instance-specific label smoothing technique that promotes predictive diversity without the need for a separately trained teacher model. We provide an empirical evaluation of the proposed method, which, we find, often outperforms classical label smoothing.

1 Introduction

The paper investigates why multi-generational self-distillation improves generalization and attributes its success to regularization involving both labels and the softmax probability simplex. It interprets teacher-student training through amortized MAP estimation and proposes Beta smoothing as a stronger alternative to classical label smoothing.

The paper addresses the unresolved mechanism behind multi-generational self-distillation and its reported generalization improvements.
Teacher-student training is interpreted as amortized MAP estimation, with teacher predictions acting as instance-specific priors conditioned on inputs.
Self-distillation is attributed to regularization on both the label space and the softmax probability simplex, with the latter tested systematically across benchmark datasets.
Beta smoothing is proposed as a new regularization technique that improves upon classical label smoothing at little extra cost.

2 Related Works

Related work frames knowledge distillation as a broad model-compression and knowledge-transfer technique with extensions to hidden-layer, data-free, Bayesian, ensemble, and theoretical settings. Prior studies also analyze how distillation objectives affect student models.

Knowledge distillation has expanded from model compression to applications involving hidden representations, data-free settings, Bayesian distributions, and ensembles.
Prior analyses study distillation through gradient comparisons, theoretical models, and experiments examining teacher size and student behavior.

3 Preliminaries

The paper formulates k-class neural classification probabilistically and describes teacher-student training as combining ground-truth cross-entropy with a temperature-scaled distillation loss. In self-distillation, teacher and student share an architecture, while temperature scaling flattens teacher predictions.

Neural classifiers model labels categorically through softmax probabilities over the probability simplex, with maximum likelihood yielding cross-entropy training.
A student is trained from a pretrained teacher using a total objective that combines cross-entropy and distillation losses.
The hyperparameters α and T control the loss weighting and temperature scaling, respectively; self-distillation uses identical teacher and student architectures.
Scaling only teacher predictions, rather than both teacher and student predictions, can produce more calibrated predictions.

4 Multi-Generation Self-Distillation: A Close Look

Sequential self-distillation improves across generations as teacher predictions become more uncertain and diverse, but average uncertainty alone does not capture their population spread. Experiments indicate that confidence diversity is closely associated with student accuracy and motivates measuring predictive spread on the probability simplex.

4.3 Sequential Self-Distillation Experiment: Multi-generational self-distillation uses each generation’s model as the next teacher, and student performance generally improves over generations.
4.2 Confidence Diversity: Average predictive uncertainty cannot distinguish teachers with similar mean entropy but different population spreads across probability-simplex predictions.
4.2 Confidence Diversity: Confidence diversity is estimated through one-dimensional entropy of the true-label softmax values, simplifying entropy estimation in high-dimensional class spaces.
4.3 Sequential Self-Distillation Experiment: Increasing temperature raises teacher uncertainty and diversity, producing drastic student accuracy gains that exceed the best 10-generation BAN result, while excessive temperature increases NLL.
4.2 Confidence Diversity: Confidence diversity plateaus after T = 2.5, coinciding with stagnant student accuracy despite continued increases in average predictive uncertainty.

5 An Amortized MAP Perspective of Self-Distillation

The paper reframes self-distillation as amortized MAP estimation with an instance-specific prior over softmax outputs. This perspective links distillation to label smoothing while emphasizing confidence diversity alongside predictive uncertainty.

5 An Amortized MAP Perspective of Self-Distillation: Self-distillation can be formulated as amortized MAP estimation of softmax probability outputs using an instance-specific Dirichlet prior.The resulting objective combines cross-entropy with instance-specific regularization and approximates MAP solutions even when prior parameters and labels are unavailable at test time.
5 An Amortized MAP Perspective of Self-Distillation: A uniform instance-independent Dirichlet prior yields the standard label-smoothing objective, which increases predictive uncertainty but not confidence diversity.This follows from assuming p(z|x)=p(z) and choosing equal prior parameters across labels.
5 An Amortized MAP Perspective of Self-Distillation: A teacher network supplies an instance-specific prior through exponentiated logits, with temperature and offset parameters controlling prior flattening.Increasing T or γ flattens the prior, while T→∞ recovers the uniform prior associated with classical label smoothing.
5 An Amortized MAP Perspective of Self-Distillation: The sample-reweighted distillation objective produced weights too similar to regular distillation under practically useful temperatures.The authors therefore retained the standard distillation loss and left alternative teacher objectives for future work.
5 An Amortized MAP Perspective of Self-Distillation: The MAP perspective interprets multi-generational self-distillation as implicitly flattening and diversifying instance-specific priors.The authors argue that tuning T and γ may achieve similar or better effects more directly than repeated distillation generations.

6 Beta Smoothing Labels

Beta smoothing uses EMA predictions to rank examples and assign instance-specific smoothing. More confident examples receive less smoothing, while difficult examples receive more, encouraging confidence diversity without a separately trained teacher.

6 Beta Smoothing Labels: Beta smoothing assigns less label smoothing to high-confidence examples and more smoothing to challenging examples using EMA-based confidence rankings.The ranking determines how Beta-distributed values are mapped to ground-truth and non-ground-truth class priors.
6 Beta Smoothing Labels: The method uses EMA predictions to stabilize per-iteration rankings and substitutes Beta-smoothed labels for teacher predictions during training.The authors report a significant performance boost from using EMA predictions for the ranking.
6 Beta Smoothing Labels: Beta smoothing implements an instance-specific prior that encourages confidence diversity without training a separate teacher model.Because its prior changes throughout training, it does not strictly correspond to the paper’s MAP formulation.
6 Beta Smoothing Labels: The method differs from classical label smoothing through additional spreading of the ground-truth class, enabling systematic tests of confidence diversity.The paper presents Beta smoothing as a low-cost alternative intended to improve over classical label smoothing.

7 Empirical Comparison of Distillation and Label Smoothing

Across CIFAR-100, CUB-200, and Tiny-ImageNet experiments, regularization improved accuracy and calibration over cross-entropy. Self-distillation generally exceeded label smoothing, while Beta smoothing often matched or surpassed label smoothing without a separate teacher.

7.1 Experimental Setup: The experimental design aimed to separate predictive-uncertainty and confidence-diversity effects, but hyper-parameter tuning was omitted because of limited computational resources.The authors note that all methods could potentially be improved through tuning.
7.2 Results: Self-distillation outperformed label smoothing in every reported experiment, while Beta smoothing outperformed label smoothing except for CIFAR-100 with ResNet.Beta smoothing achieved performance comparable to self-distillation on CUB-200 without requiring a separate teacher model.
7.2 Results: All tested regularization techniques improved test accuracy over the cross-entropy baseline across the evaluated datasets and architectures.The experiments used CIFAR-100, CUB-200, and Tiny-ImageNet with ResNet and DenseNet models.
7.2 Results: The accuracy gains from Beta smoothing over label smoothing are presented as evidence that confidence diversity contributes beyond predictive uncertainty.The two methods differ by the additional spreading applied to ground-truth classes.
7.2 Results: All regularization techniques improved calibration, with self-distillation generally producing smaller expected calibration errors than label smoothing.Beta smoothing compared favorably to self-distillation in most experiments.

8 Discussion and Future Directions

The paper links self-distillation performance to diversity in teacher predictions and interprets distillation as instance-specific regularization. It also connects this framework to calibration, soft-label methods, and future priors for regularization.

8 Discussion and Future Directions: Teacher-prediction diversity is correlated with student performance in self-distillation and motivates the paper’s amortized MAP interpretation.The interpretation treats teacher-student training as instance-specific regularization and suggests more efficient priors.
8 Discussion and Future Directions: Distillation can yield more calibrated models, which the paper attributes to avoiding temperature scaling on student models during training.The paper argues that applying student temperature scaling could push logits larger and produce over-confident predictions.
8 Discussion and Future Directions: Soft-label methods such as mixup can be viewed as encouraging confidence diversity or implementing instance-specific regularization.Mixup creates label diversity by taking random convex combinations of training examples and labels.

Statement of the Potential Broader Impact

The paper presents a theoretically grounded interpretation of self-distillation as a route toward more generalizable and reliable models. It also warns that prior choices can introduce fairness and privacy risks.

Statement of the Potential Broader Impact: The paper’s new interpretation is intended to deepen understanding of self-distillation, a widely used technique for improving accuracy.It presents this understanding as potentially beneficial across future deep-learning applications.
Statement of the Potential Broader Impact: The paper addresses limited understanding of why distillation works by offering a theoretically grounded explanation for its success.It frames understanding underlying mechanisms as important because blindly applying methods can be dangerous.
Statement of the Potential Broader Impact: The proposed regularization perspective may support more generalizable and reliable models for sensitive applications such as healthcare and autonomous driving.The paper emphasizes that incorrect machine predictions in these domains can have catastrophic consequences.
Statement of the Potential Broader Impact: The findings suggest that deep learning can still benefit from regularization, motivating further research into more efficient and principled approaches.The paper characterizes this as a form of overfitting different from conventional wisdom.
Statement of the Potential Broader Impact: Using priors for neural-network regularization can introduce unwanted bias, potentially creating fairness and privacy issues.This risk depends on the choice of priors used during training.

A.1 On Label Smoothing and Predictive Uncertainty Regularization

The appendix shows that the derived objective is equivalent to classical label smoothing, while contrasting its uncertainty regularization with the absence of confidence-diversity regularization.

A.1 On Label Smoothing and Predictive Uncertainty Regularization: The derived objective is equivalent to label smoothing when 1 −ϵ = k+β / k(1+β), up to a constant factor of (1 + β).This equivalence follows after rearranging the objective’s terms.
A.1 On Label Smoothing and Predictive Uncertainty Regularization: Label smoothing regularizes predictive uncertainty but does not regularize confidence diversity, pushing high-capacity models toward the smoothed soft label.The passage connects this behavior to reported hampered distillation performance.
A.1 On Label Smoothing and Predictive Uncertainty Regularization: The appendix introduces predictive-uncertainty regularization as a closely related alternative to label smoothing.Prior work is cited as showing that it can perform better than label smoothing.
A.1 On Label Smoothing and Predictive Uncertainty Regularization: Direct predictive-uncertainty regularization has the same global optimum as label smoothing, although optimization details and model capacity can produce practical differences.The equivalence is established for the stated objective and sample-pair optima.

A.2 Additional Experiments with Temperature Scaling on Student Models

Additional experiments examine how temperature scaling, sequential generations, smoothing, dataset size, weight decay, and related variants affect distillation. Across these tests, calibration and accuracy patterns support predictive diversity and the interpretation of distillation as instance-specific regularization.

A.2 Additional Experiments with Temperature Scaling on Student Models: Student models without student-side temperature scaling generally achieve better calibration than their teachers, whereas student scaling produces nearly teacher-like ECE.The experiments also note that high temperature with α = 0.4 can yield overly unconfident teacher predictions.
A.3 Additional Experiments on Sequential Self-Distillation with Different Temperatures: Accuracy and NLL across five sequential generations correlate strongly with confidence diversity at temperatures T = 2 and T = 3.The evaluations use student predictions for test accuracy and NLL, while predictive uncertainty and confidence diversity are measured from teacher predictions on the training set.
A.4 Additional Experiments with Different Amount of Label Smoothing ϵ: Experiments varying label smoothing at ϵ = 0.1 and ϵ = 0.3 compare cross-entropy, label smoothing, Beta smoothing, and self-distillation across CIFAR-100, CUB-200, and Tiny-ImageNet.Beta smoothing and self-distillation are adjusted so their average smoothing matches the corresponding label-smoothing setting.
A.6 Additional Experiments with CIFAR-10 When Varying Trainset Size: As training-set size increases on CIFAR-10, both teacher and student accuracy rise, but the student’s relative gain over the teacher decreases.This pattern is presented as evidence that distillation behaves like regularization whose effect diminishes with more data.
A.7 Additional Experiments with CIFAR-100 When Varying Weight Decay: Increasing weight decay on CIFAR-100 produces smaller self-distillation improvements, while cross-entropy baselines also gain noticeably from adjusted weight decay.The distillation comparison uses the same teacher trained with weight decay 10^-4 to keep the prior quality fixed.