Towards Understanding Ensemble, Knowledge Distillation and Self-Distillation in Deep Learning

Zeyuan Allen-Zhu; Yuanzhi Li

Source-linked AI summary

Towards Understanding Ensemble, Knowledge Distillation and Self-Distillation in Deep Learning

Zeyuan Allen-Zhu, Yuanzhi Li

arXiv:2012.09816v3Dec 17, 2020cs.LGcs.NEmath.OCstat.ML

TL;DR

The paper asks why simple averaging of same-architecture networks trained on the same data can improve test performance, and how that advantage can be transferred to one model. It develops a theory for multi-view classification and proves ensemble improvement, ensemble-to-single-model distillation, and self-distillation effects. The results distinguish deep-learning ensembles from random-feature and other traditional explanations.

Problem
Existing theory does not satisfactorily explain uniform averaging of independently initialized, same-architecture neural networks trained on the same data, nor its distillation and self-distillation behavior.
Method
The paper proposes a multi-view data structure and analyzes certain two-layer convolutional networks trained by gradient descent, including ensemble, distillation, and self-distillation.
Results
For the stated multi-view setting, the paper proves that independently trained ensembles improve test accuracy, their outputs can be distilled into a single model, and self-distillation also improves accuracy.
Takeaways & Limitations
The findings suggest that ensemble and distillation benefits in deep learning involve feature learning and differ from random-feature explanations.
Takeaways & Limitations
The theory applies to certain multi-view classification tasks and certain two-layer convolutional networks, while ensemble improvement is not universal and may fail for Gaussian-like inputs.

Abstract

from arXiv · show

We formally study how ensemble of deep learning models can improve test accuracy, and how the superior performance of ensemble can be distilled into a single model using knowledge distillation. We consider the challenging case where the ensemble is simply an average of the outputs of a few independently trained neural networks with the SAME architecture, trained using the SAME algorithm on the SAME data set, and they only differ by the random seeds used in the initialization. We show that ensemble/knowledge distillation in Deep Learning works very differently from traditional learning theory (such as boosting or NTKs, neural tangent kernels). To properly understand them, we develop a theory showing that when data has a structure we refer to as ``multi-view'', then ensemble of independently trained neural networks can provably improve test accuracy, and such superior test accuracy can also be provably distilled into a single model by training a single model to match the output of the ensemble instead of the true label. Our result sheds light on how ensemble works in deep learning in a way that is completely different from traditional theorems, and how the ``dark knowledge'' is hidden in the outputs of the ensemble and can be used in distillation. In the end, we prove that self-distillation can also be viewed as implicitly combining ensemble and knowledge distillation to improve test accuracy.

1 Introduction

The paper studies why uniformly averaging independently trained same-architecture networks can outperform individual models, and why that advantage can be distilled into one model. It develops a multi-view theory and reports empirical evidence that these effects differ from random-feature explanations.

Uniformly averaging a few independently trained networks with the same architecture, algorithm, and data can improve test accuracy over individual models.The networks differ only through initialization and/or training randomness; ten ResNet-34 models on CIFAR-100 can yield a 5% test-accuracy improvement.
Existing ensemble theories mainly address boosting, resampling, heterogeneous models, or random features rather than this standard deep-learning setting.The paper identifies a gap for uniform averaging of same-architecture networks trained by SGD on the same training set.
Training directly toward an average of individual networks performs much worse than ensemble, whereas knowledge distillation transfers ensemble performance to a single model.Distillation trains the student to match ensemble outputs, or soft labels, over the original training data.
Self-distillation also improves test accuracy, and the paper frames it as implicitly combining ensemble with knowledge distillation.These observations motivate a theory distinct from traditional explanations based on random-feature-space enlargement.
For multi-view data and certain two-layer convolutional networks, the theory gives computational and statistical results for single models, ensembles, distillation, and self-distillation.The stated results include bad single-model generalization despite efficient zero training error, improved ensemble accuracy, successful distillation, and self-distillation.
Empirical results indicate that knowledge distillation fails for random feature mappings, special data structure is needed for neural-network ensembles, and label noise or non-convexity may not explain their advantage.The paper therefore argues that deep learning should be studied as feature learning rather than merely feature selection.

2 Our Methodology and Intuition

The paper argues that deep-learning ensemble and distillation are feature-learning phenomena rather than random-feature selection. Under multi-view data, different initializations learn different useful features, enabling ensemble improvement and distillation through soft outputs.

2.1 A Failure Attempt Using Random Feature Mappings: NTK-style random-feature theory cannot explain deep-learning distillation because it treats training as linear prediction over fixed initialization-dependent features.In that framework, ensemble enlarges the available feature space, whereas deep-learning experiments show ensemble features can be learned by one model.
2.1 A Failure Attempt Using Random Feature Mappings: Training the average of independently trained neural networks performs much worse than ensembling their outputs, while knowledge distillation transfers ensemble performance to one model.These contrasting outcomes motivate studying how soft ensemble outputs encode information unavailable from hard-label training alone.
2.2 Ensemble in Deep Learning: a Feature Learning Process: In the multi-view construction, each class has multiple discriminative features, but cross-entropy training quickly learns only one feature per class before memorizing remaining examples.The resulting single model reaches perfect training accuracy yet only about 90% test accuracy in the motivating binary example.
2.2 Ensemble in Deep Learning: a Feature Learning Process: Random initialization makes individual networks choose different views, so an ensemble of eO(1) models covers all class features and classifies every motivating example correctly.The paper connects this mechanism to different image regions or channels being used by independently trained vision models.
2.2 Ensemble in Deep Learning: a Feature Learning Process: Ensemble outputs retain small non-target responses that reveal additional views, allowing distillation to teach a single model features absent from its direct hard-label training.The paper reports that distilled models learn most features, leaving little further ensemble benefit.
2.4 Significance of Our Technique: The theory emphasizes that initialization-dependent feature-learning order links optimization randomness to generalization, unlike convex methods with a unique regularized optimum.The authors present this as extending analyses that separate optimization from statistical generalization.

3 Problem Setup

The paper studies k-class classification on patch-based inputs containing multiple orthogonal features per class, mixing multi-view and single-view examples with structured and Gaussian noise. It analyzes a two-layer convolutional learner trained by gradient descent with smoothed ReLU.

3.1 Data Distribution and Notations: The setup uses k-class inputs with P = k^2 patches of dimension d = poly(k), and assumes sufficiently large k while hiding polylogarithmic factors.The main text uses example parameter choices depending on k, with broader theorem ranges deferred to the appendix.
3.1 Data Distribution and Notations: Each class has two orthogonal unit features, and examples contain the label’s features plus randomly sampled features from other classes.The model distinguishes multi-view data, where multiple label features appear, from single-view data, where some correct-label features are missing.
3.1 Data Distribution and Notations: Feature-bearing patches receive nonnegative random coefficients, while other patches contain only feature or Gaussian noise whose placement and weights may be arbitrary.The construction permits unequal feature weights and allows initialization to determine which features are learned first.
3.1 Data Distribution and Notations: The final distribution mixes multi-view and single-view samples, with D = Dm with probability 1 − µ and D = Ds with probability µ.The training set is divided accordingly into Zm and Zs, with a specified sample-size regime ensuring representation of each label in single-view data.
3.2 Learner and Training: The learner is a two-layer convolutional network with smoothed ReLU, trained on cross-entropy by gradient descent from random initialization.Smoothed ReLU suppresses low-magnitude feature noise while remaining linear at large inputs; moderate over-parameterization is used for optimization analysis.

4 Main Theorems and Explanations

The theorems show that single models can fit the training set yet generalize poorly, while averaging a few independently initialized models achieves near-perfect test accuracy. Matching ensemble outputs distills this performance, and self-distillation improves a single model by combining feature sets.

Theorem 1 (single model): A trained single model achieves perfect training accuracy but has consistently bad test accuracy, around 0.5µ in the stated setting.The mechanism is a random initialization-dependent lottery-winning set: the model learns one of two features per class and misses the complementary single-view cases.
Theorem 2 (ensemble): An ensemble of K = eΩ(1) independently trained models with m = polylog(k) achieves perfect training accuracy and almost-perfect test accuracy.Different lottery-winning sets depend on initialization, and their union covers all possible features.
Theorem 3 (ensemble distillation): Knowledge distillation trains a randomly initialized model against truncated scaled ensemble logits rather than only the true labels.The procedure uses a larger distillation learning rate, and the theorem guarantees perfect training accuracy and almost-perfect test accuracy for most training iterations.
Theorem 3 (ensemble distillation): The distilled model learns all features represented by the ensemble, so further ensembling independently distilled models provides no additional empirical boost.This is consistent with the paper’s claim that ensemble performance is transferred into a single network.
Theorem 4 (self-distillation): Self-distillation improves a model by matching the output of an independently trained single model under an additional data assumption.The resulting feature set is MF ∪ MG, combining features learned by the original and teacher models; the paper states the distilled model generalizes better than either individual model.
Theorem 4 (self-distillation): Theorem 4 formalizes self-distillation as implicitly combining two models’ feature sets before learning them in one network.The paper compares this process to first ensembling the models and then distilling that ensemble into a new model.

5 Our Empirical Results at a High Level

The empirical results distinguish deep-learning ensembles from random-feature and traditional variance-reduction accounts: ensemble benefits depend on the data and are not reproduced by direct training, while knowledge distillation and self-distillation can retain the benefit.

Ensemble behavior in deep learning differs from ensemble methods based on random feature mappings and neural tangent kernels.The experiments compare finite-width neural networks, NTK variants, and Gaussian-process-style last-layer training.
Directly training the sum of independently initialized neural networks fails to improve test accuracy, unlike independently training models and then ensembling them.For random-feature models, direct training can instead outperform their ensemble.
On Gaussian-like data and several target-network settings, ensemble does not improve test accuracy, indicating that data structure matters.The reported comparison covers fully connected, residual, and convolutional target networks.
Knowledge distillation transfers ensemble performance to a single deep neural network but not to a single model based on random feature mappings.The contrast supports viewing deep learning as feature learning rather than merely selecting prescribed features.
Variance reduction alone does not explain the deep-learning ensemble benefit: standard datasets show little individual accuracy variance, whereas Gaussian-like experiments can reduce output variance without improving accuracy.On CIFAR-100, individual-model test-accuracy standard deviation is reported as 0.1 ∼0.4%.
Self-distillation and knowledge-distillation ensembles show that distilled models learn most ensemble features, leaving little additional gain from re-ensembling them.The same empirical pattern is reported for independently initialized models trained with knowledge distillation.

6 Conclusion and Discussion

The paper presents a multi-view theory explaining when ensemble improves test accuracy and when its performance can be distilled into one model. It also reports that ensemble may fail to help on Gaussian-like data and suggests extensions to data augmentation.

Ensemble does not always improve deep-learning test accuracy, especially when inputs follow a Gaussian-like distribution.
The proposed multi-view structure yields provable ensemble accuracy improvements for two-layer neural networks.
Figure 9 supports the multi-view hypothesis by showing that collapsing or removing intermediate-layer channels often has limited effect on single-model accuracy, while ensemble accuracy changes little.
A single model trained to simulate ensemble outputs can match ensemble test accuracy and outperform a clean model trained directly on labels.
The framework may extend to random-cropping augmentation as another way to enforce multi-view learning.

A Data Distribution and Notations (Full Version)

This section defines a synthetic multi-view and single-view classification distribution over patch-based inputs. It specifies feature structure, patch assignment, noise, weights, sampling, and experimental training settings.

The task is k-class classification over P patches, each of dimension d, with inputs X=(x_1,...,x_P) and labels y∈[k].
Each class has two orthogonal unit features, with data containing two main class features plus randomly sampled minor features.
Feature weights are assigned across disjoint patches, with one main feature receiving much larger total weight than the other main and minor features.
The construction includes feature noise and independent Gaussian noise, while unassigned patches are also filled with Gaussian noise.
The concept class is designed to challenge linear classifiers and constant-degree polynomials because accumulated low-magnitude noise can overwhelm the signal.
The final distribution mixes multi-view and single-view data, with label-uniform sampling and a training set Z=Z_m∪Z_s.

B.1 Real-Life Data: Single Model vs Ensemble vs Distillation

The real-data experiments compare independently trained models, their output ensembles, direct training of averaged models, and knowledge distillation. Additional experiments examine self-distillation and intermediate-layer views.

Single Model vs Ensemble: Ensemble accuracy is computed by averaging the outputs of 10 independently trained models before predicting test labels.
Single Model vs Ensemble: Directly training a larger network formed by averaging single models is evaluated separately using the same training algorithm as single models.
Distillation: Knowledge distillation uses a weighted combination of cross-entropy and distillation objectives, with a tuned temperature parameter.
Self-Distillation: Self-distillation trains a first model with hard-label cross-entropy, then trains another randomly initialized model to match the first model’s output.
Self-Distillation: Repeating self-distillation produces only very incremental additional accuracy gains.
Intermediate-Layer Views: Figure 9 tests intermediate-layer views by splitting or averaging channels and comparing eight single models with their ensemble.

B.4 Synthetic Data: Whether Ensemble Improves Accuracy over Gaussian-Like Data

Synthetic experiments span Gaussian-like distributions, label noise, margins, and several ground-truth network types. Across many Gaussian-like settings, ensemble accuracy barely exceeds single-model accuracy despite modest run-to-run variation.

Synthetic Data Generation: Synthetic data vary Gaussian generation, class balancing, label noise, margins, and linear, fully connected, convolutional, or residual ground-truth networks.
Synthetic Data Generation: The selected dimensions target single-model test accuracies of approximately 60%–80%.
Learner Networks: Ten independently initialized learner networks are trained with SGD, tuned learning rates, and weight decay, then compared with their ensemble accuracy.
Results: Across many Gaussian-like datasets, ensemble accuracy barely exceeds the best single-model accuracy.
Results: Single-model accuracy variation is negligible relative to already modest accuracy; a 1.0% standard deviation is given as an example near 70% test accuracy.
Results: For synthetic Gaussian-like data, ensemble offers nearly no benefit even when single-model standard deviations exceed those observed on CIFAR-10/100.

C Single Model: Proof Plan and Induction Hypothesis

The proof tracks feature-selection dynamics through an induction hypothesis, showing that initialization-dependent view selection shapes single-model behavior. Under stated parameter conditions, one view can remain missing, limiting single-view accuracy while multi-view examples are classified correctly.

Initialization and view selection: At random initialization, the set M contains only polylogarithmically many selected feature-view pairs with probability at least 1 −e−Ω(log5 k).M records which view has the stronger initialization correlation for each feature index.
Initialization and view selection: For each feature index, at least one view is selected with decent probability, while the induction hypothesis prevents the complementary view from being learned by the network.This creates the missing-view phenomenon that bounds single-view test accuracy by 49.99%.
Induction hypothesis: The induction hypothesis decomposes correlations on multi-view data into the two target-view correlations plus controlled residual terms, while single-view correlations include a significant noise term.The noise term supports memorization of single-view examples, whereas the view correlations support correct classification of multi-view data.
Proof plan: The induction hypothesis is maintained across all training iterations and connects neuron-feature correlations, output behavior, growth, and regularization.The proof establishes the hypothesis for every iteration up to T, then uses it to derive the main single-model and ensemble theorems.
Technical consequences: Gradient bounds and function approximations are derived under the induction hypothesis for both positive and negative labels, supporting subsequent growth and convergence claims.These intermediate estimates control logits and feature updates throughout the proof.

D.2.3 Multi-View Error Till the End

The proof shows that multi-view prediction remains controlled through the final training stage under the induction hypothesis and the required sample-size and parameter conditions.

D.2.3 Multi-View Error Till the End: Claim D.14 establishes the final-stage multi-view guarantee when the induction hypothesis holds and Ns ≤k2ρq−1.The claim follows from bounded feature growth together with convergence on single-view data.
Final-stage control: Claim D.15 provides the per-iteration multi-view bounds used to complete the final-stage argument once the relevant neurons enter the linear regime.The proof invokes the threshold conditions and bounded feature support to control the resulting logits.
Feature contributions: For the model’s data distribution, correct-label multi-view feature contributions satisfy Vi,r,ℓ(X) ≥0.9.The corresponding single-view correct-label contribution is only bounded below by zero.
Feature contributions: Wrong-label multi-view contributions are bounded by 0.4 when the relevant feature appears, while single-view wrong-label contributions are bounded by Γ ≤1.These bounds distinguish the stronger multi-view signal from off-target contributions.

D.2.4 Multi-View Individual Error

The individual multi-view analysis bounds prediction error using training error and tracks three training stages through growth, convergence, and parameter-control lemmas.

D.2.4 Multi-View Individual Error: The paper states that error on any individual multi-view example is bounded by the training error up to a polynomial factor.The same type of guarantee extends with high probability to examples drawn from Dm.
Three-stage analysis: The proof partitions training into three stages: before T0,i, between T0,i and T0, and after T0.Different bounds control logits and feature contributions in each stage.
Three-stage analysis: Claim D.17 controls multi-view behavior during stage 2, including the cases where neither or at least one of the two relevant views appears.The analysis uses the appearance probability s and parameter Υ to handle these cases.
Growth comparison: Lemma D.19 compares two nonlinear growth sequences and shows that a sequence with stronger updates reaches its threshold earlier under the stated initialization and step-size conditions.This comparison supports the view-lottery argument in the feature-learning dynamics.
Growth and convergence: The analysis bounds auxiliary quantities such as A(t) and Φ(t), yielding controlled correlations and completing the induction-based error argument.The relevant bounds include A(t) i ≤eO(1) and the final consequences of Lemmas D.21–D.23.
Scope conditions: The proof requires Parameter D.1, with the tightest upper bounds involving γ, s, and Ns in the cited stages.These restrictions are explicitly identified as necessary parameter choices for the proof.

E Single Model and Ensemble: Theorem Statements

The theorem statements formalize distinct single-model and ensemble outcomes: perfect training and good multi-view testing for a single model, with ensemble models additionally achieving good single-view testing when sufficiently large.

Single model: Theorem 1 states perfect training accuracy for every training example with probability at least 1 −e−Ω(log2 k) under the stated parameter regime.The theorem applies for sufficiently large k and after sufficiently many iterations.
Single model: Theorem 1 states that multi-view testing is good, with eO(1) ≥Φ(T) i ≥0.4Φ(T) j for every i, j ∈[k].The theorem separately states that single-view testing is bad for the view pairs in M.
Theorem scope: With two equally likely views, the stated testing accuracy is close to 1; more general settings may yield another constant µ.The equal-probability assumption explains the near-one value in the simplified theorem statement.
Ensemble model: Theorem 2 preserves perfect training and good multi-view testing for the ensemble model.These two guarantees are stated as the same as for the single model.
Ensemble model: When K ≥polylog(k), the ensemble model has good single-view testing.This is the ensemble-specific theorem guarantee distinguishing it from the single-model result.

E.1 Proof of Theorem 1

The proof establishes perfect training accuracy and analyzes how the trained model performs on multi-view and single-view data. It then shows that averaging sufficiently many independently trained models improves the ensemble’s class-score separation.

Training accuracy: The training objective remains monotonically non-increasing under full gradient descent, yielding perfect training accuracy at the final iteration.The objective is O(1)-Lipschitz continuous, so its value does not increase during training.
Multi-view testing: For multi-view data, the proof maintains a lower bound on the correct-class score relative to competing classes throughout the final analysis.The argument invokes the induction hypothesis and function-approximation claims to establish the multi-view guarantee.
Single-view testing: Single-view examples can be misclassified when the unobserved feature determines the competing class, but the proof bounds the resulting correct-class output by O(ρ) plus an inverse-polylogarithmic term.This case occurs with probability one-half for the relevant missing-feature configuration.
Generalization analysis: The proof combines feature-score growth and function approximation to control incorrect-class outputs on single-view and multi-view examples.The relevant feature appears with probability s/k, while competing classes are suppressed with high probability.
Ensemble effect: Once K ≥ m^Ω(1) models are ensembled and Γ ≤ 1/m^Ω(1), averaging gives the correct class a margin of 1/polylog(k) over every incorrect class.This is the central ensemble-separation conclusion used by the proof.

F.1 Using Ensemble for Knowledge Distillation

This section develops ensemble and self-distillation analyses using a truncated scaled-logit target. The stated results establish perfect training accuracy and favorable multi-view and single-view testing under the paper’s assumptions and parameter ranges.

Ensemble distillation: The ensemble-distillation procedure trains a new network from random initialization to match the ensemble’s output through iterative weight updates.The theorem considers K independently trained models and a separate distilled network.
Theorem guarantee: With probability at least 1 − e^−Ω(log^2 k), the ensemble-distillation guarantees hold for at least 90% of iterations through T.The theorem also specifies polynomial iteration and learning-rate conditions.
Theorem guarantee: The ensemble-distilled model achieves perfect training accuracy and favorable multi-view and single-view testing under the stated induction and parameter conditions.The proof tracks feature scores, bounds incorrect-class logits, and establishes threshold growth after iteration T0.
Self-distillation: Self-distillation uses a learn stage followed by a distill stage, with separate lottery-winning feature sets for the original and distilled models.The analysis assumes m = polylog(k) and allows shorter training than the single-model case because ensemble distillation can learn all training data.
Training-time comparison: The proof explains that single models require longer training to memorize single-view data, whereas ensemble distillation permits a shorter T under its parameter regime.The stated condition is N ≥ ηT · poly(k) with ηT ≥ poly(k), rather than ηT ≥ N · poly(k).
Proof strategy: The technical proof maintains an induction hypothesis across all iterations and uses gradient, function-approximation, growth, and testing claims to derive the theorem.The authors note that the distillation proof uses fewer and simpler claims than the single-model proof.

G.5 Proof of Theorem 4

The self-distillation proof combines a trained base model with a second network and analyzes two training stages. Under the balanced-data assumption, the resulting model improves coverage of single-view features while preserving multi-view performance.

Proof setup: The proof of self-distillation reuses the single-model and ensemble-distillation arguments, with the base model and distilled model contributing independent lottery-winning feature sets.The construction assumes balanced marginal feature distributions for multi-view data.
Stage 2: multi-view data: During stage 2, the multi-view feature quantity remains unchanged after scale-up because it already exceeds the threshold needed to eliminate further gradient updates.Consequently, the final model retains the nearly perfect multi-view accuracy established after stage 1.
Stage 2: single-view data: For features in the distilled model’s winning set, stage 2 increases the feature quantity to at least 1/(2τ), enabling correct predictions on the corresponding single-view examples.The resulting correct-class score is Ω(log^2 k), larger than competing-class scores with high probability.
Stage 2: single-view data: Features already won by the model in stage 1 also remain sufficient after scale-up, yielding correct predictions on their associated single-view examples.Their feature quantity reaches at least Ω(log^5 k) by the end of stage 2.
Combined coverage: Because the two winning sets are independent and each covers k(1 − o(1)) feature-label pairs, their union covers at least 3k/2(1 − o(1)) pairs.The proof uses this union size to obtain the final self-distillation accuracy lower bound.