Towards Understanding Ensemble, Knowledge Distillation and Self-Distillation in Deep Learning

1 Introduction

• In practice, averaging a few independently trained networks of the same architecture, trained on the same data with the same method, can substantially outperform an individual model; ten ResNet-34 models on CIFAR-100 improve test accuracy by 5%.Knowledge distillation transfers the ensemble’s performance to a single model by matching soft labels rather than true labels.Source: §1 · p. 2
• The paper targets an underexplained regime absent from most prior ensemble theory: uniform averaging of same-architecture neural networks trained by SGD on the same dataset, differing mainly through randomness.The introduction contrasts this setting with boosting, bagging, heterogeneous ensembles, and random-feature or tree ensembles.Source: §1 · p. 2

1.1 Our Theoretical Results at a High Level

• Under a multi-view data structure, a single model can achieve zero training error yet have prediction error between 0.49µ and 0.51µ, while an ensemble of independently trained models improves test accuracy.The theory considers two-layer convolutional networks with smoothed-ReLU activation, trained by gradient descent from random Gaussian initialization on N i.i.d. samples.Source: §1.1 · p. 3
• The ensemble’s superior test accuracy can be distilled into a same-architecture single model with prediction error at most 0.01µ, using gradient descent to match ensemble outputs on the original training data.This distillation is computationally efficient and holds with high probability under the stated multi-view setting.Source: §1.1 · p. 3
• Self-distillation also improves test accuracy: matching a single model’s outputs yields a same-size model with prediction error at most 0.26µ, interpreted as implicit ensemble plus knowledge distillation.The result holds with high probability and can be trained efficiently under the same theoretical setting.Source: §1.1 · p. 3

1.2 Our Empirical Results at a Glance

• Empirical evidence indicates that knowledge distillation fails for random feature mappings, neural-network ensembles require special data structure such as multi-view structure, and label-noise or optimization variance may not explain their gains.These findings motivate analyzing deep-learning ensembles as distinct from random-feature ensembles.Source: §1.2 · p. 3, Figure 1 · p. 3

2.1 A Failure Attempt Using Random Feature Mappings

• NTK explanations fail empirically: training the average can outperform an NTK ensemble, deep-network ensembles offer larger gains than comparable individual models, and NTK ensemble outputs cannot be distilled.These contradictions motivate studying ensemble and distillation as feature learning rather than selecting combinations of prescribed random features, while raising the question of what dark knowledge soft labels contain beyond hard labels.Source: §2.1 · p. 4

2.2 Ensemble in Deep Learning: a Feature Learning Process

• Ensemble gains are not explained by Gaussian-like inputs or reduced output variance: accuracy may remain unchanged despite lower variance, whereas equally well-trained networks can still ensemble substantially better.The paper therefore studies structured, noiseless inputs where all individual networks reach 100% training accuracy and nearly identical test accuracy.Source: §2.2 · p. 5

2.3 Our Approach: Learning Multi-View Data

• Multi-view data contains multiple class-relevant features that are jointly present in most examples but partially missing in the remainder, a structure common in vision datasets.The motivating example has 80% multi-view data and 20% single-view data; car images may reveal class identity through headlights, wheels, or windows, with occasional missing or misleading features.Source: §2.3 · pp. 6–7, §2.2 · p. 6
• Under multi-view data, independently initialized networks learn complementary features, so an ensemble recovers all views and improves test accuracy while distillation transfers those views into one model.Each network first learns one feature per class, correctly classifying 90% of examples, then memorizes the remaining 10%; ensembles covering both features classify all data correctly, while soft outputs force distilled models to learn missing features.Source: §2.3 · pp. 7–8

2.4 Significance of Our Technique

• The technique links generalization to optimization because initialization randomness determines feature-learning order, unlike convex or NTK settings where features are prescribed or initialization does not affect the unique solution.The authors argue that understanding ensemble and distillation requires integrating training dynamics with generalization rather than treating optimization and statistics separately.Source: §2.4 · p. 8

3 Problem Setup

• The theoretical analysis primarily uses parameter choices such as P = k^2 and µ = 1/poly(k) for sufficiently large k, while the appendix covers a broader valid parameter range.The training set uses N = k^1.2/µ samples in the main setting, with each label represented sufficiently often among single-view examples.Source: §3 · p. 9, §3.1 · p. 10

3.1 Data Distribution and Notations

• The data distribution mixes multi-view and single-view examples in a k-class, P-patch classification problem, with each patch d-dimensional and P = k^2.The distribution D samples multi-view data with probability 1−µ and single-view data with probability µ; the main setting uses sufficiently large k and d = poly(k).Source: §3.1 · pp. 9–10
• Each class has two orthogonal unit features that appear across selected patches, while other patches contain feature or Gaussian noise and may include features from competing classes.Feature weights, patch assignments, and noise distributions can vary, including different weights for the two views of the same class.Source: §3.1 · pp. 9–10

3.2 Learner Network

• The learner is a moderately over-parameterized two-layer convolutional network with smoothed ReLU, trained by gradient descent on cross-entropy from random initialization.Smoothed ReLU is linear for large inputs and suppresses low-magnitude feature noise; the analysis allows m from polylog(k) to k hidden units per class and focuses on q = 4.Source: §3.2 · p. 11

4 Main Theorems and Explanations

• A single model achieves perfect training accuracy but consistently poor test accuracy, around 0.5µ, despite random initialization.With high probability, its test accuracy lies in [0.49µ, 0.51µ].Source: §4 · pp. 11–12
• An ensemble of K = e^Ω(1) independently trained models retains perfect training accuracy and attains almost-perfect test accuracy with only m = polylog(k) over-parameterization.The improvement follows because initialization-dependent lottery-winning feature sets collectively cover all possible features.Source: §4 · p. 12

4.1 Knowledge Distillation for Ensemble

• Ensemble distillation trains a new, independently initialized network against truncated, scaled ensemble logits and yields perfect training and almost-perfect test accuracy for most iterations.The truncation stabilizes analysis, while the distillation objective uses a larger learning rate than the cross-entropy objective.Source: §4.1 · pp. 12–13
• The distilled model has learned all features available in the ensemble, so independently distilling multiple individual models provides no further ensemble performance boost.This interpretation is consistent with the empirical finding reported in Figure 8.Source: §4.1 · p. 13

4.2 Self Knowledge Distillation as Implicit Ensemble

• Under balanced feature marginals, self-distillation from an independently trained model improves test accuracy while preserving perfect training accuracy.The theorem holds with probability at least 1 −e^−Ω(log^2 k), and the resulting model generalizes better than both the original model and the teacher model.Source: §4.2 · p. 14
• Self-distillation implicitly ensembles the student and teacher feature sets and distills their union into the student through the original and distillation gradients.The proof interprets this as merging ensemble-then-distill into directly training one model to match another.Source: §4.2 · p. 14

• Figure 6 compares independent models, their ensemble, direct summed-model training, ensemble distillation, and self-distillation, showing that deep-learning ensemble gains cannot be reproduced by direct training.Directly training the summed network fails to improve test accuracy over single models, whereas ensemble distillation and self-distillation work well; the tested single models match state-of-the-art CIFAR-10/100 performance.Source: Figure 6 · p. 15, §5 · p. 15
• Knowledge distillation works for neural-network ensembles but not random-feature ensembles, and models trained by distillation from different seeds gain little from being ensembled.Figure 8 supports that distillation-trained models have learned most ensemble features and exhibit less variety than independently hard-label-trained models.Source: §5 · pp. 16–17, Figure 8 · p. 17
• Unlike random-feature or NTK models, deep-learning ensemble gains can be distilled into one model, and self-distillation substantially improves real neural networks.For random-feature mappings, direct summed training can outperform the ensemble, but the ensemble’s superior performance cannot be distilled; these contrasts suggest studying neural networks as feature-learning systems rather than prescribed-feature selection systems.Source: §5 · pp. 15–16
• Ensemble gains are not explained by variance reduction: CIFAR-100 individual test accuracies vary by only 0.1 ∼0.4% standard deviation, while Gaussian-like-data models vary by 0.5 ∼1.0% yet gain nothing from ensembling.Ensemble helps standard datasets with essentially no label noise, but fails on Gaussian-like data even with label noise; individually trained models typically already reach 100% training accuracy and nearly zero training loss.Source: §5 · p. 16
• Special data structure, particularly multi-view structure, is presented as necessary for deep-learning ensemble gains, whereas Gaussian-like data without it shows no improvement despite non-convex training and varied random seeds.CIFAR-10/100 are described as possessing multi-view structure that standard networks such as ResNets exploit during training.Source: §5 · p. 17, Figure 7
• The experiments identify hard-label training as a key source of the feature diversity that makes neural-network ensembles effective.Knowledge-distilled models retain most features from the ensemble, so their ensemble does not exceed the original single-model ensemble, consistent with the theoretical account.Source: §5 · p. 17

6 Conclusion and Discussion

• The work provides, to the authors’ knowledge, the first theoretical result on deep-learning ensembles and evidence that they differ from ensembles in random feature models.Ensembling does not always improve deep-learning test accuracy, particularly for Gaussian-like input distributions.Source: §6 · p. 17
• Multi-view structure lets ensembles improve test accuracy and lets a single model match the ensemble by simulating its outputs on the same training data.The distilled model can outperform any single clean model trained directly on the original labels.Source: §6 · pp. 17–18
• Figure 9 supports the multi-view hypothesis: collapsing or discarding input channels often barely reduces single-model accuracy, while ensemble accuracy changes little even when individual accuracy drops.The authors relate this behavior to distinct features learned across ResNet channels and ensembles collecting complementary views.Source: Figure 9 · p. 18
• The multi-view framework may extend to random-cropping data augmentation and guide principled methods for improving one network toward ensemble-level accuracy.The proposed extension is presented as a potential application rather than a proved result.Source: §6 · p. 18

Appendix I: Missing Details

• Appendix I Section A formally defines the data distribution, expanding Section 3.1 with fuller discussion and complete noise-parameter specifications.This appendix supplies missing formal details underlying the data model.Source: Appendix I · p. 19
• Appendix I Section B documents experiment setups and additional experiments, while Appendix II contains the full proofs beginning with Section C’s technical intuitions and proof plan.These sections provide implementation and proof-related materials supporting the main paper.Source: Appendix I · p. 19

A Data Distribution and Notations (Full Version)

• The theoretical setting uses k-class examples represented as P-patch inputs with patch dimension d, focusing on P = k^2 and d = poly(k).Source: A Data Distribution and Notations… · p. 19
• Each class has two unit features, assumed mutually orthogonal across classes and views, although the analysis also extends to incoherent features.Source: A Data Distribution and Notations… · p. 19
• The distributions construct examples around the two label features plus Õ(s) minor features, assign features to disjoint patches, and distinguish multi-view data from single-view data by randomly selecting one label feature.The sparsity parameter satisfies s ∈ [1, k^0.2], while each feature receives C_p disjoint patches.Source: A Data Distribution and Notations… · p. 19
• The construction gives the selected feature vy,bℓ total weight much larger than vy,3−bℓ and minor features, while adding feature noise and independent Gaussian noise, with ρ = k−0.01 and Γ = 1/polylog(k).Feature noise coefficients satisfy αp,v′ ∈ [0, γ], and ξp is an independent random Gaussian noise.Source: A Data Distribution and Notations… · p. 20
• The concept class is designed so accumulated low-magnitude noise defeats linear classifiers, while low-degree polynomials cannot simultaneously fit the signal and suppress large-noise inputs.The passage motivates ReLU-like convolutional neural networks as necessary in this setting and conjectures analogous limitations for kernel methods.Source: A Data Distribution and Notations… · p. 21
• The final distribution mixes multi-view samples with probability 1 − μ and single-view samples with probability μ, using N training examples and focusing on μ = 1/poly(k).The stated sample choice is N = k^1.2/μ, so each label appears at least e^Ω(1) times in the single-view training set.Source: A Data Distribution and Notations… · p. 21

B Experiment Details

• Experiments use CIFAR-10/100 with SimpleCNN and pre-activation ResNet architectures, including width-scaled variants derived from prior architectures [8] [88].Source: B Experiment Details · p. 21
• Regular neural networks use momentum SGD with batch size 125 for 140 epochs, scheduled learning-rate decays, and crop, flip, normalization, and cutout augmentation.Neural-kernel models instead use Adam with initial learning rate 0.001, batch size 50, 200 epochs, ZCA preprocessing, and cutout augmentation.Source: B Experiment Details · p. 21

B.1 Real-Life Data: Single Model vs Ensemble vs Distillation

• Figure 6 evaluates single models, ensembles, direct training, knowledge distillation, and self-distillation on CIFAR-10/100 using independently initialized runs.Ensemble predictions average the outputs of 10 independently trained models, while single-model ResNet results report means and standard deviations across runs.Source: §B.1 · pp. 21–22
• Self-distillation uses a best checkpoint trained with hard labels to supervise a newly initialized model through the knowledge-distillation objective [42].The distillation objective tunes its cross-entropy-to-distillation weight and temperature parameters within a reasonable range.Source: §B.1 · p. 22
• Repeated self-distillation yields only incremental additional accuracy gains, while a three-stage procedure achieves test accuracy extremely close to the two-stage procedure.The three-stage variant trains independent models F and G, then distills G to match F.Source: §B.1 · p. 22

B.2 Real-life Data: Ensemble over Distillations of Ensemble

• Figure 8 studies whether ensembling models distilled from an ensemble improves performance by training 10 distillation models and averaging their outputs.The distillation runs share all hyperparameters and differ only in initialization and SGD randomness; the final ensemble is an unweighted average.Source: §B.2 · p. 22

B.3 Real-life Data: Justifying the Multi-View Assumption

• Figure 9 investigates whether real-life training exhibits multiple data views that support image labels, including at intermediate ResNet layers.The experiments treat outputs after the first or second block of a pretrained ResNet as inputs while training later blocks.Source: §B.3 · pp. 22–23
• The multi-view experiment varies ResNet-28-M with M ∈{1, 2, 4, 10} and compares channel splitting or channel averaging while training eight models.Splitting uses N/8, N/4, or N/2 channel chunks, whereas averaging reduces inputs to N/8, N/4, or N/2 channels.Source: §B.3 · p. 23
• Figure 9 reports accuracies for the eight single models and their ensemble, using distinct seeds for pretrained upper layers to make ensemble comparisons fair.Single-model accuracies occupy the first half of the figure rows, and ensemble accuracies occupy the second half.Source: §B.3 · p. 23

B.4 Synthetic Data: Whether Ensemble Improves Accuracy over Gaussian-Like Data

• The synthetic benchmark uses k = 10 labels with Gaussian or Gaussian-mixture inputs, uniform or rejection-sampled class distributions, and either clean labels or 10% random label flips.These settings vary input distributions, class balance, and label noise while testing ensemble behavior on Gaussian-like data.Source: §B.4 · p. 23
• Data are generated from unknown small ground-truth networks that may be linear, fully connected, convolutional, or residual.The considered architectures include fc2, conv3, res3, and resconv3 examples.Source: §B.4 · p. 23
• The study also compares ordinary generation with generation imposing margins across labels and selects dimension d to yield roughly 60%–80% single-model accuracy.Dimension selection targets a moderate single-model test-accuracy range for each setting.Source: §B.4 · p. 23
• Learners include fully connected, convolutional, and residual networks with m = 200 neurons, trained using momentum-SGD with tuned learning rate and weight decay.For each data/learner pair, the experiment trains 10 single models and compares their best accuracy with ensemble accuracy.Source: §B.4 · p. 23
• The 10 single models show negligible accuracy variation across random initializations, such as a 1.0% standard deviation around 70% test accuracy.Figure 12 reports standard deviations over 10 runs and shows nearly no ensemble benefit despite greater variability than on CIFAR10/100.Source: §B.4 · p. 24, Figure 12 · p. 27
• Ensembles barely exceed the best single-model accuracy across a plethora of Gaussian-like synthetic dataset settings.The abbreviated comparison selects the best single and ensemble models across learner networks; detailed results appear in Figures 10 and 11.Source: §B.4 · p. 24, Figure 10 · p. 25, Figure 11 · p. 26
• Data generated from a target convolutional network provide evidence that convolutional data alone may not be sufficient for ensembles to work.Source: §B.4 · p. 26

C Single Model: Proof Plan and Induction Hypothesis

• The induction hypothesis tracks feature correlations, noise contributions, and regularization throughout training, implying that one view can remain missing and single-view accuracy cannot exceed 49.99%.It also separates multi-view correlations, which support correct classification, from noise terms that enable memorization on single-view data.Source: C Single Model: Proof Plan and… · pp. 28–29

D Single Model: Technical Proofs

• The technical proof establishes that Induction Hypothesis C.3 holds for every iteration t = 0, 1, . . . , T under the stated parameter regime.The proof section specifies ranges for activation smoothing, off-target feature magnitude, single-view sample size, and feature magnitudes.Source: D Single Model: Technical Proofs · p. 29

D.1 Gradient Calculations and Function Approximation

• Gradient calculations bound positive and negative gradients, while function approximations control logits and related error terms under the induction hypothesis.These bounds apply to multi-view data and to incorrect-class logits on single-view data, with the stated sparsity and parameter conditions.Source: §D.1 · pp. 30–32

D.2 Useful Claims as Consequences of the Induction Hypothesis

• The induction hypothesis’s consequences provide the intermediate claims needed to complete the induction proof.The subsequent analysis uses these consequences to establish correlation growth and control single-view behavior.Source: §D.2 · p. 32

D.2.1 Correlation Growth

• Feature correlations grow from sufficiently large initialization, and after threshold times the relevant correlations exceed the defined upper or lower thresholds.The growth argument distinguishes correct-label contributions from wrong-label cases and relies on the positive and negative gradient bounds.Source: §D.2.1 · pp. 32–34

D.2.2 Single-View Error Till the End

• For single-view data, noise correlations grow enough to support memorization, yielding bounds on single-view error from T0 through the end of training.Claim D.12 is proved using the noise lower bound in Claim D.13, under γ ≤eO(σ0k).Source: §D.2.2 · pp. 34–35
• F(t) ≤ e−log5 k for every t ≥ t1, proving Claim D.12a.Source: §D.2.2 · p. 35
• Claim D.12b is proved by contradiction using the induction hypothesis after ReLU enters the linear regime.Source: §D.2.2 · p. 35
• The iterations from T0 to T are partitioned into 4m0 consecutive stages, with each stage having a similar partial sum.Source: §D.2.2 · pp. 35–36
• Continuing the stage argument through stage T2m0, the cited passages state that the resulting conclusion holds for every t and for all (r, ℓ) ∈M(0) i × [2].Source: §D.2.2 · p. 36

• Claim D.14 establishes multi-view convergence through the end when Induction Hypothesis C.3 holds for every t < T and Ns ≤ k^2ρ^(q−1).Source: §D.2.3 · p. 36
• Claim D.14 follows from Claim D.15 together with Λ_i^(t) = eO(1) from Induction Hypothesis C.3g and Claim D.12b for single-view convergence.Source: §D.2.3 · p. 36
• For t ≥ T0, Claim D.15 applies under Induction Hypothesis C.3 and distinguishes multi-view data from single-view data with incorrect index i ≠ y.Source: §D.2.3 · p. 37
• The distribution yields activation bounds: correct indices satisfy Vi,r,ℓ(X) ≥ 0.9 on Zm and ≥ 0 on Zs, while incorrect indices are bounded above by 0.4 on Zm and Γ ≤ 1 on Zs.Source: §D.2.3 · p. 37

D.2.4 Multi-View Individual Error

• The multi-view individual error is bounded by the training error up to a polynomial factor, including with high probability for samples from D^m.The claim applies to every multi-view data point and extends to a random sample with probability at least 1 − e^−Ω(log^2 k).Source: §D.2.4 · pp. 38–39

D.2.5 Multi-View Error in Stage 2

• During Stage 2, Claim D.17 bounds the maximum multi-view error under Induction Hypothesis C.3 and the parameter choice Υ = eΘ(1/k^0.2).The proof separates cases according to whether relevant view vectors belong to P(X), with probabilities involving (1 − s/k)^2 and s/k.Source: §D.2.5 · pp. 39–41

D.3 Tensor Power Method Bound

• Lemma D.19 shows that, under its growth-rate and initialization conditions, the sequence x_t reaches its threshold before the slower sequence y_t.The comparison applies to updates of the form x_{t+1} = x_t + ηC_t x_t^{q−1} and y_{t+1} ≤ y_t + ηS C_t y_t^{q−1}, with q ≥ 3.Source: §D.3 · pp. 41–42

D.4 Main Lemmas for Proving the Induction Hypothesis

• The technical lemmas in D.4 are introduced as the ingredients that will be combined in Section D.5 to prove Induction Hypothesis C.3.This subsection serves as the setup for the main induction analysis rather than presenting the combination itself.Source: §D.4 · p. 42

D.4.1 Lambda Lemma

• The Lambda Lemma bounds the relevant diagonal quantities by eO(1) under the induction hypothesis and the stated condition on N_s.Its proof controls auxiliary A and B sequences, with a negligible e^−Ω(log^5 k) contribution.Source: §D.4.1 · pp. 43–44

D.4.2 Off-Diagonal Correlations are Small

• Lemma D.22 bounds off-diagonal correlations across all three training stages, with the tightest parameter restrictions arising in the Stage 2 analysis.The required upper bounds on γ and s are explicitly identified as the tightest parameter choices in the proof.Source: §D.4.2 · pp. 45–47

D.4.3 View Lottery Winning

• The view lottery selects a winning view for each label, while the other view’s contribution remains negligible, with the proof covering three training stages.The bounds are established through the early, intermediate, and late iteration regimes under the induction hypothesis and parameter restrictions.Source: §D.4.3 · pp. 47–49

D.4.4 Neuron Lottery Winning: M(0)

• Neurons outside M(0) have negligible correlations for each label, so only neurons in M(0) win the neuron lottery.The proof reuses the three-stage argument for view lottery winning, with the corresponding induction hypothesis substituted where needed.Source: §D.4.4 · p. 50

D.4.5 Noise Correlation is Small

• Noise correlations are negligible for multi-view data and non-winning cases, except that single-view examples on lottery-winning views can correlate significantly with their noise.The bound includes |⟨w(t) i,r, ξp⟩| ≤ eO(σ0γk) under the stated conditions N ≥ k5 and (σ0γk)q−2 ≤ ρq−1.Source: §D.4.5 · pp. 50–51

D.4.6 Diagonal Correlations are Nearly Non-Negative

• The analysis proves that diagonal correlations cannot remain substantially negative during the relevant iterations, establishing the required near-nonnegativity control.The result is derived for every t ∈ [T0, T] under the earlier parameter choices.Source: §D.4.6 · pp. 52–53

D.5 Putting All Together

• Under Parameter D.1 and sufficiently small η ≤ 1/poly(k), Induction Hypothesis C.3 holds for all iterations t = 0, 1, . . . , T.The proof verifies each component C.3a–C.3i using the established bounds and claims.Source: §D.5 · pp. 53–54

Single model

• Under Parameter D.1 and sufficiently large k, a single model achieves perfect training accuracy with probability at least 1 −e−Ω(log2 k).The theorem applies after η many iterations and for every m ∈.Source: E Single Model and Ensemble: Theorem… · p. 54
• The single model has good multi-view testing but bad single-view testing: the former is bounded below by 0.4Φ(T), whereas the latter is at most eO.These properties hold for every relevant view pair and every (i, ℓ) ∈M.Source: E Single Model and Ensemble: Theorem… · p. 54
• With two equally probable views, the single-model testing accuracy is close to 1; more general view probabilities can yield another constant µ.The equal-probability assumption explains the stated accuracy level and is presented as a simplifying case.Source: E Single Model and Ensemble: Theorem… · p. 54

Ensemble

• In the same setting, the ensemble has perfect training and good multi-view testing, matching the corresponding single-model properties.The ensemble theorem explicitly states both properties as the same as for the single model.Source: E Single Model and Ensemble: Theorem… · p. 55
• When K ≥ polylog(k), the ensemble achieves good single-view testing, unlike the single model.The supplied theorem statement identifies this as the ensemble’s distinct single-view result.Source: E Single Model and Ensemble: Theorem… · pp. 54–55

E.1 Proof of Theorem 1

• Theorem 1’s proof concludes that full gradient descent reaches perfect training accuracy because the objective is monotonically non-increasing.The monotonicity follows from the objective being O(1)-Lipschitz continuous under full gradient descent.Source: §E.1 · p. 55
• For multi-view test accuracy, the proof combines Claim D.16 with function-approximation Claim D.9 to establish the required high-probability behavior at iteration T.The supplied passage states the proof’s dependence on the cross-index bound and function approximation, but does not include the resulting inequality in full.Source: §E.1 · p. 55
• For single-view data, the proof bounds the target-class output by O(ρ) + 1/polylog(k) in the relevant case.This bound uses the single-view distribution definition together with Claim D.9 and the event that the observed view differs from the relevant label view.Source: §E.1 · p. 55
• Theorem 1’s conclusion holds with probability at least 1/2(1 − o(1)) for every relevant tuple, and the proof finishes using |M| ≥ k(1 − o(1)).The probability statement is established for every (i, ℓ) ∈ M and every single-view example with y = i.Source: §E.1 · p. 56

E.2 Proof of Theorem 2

• Theorem 2’s proof shows that an ensemble of K ≥ m^Ω(1) models achieves G_y(X) ≥ G_i(X) + 1/polylog(k) for every incorrect class i, assuming Γ ≤ 1/m^Ω(1).The result follows after averaging the models’ outputs under the stated ensemble-size and Γ conditions.Source: §E.2 · p. 56

F Knowledge Distillation: Theorem Statement

• The knowledge-distillation analysis defines a truncated scaled logit using temperature τ = 1/log^2 k to study improved generalization from ensemble and self-distillation.The truncation is chosen because it is easier to analyze than the temperature-based logit function associated with the seminal knowledge-distillation work [42].Source: F Knowledge Distillation: Theorem… · p. 56

F.1 Using Ensemble for Knowledge Distillation

• Ensemble distillation achieves perfect training accuracy, good multi-view testing, and good single-view testing for most iterations under polynomial-time and learning-rate conditions.Theorem 3 applies to K=e^Θ(1) independently trained models and holds with probability at least 1−e^−Ω(log^2 k) for at least 90% of iterations.Source: §F.1 · p. 57

F.2 Self-Distillation: Using a Single Model to Distill Itself

• Self-distillation uses a two-stage learn-then-distill procedure and achieves perfect training accuracy, good multi-view testing, and improved single-view testing under the balanced-data assumption.The model is first trained from random initialization, then distilled for another T′=poly(k)/η iterations after scaling the teacher network by log^4 k.Source: §F.2 · p. 57

G Knowledge Distillation Proof for Ensemble

• The ensemble-distillation proof maintains a simpler induction hypothesis than the single-model proof, requiring shorter training because ensemble distillation learns all training data directly.The proof permits m=polylog(k) and uses N≥ηT·poly(k), ηT≥poly(k), rather than the longer single-model training requirement.Source: G Knowledge Distillation Proof for… · pp. 58–59

G.1 Gradient Calculations and Function Approximation

• Gradient and function-approximation claims show that the distilled ensemble responds strongly when a feature appears in the input and weakly otherwise, supporting the induction hypothesis.After a polylogarithmic scale-up, Gi(X)≥log^4 k when either feature is present and Gi(X)≤1 when neither is present.Source: §G.1 · pp. 59–60

G.2 Useful Claims as Consequences of the Induction Hypothesis

• The proof’s useful claims establish correlation growth, control single-view error after a threshold T0, and bound multi-view error through the end of training.After T0, single-view examples obtain a large correct-class logit and inverse-polynomial error, while multi-view behavior is controlled by telescoping the updates.Source: §G.2.1 · p. 61, §G.2.2 · p. 61, §G.2.3 · p. 61

G.3 Main Lemmas for Proving the Induction Hypothesis

• The main lemmas preserve bounded correlations across iterations and show that correlations between distinct features remain small, completing the technical route to the induction hypothesis.Correlation growth remains bounded through T, while the off-diagonal-correlation lemma separately handles iterations before and after T0.Source: §G.3.1 · pp. 62–63, §G.3.2 · pp. 63–64

G.3.2 Off-Diagonal Correlations are Small

• The analysis shows that off-diagonal correlations are small, with extremely small off-label logit terms and negligible single-view contributions in the relevant formula.The argument uses telescoping for all t≥T0 and applies Claim G.11.Source: §G.3.2 · p. 64

G.3.3 Noise Correlation is Small

• Noise correlation is bounded through Lemma G.14 under Parameter G.2 and the induction hypothesis, with the displayed proof relying on N≫T.The text notes that T≥N would require a more careful calculation and is omitted for concision.Source: §G.3.3 · pp. 64–65

G.3.4 Diagonal Correlations are Nearly Non-Negative

• Lemma G.15 establishes that diagonal correlations are nearly non-negative under Parameter G.2 and the induction hypothesis.Its proof is stated to be almost identical to Lemma D.27 and is skipped.Source: §G.3.4 · p. 65

G.4 Proof of Theorem 3

• Theorem 3’s proof concludes the test-accuracy result for multi-view data and separately for single-view data.The multi-view conclusion invokes Claim D.16; the single-view conclusion follows when i,ℓ≥Ω(log k).Source: §G.4 · p. 65

G.5 Proof of Theorem 4

• Assumption 4.1 and concentration identify the relevant self-distillation set MF as a subset of M, allowing the established statements about M to transfer.The two view-specific distributions are assumed identical on multi-view data.Source: §G.5 · p. 65
• After stage 2, F retains nearly perfect accuracy on multi-view data because its relevant Φi values already exceed the required threshold before stage 2.The stage-1 lower bound remains sufficient after scaling, so Φi does not change during stage 2.Source: §G.5 · p. 66
• For indices in MG, stage 2 raises Φi,ℓ enough that F correctly predicts the corresponding single-view examples with high probability.The resulting target logit is Ω(log^2 k) and dominates the competing logits.Source: §G.5 · p. 66
• For indices in MF, the stage-1 lower bound persists through stage 2, so F also correctly predicts their corresponding single-view examples.Scale-up strengthens the bound to Ω(log^5 k) at the end of stage 2.Source: §G.5 · p. 66
• Self-distillation combines the independently learned sets MF and MG, yielding the Theorem 4 accuracy guarantee through their high-probability union.The sets are treated as independent and each has size k(1 − o(1)), which supports the final accuracy conclusion.Source: §G.5 · p. 66

H Simple Probability Lemmas

• Proposition H.1 bounds Gaussian maxima so that, with probability at least 1 − δ, only O(log(1/δ)) indices can violate the stated threshold behavior.This proposition is used as a basic concentration ingredient for the later random-set arguments.Source: H Simple Probability Lemmas · p. 67
• Proposition H.2 controls comparisons between maxima of independent Gaussian sequences with variances 1 and σ^2, including the stated O(τ log m + 1/poly(m)) bound.Its proof treats σ ≤ 1 and σ > 1 separately using Gaussian tail and anti-concentration estimates.Source: H Simple Probability Lemmas · pp. 67–68
• These Gaussian lemmas imply Proposition C.2: at most one of (i,1) and (i,2) belongs to M, while the probability that neither belongs is at most 1/log m.The conclusion follows from constant-probability ratios between the competing scores and Proposition H.2.Source: H Simple Probability Lemmas · p. 68