PaperlayerView exact revision on arXiv

Source-linked AI summary

Self-Distillation Amplifies Regularization in Hilbert Space

Hossein Mobahi, Mehrdad Farajtabar, Peter L. Bartlett

arXiv:2002.05715v3cs.LGstat.ML

TL;DR

Self-distillation often improves held-out accuracy despite receiving no new task information, but the reason has lacked rigorous understanding. This paper analyzes it in Hilbert-space regularized regression and shows that repeated iterations progressively restrict basis functions, potentially shifting from reduced over-fitting to under-fitting.

  • Problem

    Self-distillation can improve held-out accuracy even though its repeated training loop receives no new task information, and this phenomenon lacks rigorous understanding.

  • Method

    The paper theoretically analyzes self-distillation for nonlinear regression in a Hilbert space with ℓ2 regularization and empirically studies related behavior in neural networks.

  • Results

    Self-distillation progressively limits the basis functions representing the solution, while experiments show training accuracy drops and test accuracy can first rise before falling from over-regularization.

  • Takeaways & Limitations

    A few self-distillation rounds may reduce over-fitting, but excessive rounds may cause under-fitting and worse performance.

  • Takeaways & Limitations

    The analysis is posed for Hilbert-space regularized regression, and extending it to cross-entropy loss or other regularizers remains future work.

Abstract

from arXiv · show

Knowledge distillation introduced in the deep learning context is a method to transfer knowledge from one architecture to another. In particular, when the architectures are identical, this is called self-distillation. The idea is to feed in predictions of the trained model as new target values for retraining (and iterate this loop possibly a few times). It has been empirically observed that the self-distilled model often achieves higher accuracy on held out data. Why this happens, however, has been a mystery: the self-distillation dynamics does not receive any new information about the task and solely evolves by looping over training. To the best of our knowledge, there is no rigorous understanding of this phenomenon. This work provides the first theoretical analysis of self-distillation. We focus on fitting a nonlinear function to training data, where the model space is Hilbert space and fitting is subject to $\ell_2$ regularization in this function space. We show that self-distillation iterations modify regularization by progressively limiting the number of basis functions that can be used to represent the solution. This implies (as we also verify empirically) that while a few rounds of self-distillation may reduce over-fitting, further rounds may lead to under-fitting and thus worse performance.

1 Introduction

Self-distillation repeatedly uses a trained model’s predictions as targets, yet its generalization benefit has lacked rigorous explanation. This paper analyzes the phenomenon theoretically in regularized nonlinear regression and connects it to progressively stronger regularization.

  • Self-distillation uses a model’s predictions as new target values and repeats retraining when teacher and student architectures are identical.
  • Although self-distilled models often achieve higher held-out accuracy, the dynamics gains no new task information and its generalization benefit remains unexplained.
  • The paper studies nonlinear regression with functions in a Hilbert space and ℓ2 regularization in that function space.
  • Self-distillation progressively limits the number of basis functions used to represent the solution, modifying the effective regularization.
  • A few self-distillation rounds may reduce over-fitting, whereas further rounds may cause under-fitting and worse performance.
  • The paper analyzes the process, relates the setting to the NTK regime, and empirically verifies behavior consistent with its regularization viewpoint.

2 Problem Setup

The paper formulates nonlinear regression as constrained, regularized function fitting in a Hilbert space and derives a kernel-based solution representation. It then establishes conditions for nontrivial solutions and bounds the multiplier governing the solution.

  • Problem Setup: The problem uses finite training pairs and seeks a function approximating the underlying mapping, with regularization supplying inductive bias among multiple fitting functions.
  • Problem Setup: The constrained formulation requires training loss within tolerance ϵ while minimizing a nonnegative regularization functional R(f).
  • Problem Setup: The regularizer is constructed from kernel-integral operators and weighted penalties, with symmetry and positivity ensuring the relevant operator is positive definite.
  • Problem Setup: The zero function is the collapse solution when normalized label energy is at most ϵ; the analysis therefore focuses on the non-collapse case.
  • Problem Setup: The representer-theorem solution uses the Green’s function and a positive-definite Gram matrix G, whose eigenvalues are strictly positive.
  • Bounds on Multiplier c: The multiplier c is determined through a nonlinear equation involving matrix inversion, so closed-form roots are unavailable; the paper instead derives lower and upper bounds.
  • Bounds on Multiplier c: Rotating labels by the orthogonal eigenvector matrix simplifies analysis because results can be converted back without changing their ℓ2 norm.

3 Self-Distillation

Self-distillation repeatedly retrains on the model’s predictions, causing the representation matrix to become progressively sparser and limiting the effective basis functions used by the solution. The process has a finite non-collapsing regime, while early stopping produces a different regularization effect.

  • Self-Distillation Process: Self-distillation uses each model’s training-data predictions as targets for the next training round.Standard training is denoted t = 0, and the next round uses y1 ≜ f0 as its target vector.
  • Collapse: Self-distillation cannot continue indefinitely: when the required norm condition fails, the solution collapses to the zero function and remains there.The analysis provides a lower bound on the number of meaningful rounds before collapse.
  • Evolution of Basis: The time-dependent matrix Bt is the product of diagonal matrices accumulated across self-distillation rounds.This matrix is the only dynamic component in the learned-function representation.
  • Evolution of Basis: As t increases, Bt becomes progressively sparser because smaller diagonal elements shrink faster than larger ones.In the limiting picture, only the entry associated with dmax remains significant relative to the others.
  • Evolution of Basis: The number of significant diagonal entries of Bt determines the effective number of basis functions used to represent the learned solution.Sparsification therefore reduces the solution’s representational expressiveness.
  • Self-Distillation versus Early Stopping: Unlike self-distillation, loss-based early stopping does not further sparsify the basis and can preserve or densify the original pattern.Its regularization effect arises from expanding the feasible set by increasing the error tolerance ϵ.

4 Illustrative Example

The illustrative example shows how self-distillation changes regularization in a Hilbert-space regression problem: smaller basis coefficients shrink faster, improving the fit initially but eventually causing underfitting.

  • Setup: The example fits a twice-differentiable function on [0, 1] under boundary conditions f(0) = f(1) = f′′(0) = f′′(1) = 0.The associated regularizer penalizes the function’s second-order derivative and therefore encourages smoother functions.
  • Setup: Noisy samples from y = sin(2πx) produce overfitting when the data are fit without regularization.The training points are evenly spaced by 0.1, with Gaussian noise of σ = 0.5.
  • Self-distillation effects: Self-distillation initially improves the quality of the fit, but additional rounds eventually over-regularize it and cause underfitting.The illustrative fit uses four self-distillation rounds with ϵ = 0.04.
  • Self-distillation effects: Self-distillation sparsifies the solution’s basis-function coefficients because smaller coefficients shrink faster across rounds.The diagonal entries of B_t represent coefficients of basis functions used to represent the solution; Figure 3 tracks them from t = 0 to t = 3 for K = 11.

5 Experiments

The experiments test the regularization account of self-distillation in neural networks across datasets, losses, architectures, and comparisons with early stopping. They observe the predicted pattern of initial test improvement followed by decline from over-regularization, while noting that the theoretical NTK connection is not always faithful for finite-width networks.

  • NTK connection: The analysis can represent self-distillation in the NTK regime because both the studied regularized regression and NTK training reduce to kernel regression.Finite-width architectures may not always be faithfully approximated by the NTK analysis.
  • Interpretation: The empirical pattern is consistent with progressively amplified regularization: test performance may rise initially but decline after excessive self-distillation.The authors state that their deep-model results follow this pattern.
  • Setup: The experiments use Resnet and VGG networks trained on CIFAR-10 and CIFAR-100, with each plotted curve averaging 10 randomly initialized runs.The x-axis records self-distillation rounds, and envelopes show standard deviation.
  • 5.1 Motivation: Under ℓ2 loss, training accuracy drops across self-distillation rounds, while test accuracy can first increase and later decrease from over-regularization and underfitting.Both trends are reported for the CIFAR-10 and CIFAR-100 experiments in Figure 4.
  • 5.2 Results: Similar self-distillation behavior is observed empirically with cross-entropy loss, although the theory applies only to ℓ2 loss.The cross-entropy experiments use Resnet on CIFAR-10 and CIFAR-100.
  • 5.2 Results: The experiments compare self-distillation with early stopping and test ℓ2 self-distillation on VGG16 to examine whether the findings extend beyond one architecture.Figure 6 covers Resnet50 on CIFAR-10 and VGG16 on CIFAR-100.

6 Conclusion

The paper rigorously analyzes self-distillation for regularized regression in a Hilbert space, showing that repeated iterations progressively restrict the basis functions used and eventually collapse the solution. It also contrasts this regularization effect with early stopping and identifies near-interpolation as favorable for the effect.

  • 6 Conclusion: Self-distillation iterations cannot continue indefinitely: the solution eventually collapses to the zero function.The analysis provides a lower bound on the number of meaningful, non-collapsed iterations.
  • 6 Conclusion: Self-distillation acts as a regularizer by progressively employing fewer basis functions to represent the solution.
  • 6 Conclusion: The paper discusses how self-distillation’s regularization effect differs from early stopping.
  • 6 Conclusion: Operating in the near-interpolation regime facilitates the regularization effect.
  • 6 Conclusion: The regression setting resembles the NTK view of wide neural networks and may provide insight into self-distillation in deep learning.
  • 6 Conclusion: Broader settings, including cross-entropy loss and other regularization forms, remain directions for further research.

B Equivalent Kernel Regression Problem

The self-distilled solution can be represented as standard kernel ridge regression with a suitably chosen kernel. This equivalent formulation connects the transformed Gram-matrix spectrum to the original spectrum and enables existing kernel generalization bounds.

  • B Equivalent Kernel Regression Problem: After t rounds of self-distillation, the regularized solution has a form that can be represented using a positive definite kernel.
  • B Equivalent Kernel Regression Problem: The self-distilled solution can be equivalently written as standard kernel ridge regression with a properly chosen kernel g†.
  • B Equivalent Kernel Regression Problem: The spectrum of the transformed Gram matrix G† relates to the spectrum of the original Gram matrix G.
  • B Equivalent Kernel Regression Problem: The transformed kernel is constructed so that its operator shares the original operator’s eigenfunctions while varying its eigenvalues.

C Proofs

The proofs establish bounds and monotonicity properties governing self-distillation’s duration and sparsifying regularization effect. In particular, meaningful iterations are guaranteed only before collapse, while lower interpolation tolerance yields higher sparsity.

  • C Proofs: The variational problem admits a solution of the stated form under the paper’s setup.
  • C Proofs: A lower bound guarantees that meaningful, non-collapsing self-distillation is possible for at least t rounds.
  • C Proofs: The exact collapse time is difficult to determine, so the proofs derive sufficient lower bounds on the iterates to guarantee non-collapse.
  • C Proofs: The sparsity index decreases with ε, meaning that lower ε yields higher sparsity.

D More on Experiments

The experiments use a reproducible, lightweight deep-neural-network setup with extended self-distillation runs on CIFAR-10 and CIFAR-100.

  • D More on Experiments: The experiments use Adam with learning rates 0.001 for CIFAR-10 and 0.0001 for CIFAR-100.
  • D More on Experiments: Training runs for up to 64000 steps, with batch sizes 16 for CIFAR-10 and 64 for CIFAR-100.
  • D More on Experiments: Weight decay regularization uses coefficients 0.0001 for CIFAR-10 and 0.00005 for CIFAR-100.
  • D More on Experiments: The experiments use the standard 50000-train and 10000-test CIFAR split.

D.2 ℓ2 Loss on Neural Network Predictions

Experiments with ℓ2 loss on neural-network predictions show that self-distillation first strengthens regularization, then causes under-fitting as iterations continue. The same trend appears on CIFAR-10 and CIFAR-100 with ResNet-50.

  • Self-distillation on CIFAR-10 and CIFAR-100 exhibits a similar progression under ℓ2 loss of neural-network predictions.The cited figures cover ResNet-50 experiments on both datasets.
  • Training loss first decreases as regularization is amplified, then increases when the model under-fits.This pattern is reported for the CIFAR-10 results.
  • The observed regularization mechanism differs from early stopping.
  • Figures 7 and 8 provide full results for ResNet-50 on CIFAR-10 and CIFAR-100, respectively.

E Mathematica Code To Reproduce Illustrative Example

The Mathematica example constructs a nonlinear regression problem, computes its kernel-related quantities, and iterates self-distillation while recording functions and spectral coefficients for visualization.

  • The example generates sample inputs and noisy sinusoidal targets, then plots the data against the ground-truth sine function.
  • The implementation limits the loop to MaxIter = 10 and stops early when no suitable root is found.
  • The code defines a loss function using a regularized inverse involving G and evaluates the mean squared residual.
  • The self-distillation loop finds a positive regularization coefficient, updates the solution, and records function and coefficient sequences.
  • Plots visualize the fitted function and the recorded coefficient sequences using line and bar charts.

F Python Implementation

The Python implementation provides a TensorFlow self-distillation loss and training loop. It replaces the teacher targets after the initial model and combines prediction loss with ℓ2 weight regularization.

  • The implementation defines self-distillation through a customized loss that can replace ground-truth labels with teacher predictions.
  • Without a teacher, the loss compares labels with the model’s softmax predictions; with a teacher, it compares teacher and student softmax outputs.
  • The total loss adds an ℓ2 regularization term over the model’s trainable weights to the prediction loss.
  • Training iterates over epochs and batches with TensorFlow GradientTape, computes gradients, and applies them through an optimizer.
  • After each distillation step, the trained model becomes the teacher for the next step.
Loading 2002.05715v3…