PaperlayerView exact revision on arXiv

Source-linked AI summary

Implicit Self-Regularization in Deep Neural Networks: Evidence from Random Matrix Theory and Implications for Learning

Charles H. Martin, Michael W. Mahoney

arXiv:1810.01075v1cs.LGstat.ML

TL;DR

The paper addresses limited understanding of how regularization operates in DNNs and applies RMT to weight matrices and their ESDs. It develops a 5+1-phase taxonomy of implicit Self-Regularization and evaluates it across smaller trained models and modern pretrained networks. The results distinguish Tikhonov-like behavior in older or smaller models from Heavy-Tailed Self-Regularization in modern models, while batch-size changes produce phase transitions associated with the generalization gap.

  • Problem

    The paper asks why regularization in deep learning differs from traditional machine learning and what theoretical framework can investigate DNN regularization.

  • Method

    The paper applies Random Matrix Theory to DNN layer weight matrices, empirical spectral densities, and capacity-control metrics, modeling phases through noise-plus-signal descriptions.

  • Results

    The study identifies 5+1 training phases, finds Tikhonov-like Self-Regularization in older or smaller models and Heavy-Tailed Self-Regularization in modern models, and exhibits all phases by changing MiniAlexNet’s batch size.

  • Takeaways & Limitations

    Implicit Self-Regularization depends on training-process choices: smaller batches produce stronger regularization and higher generalization accuracy, while larger batches produce less-well implicitly-regularized models.

  • Takeaways & Limitations

    The analysis considers individual dense-layer weight matrices and does not evaluate the proposed matrix constructions for convolutional or other multidimensional layers.

Abstract

from arXiv · show

Random Matrix Theory (RMT) is applied to analyze weight matrices of Deep Neural Networks (DNNs), including both production quality, pre-trained models such as AlexNet and Inception, and smaller models trained from scratch, such as LeNet5 and a miniature-AlexNet. Empirical and theoretical results clearly indicate that the DNN training process itself implicitly implements a form of Self-Regularization. The empirical spectral density (ESD) of DNN layer matrices displays signatures of traditionally-regularized statistical models, even in the absence of exogenously specifying traditional forms of explicit regularization. Building on relatively recent results in RMT, most notably its extension to Universality classes of Heavy-Tailed matrices, we develop a theory to identify 5+1 Phases of Training, corresponding to increasing amounts of Implicit Self-Regularization. These phases can be observed during the training process as well as in the final learned DNNs. For smaller and/or older DNNs, this Implicit Self-Regularization is like traditional Tikhonov regularization, in that there is a "size scale" separating signal from noise. For state-of-the-art DNNs, however, we identify a novel form of Heavy-Tailed Self-Regularization, similar to the self-organization seen in the statistical physics of disordered systems. This results from correlations arising at all size scales, which arises implicitly due to the training process itself. This implicit Self-Regularization can depend strongly on the many knobs of the training process. By exploiting the generalization gap phenomena, we demonstrate that we can cause a small model to exhibit all 5+1 phases of training simply by changing the batch size. This demonstrates that---all else being equal---DNN optimization with larger batch sizes leads to less-well implicitly-regularized models, and it provides an explanation for the generalization gap phenomena.

1 Introduction

The paper investigates why regularization in deep learning differs from traditional machine learning and uses RMT to characterize implicit Self-Regularization in DNN weight matrices. It proposes 5+1 training phases and reports Tikhonov-like behavior in smaller models, Heavy-Tailed Self-Regularization in modern models, and batch-size-dependent phase changes linked to the generalization gap.

  • 1 Introduction: DNN regularization is difficult to define because many training knobs and operational procedures can affect generalization beyond explicit loss penalties.The paper frames this as a theoretical question about the appropriate framework for investigating regularization in DNNs.
  • 1 Introduction: RMT analyzes DNN layer weight matrices through empirical spectral densities and capacity-control metrics, including Matrix Entropy, Hard Rank, Stable Rank, and MP Soft Rank.These metrics track the amount of Self-Regularization during training or in pretrained models.
  • 1 Introduction: Older or smaller DNNs exhibit weak, Tikhonov-like Self-Regularization, with a size scale separating signal from noise while arising implicitly from training.Their ESDs are modeled using a Spiked-Covariance variant of MP theory, with a small number of eigenvalues separating from the random bulk.
  • 1 Introduction: Modern large DNNs exhibit Heavy-Tailed Self-Regularization, retaining Hard Rank while lacking a clean size scale that separates signal from noise.Their ESDs deviate from Gaussian MP behavior and appear to belong to Heavy-Tailed random-matrix universality classes.
  • 1 Introduction: The theory organizes increasing implicit Self-Regularization into six phases: Random-like, Bleeding-out, Bulk+Spikes, Bulk-decay, Heavy-Tailed, and Rank-collapse.Successive phases have stronger ESD signatures and decreasing MP Soft Rank; Rank-collapse is the pathological over-regularized “+1” phase.
  • 1 Introduction: Reducing MiniAlexNet’s batch size from 500 to 2 produces all 5+1 phases while increasing generalization accuracy, whereas extreme Weight Norm regularization induces Rank-collapse.The result connects smaller-batch training with stronger implicit Self-Regularization and the Generalization Gap phenomenon.

2 Simple Capacity Metrics and Transitions during Backprop

The section introduces scale-invariant spectral capacity metrics for DNN weight matrices and uses training-time spectral changes to reveal implicit self-regularization.

  • Simple capacity metrics: Matrix Entropy and Stable Rank are scale-invariant capacity-control metrics defined from the spectrum of each layer weight matrix.The associated correlation matrix X = W^T W has eigenvalues equal to squared singular values, and the metrics can be computed from either representation.
  • Simple capacity metrics: Hard Rank counts nonzero singular values, while Stable Rank uses the Frobenius-to-Spectral norm ratio as a robust capacity measure.The Hard Rank generally remains full in well-trained DNNs even as weight matrices shrink in Stable Rank.
  • Simple capacity metrics: Matrix Entropy is maximal for random matrices, whereas lower entropy indicates greater non-random structure or regularity.For well-trained DNNs, entropy and Stable Rank show qualitatively similar behavior, although entropy changes can be subtle.
  • Transitions during Backprop: During MLP3 training, Stable Rank decreased by approximately a factor of two, while FC2 Matrix Entropy fell from roughly 0.92 to just below 0.91.Both metrics transition during backpropagation and track nearly the same changes across FC1 and FC2.
  • Spectral diagnostics: Scree plots provide a coarse view of eigenvalue variability, while singular- and eigenvalue-density histograms reveal finer structural changes relevant to self-regularization.The detailed density plots expose changes concentrated in a few singular values and vectors that coarse metrics and scree plots can miss.
  • Spectral transitions: Final FC2 spectra retained a quarter-circle-like bulk but developed several spikes, including a largest eigenvalue near 7.2 versus an initial upper edge near 3.2.The Frobenius norm changed little, so the Stable Rank decrease reflects an approximately twofold increase in squared Spectral norm.

3 Basic Random Matrix Theory (RMT)

RMT characterizes the empirical spectra of rectangular DNN weight matrices by comparing them with universal random-matrix laws, especially Marchenko–Pastur (MP) and Tracy–Widom predictions. Departures from these laws, including spikes and heavy tails, provide evidence of structured or regularized weight matrices.

  • MP theory: RMT analyzes the ESD of X = W^T W, using matrix dimensions and distributional assumptions to predict spectral properties at finite and infinite sizes.MP theory concerns singular values of rectangular W and equivalently eigenvalues of its correlation matrix X.
  • MP theory: The MP distribution is determined by the aspect ratio Q and element-wise variance σ^2_mp, which set the bulk support [λ−, λ+].For fixed σ^2_mp, changing Q changes the width and skew of the distribution; λ+ and σ^2_mp determine one another under the stated relation.
  • MP theory: For Q = 1, the eigenvalue density is sharply peaked with a bounded tail, while the singular-value density is more conveniently described by the Quarter-Circle Law.The Quarter-Circle case applies to square, non-symmetric random matrices.
  • Finite-size effects: Finite-size fluctuations near the MP edge follow Tracy–Widom statistics, so eigenvalues beyond the predicted bulk require distinguishing ordinary finite-size deviations from genuine outliers.For the considered DNNs with M ≳ 400, the maximum finite-size deviation is reported as Δλ_M ≲ 0.02.
  • Heavy-Tailed extensions: Strong correlations can invalidate Gaussian MP modeling, motivating Heavy-Tailed random-matrix extensions in which the ESD itself exhibits global or local heavy-tailed behavior.The paper uses these extensions to analyze Heavy-Tailed Self-Regularization in DNN weight matrices.
  • Heavy-Tailed extensions: Power-law fitting has important limitations: the CSN estimator works reasonably for 1.5 < α ≲ 3.5 but can show substantial finite-size deviations for 2 < μ < 4.The paper also notes technical challenges in selecting the MP bulk edge and relating λ+ to σ^2_mp.

4 Empirical Results: ESDs for Existing, Pretrained DNNs

Existing pretrained DNNs show a progression from MP-like spectra with separated spikes in LeNet5 to heavy-tailed, strongly correlated spectra in modern architectures. Across diverse pretrained vision and NLP models, these patterns support an RMT-based account of implicit self-regularization.

  • LeNet5: LeNet5’s FC1 spectrum is well described by an MP bulk plus outlying spikes, consistent with a spiked-covariance model.The bulk edge is approximately λ+ ≈3.5, separating the main bulk from smaller external peaks.
  • AlexNet: AlexNet’s FC1 spectrum departs from MP theory through an unfilled peak and substantial eigenvalue mass bleeding beyond the bulk.These deviations indicate that the spectrum is not adequately captured by a standard MP fit.
  • AlexNet: AlexNet’s FC2 and FC3 spectra are strongly heavy-tailed, with FC2 lacking a good MP fit and FC3 yielding a power-law fit of α ≈3.02.Both layers deviate from MP predictions in global bulk and local edge properties.
  • InceptionV3: InceptionV3’s L226 and L302 spectra are bimodal or fat-tailed, with no cleanly identifiable MP bulk edge.L226 is strongly bimodal, while L302’s edge decays into the tail and its bulk poorly fits MP density.
  • Other pretrained DNNs: Across 84 pretrained vision and NLP linear layers, heavy-tailed properties and power-law fits appear nearly universal across architectures and domains.Most ImageNet fits fall in 1.5 < α ≲3.5, and nearly all ESDs appear in the 2 < µ < 4 universality class.
  • Other pretrained DNNs: Rank collapse is limited in the surveyed models: 6 of 24 ImageNet fully connected layers show νmin ∼0, whereas all 82 AllenNLP layers have νmin > 0.The affected ImageNet layers retain heavy-tailed properties, with rank loss ranging from one zero eigenvalue to 15% of eigenvalue mass.
  • Interpretation: Together, these results support using modern RMT to characterize DNN weight matrices and distinguish Tikhonov-like from heavy-tailed self-regularization.Older or smaller models show a size scale separating signal and noise, while modern models exhibit correlations across scales without a clean bulk-spike separation.

5 5+1 Phases of Regularized Training

The paper presents a 5+1-phase taxonomy of increasing implicit Self-Regularization, using RMT and MP-based analysis of DNN weight-matrix spectra. Older or smaller models show Tikhonov-like separation between signal and noise, whereas Heavy-Tailed models exhibit stronger correlations without a single size scale.

  • MP Soft Rank: The MP Soft Rank compares the fitted MP bulk edge λ+ with the largest eigenvalue λmax, reaching 1 for purely random matrices and 0 when no MP bulk is identified.It differs from Stable Rank because it measures how well MP theory fits part of the ESD rather than low-rank approximability.
  • 5+1 Phases of Regularized Training: The 5+1 phases are Random-like, Bleeding-out, Bulk+Spikes, Bulk-decay, Heavy-Tailed, and Rank-collapse, ordered by increasing Self-Regularization.Successive phases correspond to decreasing Stable Rank and MP Soft Rank.
  • Theory of Each Phase: RMT models each weight matrix as random noise plus a signal correction, using vanilla MP theory for weak perturbations and Heavy-Tailed extensions for strong correlations.The framework distinguishes Gaussian-like bulk behavior from non-Gaussian correlated structure.
  • Bulk+Spikes: Bulk+Spikes resembles a Spiked-Covariance model, with a small number of eigenvalues separating from an MP bulk and Tikhonov-like Self-Regularization.This phase is associated with older or smaller networks such as LeNet5.
  • Bulk-decay: Bulk-decay marks the onset of Heavy-Tailed behavior, with deteriorating bulk separation, more mass beyond λ+, and spikes farther from the bulk.The phase is intermediate between a low-rank perturbation of an MP bulk and correlations at all scales.
  • Heavy-Tailed: In the Heavy-Tailed phase, MP Soft Rank is zero or near zero, Stable Rank is intermediate, and Self-Regularization is strongest without a size scale separating signal from noise.The ESD is modeled using non-Gaussian Heavy-Tailed universality classes and may be compared most easily with log-log histograms or power-law fits.

6 Empirical Results: Detailed Analysis on Smaller Models

The MiniAlexNet experiments track spectral and complexity changes during training and under explicit regularization. Training moves the layers from Random-like spectra toward Bulk+Spikes, while explicit regularization produces larger complexity reductions and more separated spikes.

  • Experimental Setup: MiniAlexNet is a scaled-down AlexNet-like network trained on CIFAR10, with convolutional and fully connected layers analyzed using RMT-based metrics and ESDs.The evaluation includes Matrix Entropy, Hard Rank, Stable Rank, MP Soft Rank, eigenvector localization, and ensembles of final runs.
  • Baseline: The baseline uses batch size 16 without Weight Norm regularization, Dropout, or other explicit regularization.Training uses SGD with momentum and evaluates layer spectra during and after training.
  • Transition in Matrix Entropy and Stable Rank: During baseline training, Matrix Entropy and Stable Rank decrease as training and test accuracies increase, then level off together.Stable Rank shrinks more prominently than Matrix Entropy, with FC1 showing the larger Stable Rank decrease and FC2 the larger entropy decrease.
  • Eigenvalue Spectrum: FC1 and FC2 begin with MP-like Random-like spectra and quickly develop Bulk+Spikes structure as eigenvalue mass shifts toward larger values.For FC1, λmax increases from roughly 3.0 to roughly 4.0 during training; for FC2, it increases from roughly 3.0 to slightly less than 4.0.
  • Eigenvector Localization: After training, spike eigenvectors are more localized than bulk eigenvectors, especially in FC1 where the spike is better separated.The localization difference is less pronounced for FC2.
  • Effect of Explicit Regularization: Explicit L2 Weight Norm or Dropout regularization causes greater decreases in complexity metrics and makes spikes more separated from the bulk.A more prominent spike produces smaller bulk MP variance σ2_mp and a smaller λ+.

7 Explaining the Generalization Gap by Exhibiting the Phases

Changing only batch size moves MiniAlexNet through progressively more self-regularized spectral phases. Smaller batches reduce MP Soft Rank and Stable Rank while improving test accuracy over the studied range, providing the paper’s proposed explanation of the generalization gap.

  • Varying Batch Size: The experiment varies only batch size, using MiniAlexNet runs with b ∈ {500, 250, 100, 50, 32, 16, 8, 4, 2}.Batch size is examined because it is not traditionally treated as a regularization parameter and relates to the generalization gap.
  • Stable Rank, MP Soft Rank, and Performance: As batch size decreases, Stable Rank and MP Soft Rank systematically decrease while test accuracy increases, whereas both training and test accuracy decrease for larger batches.The reported metrics track the changing degree of implicit Self-Regularization.
  • ESDs and Training Phases: At b = 250 and larger, FC1 has a pure MP-like Random-like ESD; at b = 100, an outlier region appears with Bleeding-out structure.These changes are observed in final ensemble ESDs across batch sizes.
  • ESDs and Training Phases: At b = 32, eigenvectors become well-separated from the bulk and the ESD resembles Bulk+Spikes.This marks a stronger self-regularization phase than the large-batch Random-like regime.
  • ESDs and Training Phases: As batch size decreases further, spikes grow and spread, producing Bulk-decay; at b = 2, the ESD becomes consistent with Heavy-Tailed behavior.For FC2, similar changes occur, with four of the five main phases exhibited and greater localization among some extreme eigenvectors.
  • Implications for the Generalization Gap: The results attribute the generalization gap to stronger implicit Self-Regularization at smaller batch sizes, with larger batches producing less-well-regularized models.The paper links this mechanism to the observed decrease in training and test accuracy for larger batch sizes.

8 Discussion and Conclusion

The discussion connects the RMT framework to self-regularization, energy landscapes, training dynamics, and practical model-selection questions. It also identifies assumptions and extensions that constrain current applicability.

  • Connections to prior work: The paper situates its approach among research on large-batch learning, energy landscapes, weight matrices, Heavy-Tailed universality, RMT, statistical physics, and signal fitting.
  • Energy landscapes and rugged convexity: RMT observations suggest that well-trained DNNs have a ruggedly convex global energy landscape rather than many degenerate local minima.
  • Self-organization: Unlike Tikhonov regularization, DNN Self-Regularization arises during training without tuning a specific exogenous regularization parameter or size scale.
  • RMT assumptions: RMT is appropriate in the DNN regime where matrix dimensions grow together, with aspect ratio Q = O(1), rather than the Q →∞ limit common in traditional statistics.
  • RMT assumptions: RMT may not apply straightforwardly to severely overtrained models because non-self-averaging can undermine the relationship between theoretical and empirical spectral densities.
  • Practical implications and extensions: The framework may help evaluate model quality, characterize robustness and interpretability, identify when to stop training, and extend analysis to convolutional layers and natural-language data.
Loading 1810.01075v1…