PaperlayerView exact revision on arXiv

Source-linked AI summary

Traditional and Heavy-Tailed Self Regularization in Neural Network Models

Charles H. Martin, Michael W. Mahoney

arXiv:1901.08276v1cs.LGstat.ML

TL;DR

The paper addresses how to characterize and control implicit regularization in DNNs, where conventional theories and definitions of regularization are inadequate. It applies RMT and Heavy-Tailed extensions to weight-matrix spectra, identifying 5+1 phases of self-regularization across models and training settings. The results distinguish Tikhonov-like behavior in smaller or older models from Heavy-Tailed behavior in modern models, while batch-size changes can produce all phases in a small model.

  • Problem

    Existing theories do not adequately explain DNN properties, while regularization is difficult to define and compare because many training knobs affect learning.

  • Method

    The paper applies RMT, including MP-based and Heavy-Tailed universality-class extensions, to empirical spectral densities of DNN weight matrices.

  • Results

    The analysis identifies 5+1 phases of increasing implicit Self-Regularization, with Tikhonov-like behavior in older or smaller models and Heavy-Tailed behavior in modern models.

  • Takeaways & Limitations

    Changing training-process knobs, particularly reducing MiniAlexNet batch size from 500 to 2, can make a small model exhibit all 5+1 phases.

  • Takeaways & Limitations

    MP-based RMT is not generally applicable to strongly correlated matrices, and applicability to convolutional layers remains more complex and incompletely explored.

Abstract

from arXiv · show

Random Matrix Theory (RMT) is applied to analyze the weight matrices of Deep Neural Networks (DNNs), including both production quality, pre-trained models such as AlexNet and Inception, and smaller models trained from scratch, such as LeNet5 and a miniature-AlexNet. Empirical and theoretical results clearly indicate that the empirical spectral density (ESD) of DNN layer matrices displays signatures of traditionally-regularized statistical models, even in the absence of exogenously specifying traditional forms of regularization, such as Dropout or Weight Norm constraints. Building on recent results in RMT, most notably its extension to Universality classes of Heavy-Tailed matrices, we develop a theory to identify \emph{5+1 Phases of Training}, corresponding to increasing amounts of \emph{Implicit Self-Regularization}. For smaller and/or older DNNs, this Implicit Self-Regularization is like traditional Tikhonov regularization, in that there is a `size scale' separating signal from noise. For state-of-the-art DNNs, however, we identify a novel form of \emph{Heavy-Tailed Self-Regularization}, similar to the self-organization seen in the statistical physics of disordered systems. This implicit Self-Regularization can depend strongly on the many knobs of the training process. By exploiting the generalization gap phenomena, we demonstrate that we can cause a small model to exhibit all 5+1 phases of training simply by changing the batch size.

1 Introduction

The paper develops an RMT-based account of implicit self-regularization in DNNs, motivated by gaps in existing theories and the difficulty of defining and controlling regularization. It identifies 5+1 training phases and evaluates them across models and training settings.

  • DNN generalization remains difficult for traditional theories because local minima were found not to explain their observed behavior.
  • Regularization in DNNs is difficult to define and compare because many training knobs affect accuracy beyond the model parameters.
  • The paper asks how to develop a principled theory of DNN regularization and how to control training knobs and the global Energy Landscape.
  • RMT analysis finds Tikhonov-like implicit Self-Regularization in older or smaller models and Heavy-Tailed Self-Regularization in modern state-of-the-art models.
  • The theory organizes increasing Self-Regularization into 5+1 visually distinct training phases based on ESD signatures and decreasing MP Soft Rank.
  • Decreasing MiniAlexNet batch size from 500 to 2 makes its fully-connected-layer ESDs vary continuously from Random-like to Heavy-Tailed phases.

2 Basic Random Matrix Theory (RMT)

This section introduces MP-based RMT for analyzing neural-network weight spectra and extends it to Heavy-Tailed universality classes for strongly correlated matrices. These tools motivate using ESD deviations and Heavy-Tailed behavior as indicators of non-random structure and self-regularization.

  • MP theory analyzes singular-value or eigenvalue densities of random rectangular matrices and predicts distribution shapes, bounds, finite-size effects, and convergence rates.
  • RMT comparisons require the matrix dimensions and an assumed element distribution from a specified Universality class, including Gaussian and Heavy-Tailed classes.
  • Table 1 compares basic MP theory, the Spiked-Covariance model, and Heavy-Tailed extensions, including known, empirical, and conjectured relationships.
  • At finite size, MP predicts a sharp upper spectral edge, while violations of MP or Tracy-Widom edge behavior provide evidence of more regular non-random structure.
  • MP-based RMT is not generally applicable to strongly correlated matrices, motivating Heavy-Tailed extensions for well-trained production-quality DNNs.
  • The paper uses Heavy-Tailed RMT extensions to build an operational theory of regularization and analyze Self-Regularization and Heavy-Tailed Self-Regularization.
  • A Heavy-Tailed ESD may indicate a scale-free, strongly correlated DNN, although it need not imply Heavy-Tailed matrix elements.

3 Empirical Results: ESDs for Existing, Pretrained DNNs

Existing pretrained DNNs show a sharp contrast in empirical spectral densities: older or smaller models fit Marchenko–Pastur structure with low-rank outliers, while larger modern models exhibit Heavy-Tailed behavior and no clean bulk–spike separation.

  • LeNet5: LeNet5’s FC1 ESD fits Marchenko–Pastur theory well in the bulk, with a small bleeding region and several low-rank outlier spikes.The bulk extends to λ+ ≈3.5, while spikes range from approximately 5 to λmax ≲25.
  • AlexNet: AlexNet’s FC2 ESD has no good Marchenko–Pastur fit, an indeterminate bulk edge, and nearly everywhere-concave density.Its power-law fit has α ≈2.25, indicating a Heavy-Tailed regime with µ ≲3.
  • Other pretrained DNNs: Across larger state-of-the-art models, power-law or truncated-power-law fits generally outperform exponential-family alternatives, with fitted α values usually between 1.5 and 3.5.The passage reports this pattern for models including VGG16, VGG19, ResNet50, and InceptionV3, with exceptions in certain InceptionV3 layers.
  • Heavy-Tailed Self-Regularization: These observations suggest that modern state-of-the-art DNNs exhibit Heavy-Tailed Self-Regularization associated with strongly correlated weight matrices.The paper relates these correlations to non-Gaussian Heavy-Tailed random-matrix universality classes.
  • Heavy-Tailed Self-Regularization: Unlike MP-like spectra with a scale cutoff λ+, strongly Heavy-Tailed spectra show correlations across every size scale and lack a clean separation between bulk and spikes.This distinction motivates treating modern DNN spectra as a form of self-organization rather than conventional bulk-plus-outlier structure.

4 5+1 Phases of Regularized Training

The paper defines six ordered training phases using RMT-based spectral features, from Random-like matrices to Rank-collapse. Increasing self-regularization corresponds to decreasing Stable Rank and MP Soft Rank, with later phases modeled by Heavy-Tailed rather than vanilla MP theory.

  • Operational Measures: MP Soft Rank is the bulk edge λ+ divided by the largest eigenvalue λmax, ranging from 1 for purely random matrices to 0 without a good MP fit.Outlying spikes make Rmp less than 1.
  • Empirical Scope: Existing LeNet5 and AlexNet/InceptionV3 models exhibit Bulk+Spikes and Heavy-Tailed behavior, respectively, while MiniAlexNet exhibits all 5+1 phases.The taxonomy compares architectures and regularization levels in trained or pre-trained networks.
  • Visual Taxonomy: The six phases are Random-like, Bleeding-out, Bulk+Spikes, Bulk-decay, Heavy-Tailed, and Rank-collapse.They form an ordered visual taxonomy of increasing implicit self-regularization.
  • RMT Diagnostics: RMT distinguishes phases through global ESD shape, local bulk-edge fluctuations, and the statistics of outlying spikes.These features include MP or Heavy-Tailed fits and fluctuations around λ+ or λmax.
  • Early Phases: Random-like, Bleeding-out, and Bulk+Spikes phases use vanilla MP theory because signal remains a relatively small perturbation of random noise.Bulk+Spikes resembles a Spiked-Covariance model and Tikhonov regularization.
  • Later Phases: Bulk-decay and Heavy-Tailed phases arise when strongly correlated signal increasingly dominates random noise, making MP theory inappropriate.The signal is modeled with Heavy-Tailed random matrices, while Rank-collapse is the theory’s degenerate final phase.

5 Empirical Results: Detailed Analysis on Smaller Models

MiniAlexNet training progressively changes its weight-matrix spectra and complexity metrics, moving FC1 from a Random-like ESD toward Bulk+Spikes. Explicit regularization intensifies these spectral changes.

  • Experimental Setup: MiniAlexNet is a scaled-down AlexNet trained on CIFAR10, with two convolutional layers and three fully connected layers analyzed during training.The study saves weight matrices after every epoch and analyzes FC1 and FC2.
  • Spectral Evolution: FC1 begins with an MP-like ESD and shifts within a few epochs toward the Bulk+Spikes phase.The initial FC1 matrix has Q ≈10.67, consistent with a Random-like phase.
  • Spectral Evolution: λmax increases from roughly 3.0 to roughly 4.0 during training, indicating further self-regularization within Bulk+Spikes.After spikes appear, visual changes are subtle but continue in the ESD.
  • Complexity Metrics: Matrix Entropy and Stable Rank gradually decrease as training and test accuracies increase, then level off together.These changes are visible in the evolving ESD.
  • Explicit Regularization: Adding L2 weight regularization or Dropout produces larger complexity decreases by pulling spike eigenvalues to much larger ESD values.The observed changes are consistent with the expected effect of explicit regularization.

6 Explaining the Generalization Gap by Exhibiting the Phases

The paper uses batch size as a single training-control variable to study the generalization gap and spectral self-regularization. Smaller batches systematically produce stronger self-regularization and move MiniAlexNet through the main spectral phases.

  • Motivation: The generalization gap is the reported tendency of DNNs to generalize less well when trained with large mini-batches around 10^3–10^4.Large batches are also more efficient for training large DNNs on modern GPUs, creating a practical trade-off.
  • Experimental Design: MiniAlexNet is trained with batch sizes from 500 to 2 to test how one optimization knob affects spectral phases and accuracy.The batch-size set is {500, 250, 100, 50, 32, 16, 8, 4, 2}.
  • Rank and Accuracy: Stable Rank and MP Soft Rank both systematically decrease as batch size decreases, while test accuracy increases over the corresponding range.The two rank measures track each other across FC1 and FC2.
  • ESD Phase Transitions: At batch size 250 and larger, FC1 is Random-like; at 100 it shows Bleeding-out; and at 32 it shows Bulk+Spikes.These transitions are identified from the changing ESD and its outlier region.
  • ESD Phase Transitions: Further batch-size reductions produce Bulk-decay and Heavy-Tailed ESDs, with b = 2 showing curvature consistent with Heavy-Tailed behavior.The experiment exhibits all five main phases by changing only batch size.

7 Discussion and Conclusion

The theory is intended to extend beyond the layer types analyzed here, but applying it to convolutional layers appears more complex than the presented framework.

  • Applying the RMT-based theory to convolutional layers is identified as an open practical question.Initial results suggest applicability, but the situation is more complex than the framework described here.
  • The discussion presents these extensions as promising directions for future investigation.
  • The current account therefore does not establish a complete treatment of other layer types.
Loading 1901.08276v1…