PaperlayerView exact revision on arXiv

Source-linked AI summary

Empirical Analysis of the Hessian of Over-Parametrized Neural Networks

Levent Sagun, Utku Evci, V. Ugur Guney, Yann Dauphin, Leon Bottou

arXiv:1706.04454v3cs.LG

TL;DR

The paper asks how Hessian geometry characterizes over-parametrized neural-network loss surfaces and their optimization-relevant structure. Through empirical spectrum analysis and a Hessian decomposition, it finds a near-zero bulk with data- and algorithm-dependent outliers, and argues that apparently different batch solutions can be connected through flat regions. The conclusions are limited by its vague definitions of data complexity and over-parametrization.

  • Problem

    In high-dimensional deep learning, the relationship between empirical and population losses, Hessian geometry, and optimization behavior can differ from fixed-dimensional theory.

  • Method

    The paper performs a phenomenological study of Hessian spectra, analyzes a covariance-based Hessian decomposition, and examines connectedness between large- and small-batch solutions.

  • Results

    The Hessian spectrum has a near-zero bulk and relatively few outliers; data complexity, network size, and batch method affect these components differently, while large- and small-batch solutions can lie in the same basin.

  • Takeaways & Limitations

    Flat Hessian regions and over-parametrization challenge isolated-basin and wide-versus-narrow-basin interpretations of neural-network solutions.

  • Takeaways & Limitations

    The paper uses vague notions of data complexity and over-parametrization and notes that realistic definitions should account for architecture.

Abstract

from arXiv · show

We study the properties of common loss surfaces through their Hessian matrix. In particular, in the context of deep learning, we empirically show that the spectrum of the Hessian is composed of two parts: (1) the bulk centered near zero, (2) and outliers away from the bulk. We present numerical evidence and mathematical justifications to the following conjectures laid out by Sagun et al. (2016): Fixing data, increasing the number of parameters merely scales the bulk of the spectrum; fixing the dimension and changing the data (for instance adding more clusters or making the data less separable) only affects the outliers. We believe that our observations have striking implications for non-convex optimization in high dimensions. First, the flatness of such landscapes (which can be measured by the singularity of the Hessian) implies that classical notions of basins of attraction may be quite misleading. And that the discussion of wide/narrow basins may be in need of a new perspective around over-parametrization and redundancy that are able to create large connected components at the bottom of the landscape. Second, the dependence of small number of large eigenvalues to the data distribution can be linked to the spectrum of the covariance matrix of gradients of model outputs. With this in mind, we may reevaluate the connections within the data-architecture-algorithm framework of a model, hoping that it would shed light into the geometry of high-dimensional and non-convex spaces in modern applications. In particular, we present a case that links the two observations: small and large batch gradient descent appear to converge to different basins of attraction but we show that they are in fact connected through their flat region and so belong to the same basin.

1 INTRODUCTION

The paper studies supervised-learning loss geometry through Hessian spectra, motivated by high-dimensional settings where empirical and population losses may differ. It reports a near-zero spectral bulk, data- and algorithm-dependent outliers, and connected solution regions that challenge classical basin interpretations.

  • Framework: The framework models supervised learning with N input-label pairs, M parameters, a predictor f(w, x), and a non-negative loss minimized over w.The loss measures prediction error relative to true labels, and the objective is to find a minimizing parameter w*.
  • Motivation: When M is comparable to or larger than N, empirical and population losses may behave drastically differently despite pointwise convergence as N grows for fixed M.The law of large numbers gives convergence at fixed parameter dimension, but modern deep-learning regimes often violate that setting.
  • Optimization: Hessian information is relevant to optimization because eigenvalue signs characterize critical-point type and eigenvectors identify local directions of change, but computing the Hessian is expensive.Gradient methods, stochastic approximations, and second-order methods address different computational costs in optimization.
  • Hessian geometry: At the bottom of the landscape, most Hessian eigenvalues lie near zero while a small number remain relatively large.The paper treats this spectral structure as empirical evidence of substantial flatness in over-parametrized neural-network landscapes.
  • Hessian geometry: The Hessian decomposition links its dominant covariance term to gradients of model outputs, with over-parametrization producing degeneracy when parameters outnumber samples.The paper argues that the second decomposition term becomes negligible as training progresses.
  • Data, architecture, and algorithm: More complex data produce more outliers, network growth does not change the density of large eigenvalues, and large-batch methods produce equally numerous but larger outliers.Small negative eigenvalues may remain after apparent training convergence, complicating interpretations based only on positive curvature.
  • Basins of solutions: The paper argues that large- and small-batch solutions can belong to the same basin through flat regions, rather than occupying isolated wide and narrow basins.This perspective connects Hessian flatness, over-parametrization, redundancy, and solution-space connectedness.
  • Scope: The paper leaves data complexity and over-parametrization only vaguely defined and says realistic definitions should account for architecture.A detailed treatment is deferred to future work.

2 SOME PROPERTIES OF THE HESSIAN

The Hessian is analyzed through its exact spectrum and a decomposition into model- and loss-dependent terms. Near a local minimum, the Hessian can be approximated by a sum of scaled outer products of output gradients, whose structure yields many trivial eigenvalues in over-parameterized settings.

  • Spectrum: The full Hessian spectrum is computed exactly at random initialization and after gradient-descent training in a two-hidden-layer network with 5K parameters.The computation uses Hessian-vector products up to machine precision.
  • Decomposition: The Hessian is decomposed into two matrices associated with the model output function and the convex loss function.Examples include mean-square regression and softmax-based classification losses.
  • Decomposition: Because the loss is convex, its second derivative is non-negative, enabling a reformulation using its square root.
  • Approximation: Near a local minimum, if loss derivatives and model-output Hessians are uncorrelated, the second decomposition term can be ignored and the Hessian approximated by the first term.
  • Approximation: The approximate Hessian is a sum of scaled rank-one outer products of output gradients, implying at least M − N trivial eigenvalues.It can be represented as an M × N matrix multiplied by its transpose.

3 IMPLICATIONS OF FLATNESS FOR THE GEOMETRY OF THE ENERGY

The experiments examine how data, model size, and optimization algorithm shape Hessian spectra, finding a near-zero bulk with data- and algorithm-dependent outliers and small trailing negative eigenvalues.

  • 3.1 THE RELATION BETWEEN DATA AND EIGENVALUES: The number of large eigenvalues above the spectral gap matches the number of dataset classes.This relationship is illustrated by the ordered eigenvalue plot in Figure 2.
  • 3.2 THE RELATION BETWEEN NUMBER OF PARAMETERS AND EIGENVALUES: Increasing network size preserves the shape of the right edge of the Hessian spectrum while scaling the small-scale eigenvalues.The experiments vary hidden-layer size while fixing the MNIST training set, architecture, and algorithm.
  • 3.4 TRAILING NEGATIVE EIGENVALUES AFTER TRAINING: Negative eigenvalues remain after training, but their magnitudes are orders of magnitude smaller than the large positive eigenvalues.Their presence indicates that training has not yet reached a local minimum, although the practical relevance of this fine-grained scale may be limited.
  • 3.4 TRAILING NEGATIVE EIGENVALUES AFTER TRAINING: When network size increases, the ratio of negative eigenvalues remains unchanged and their ranked shape appears to converge across experiments.

4 DISCUSSION ON BASINS OF SOLUTIONS AND GENERALIZATION

The paper argues that large- and small-batch methods can reach solutions with different generalization properties without occupying different basins, because flat regions connect them. It uses carefully scaled comparisons and line interpolations to challenge basin-based interpretations.

  • 4.1 ARE THEY REALLY DIFFERENT BASINS? CASE OF CIFAR10 WITH LB VS SB: Comparisons should use iteration-based time scales because different batch sizes take different numbers of optimization steps per epoch.The experiments therefore compare procedures using similar numbers of steps and the same learning rate.
  • 4.1 ARE THEY REALLY DIFFERENT BASINS? CASE OF CIFAR10 WITH LB VS SB: Internal network symmetries can create apparent barriers between weight-space solutions that represent identical networks.The paper highlights node-permutation or flip symmetry as a reason to interpret straight-line comparisons cautiously.
  • 4.1 ARE THEY REALLY DIFFERENT BASINS? CASE OF CIFAR10 WITH LB VS SB: A large-batch-to-small-batch CIFAR10 experiment produced an approximately 1% accuracy increase during the small-batch phase.The experiment continued training the large-batch endpoint with a smaller batch size while keeping the learning rate fixed.
  • 4 DISCUSSION ON BASINS OF SOLUTIONS AND GENERALIZATION: Large- and small-batch solutions can have different generalization properties while remaining in the same basin through a flat connecting region.The straight-line interpolation between the solutions contains no loss barrier, despite their differing properties.
  • 4.2 FURTHER DISCUSSIONS: Reducing batch size increases stochastic-gradient noise, whose fluctuations are argued to align with directions associated with large Hessian eigenvalues.This provides an alternative interpretation of the loss increase observed when small-batch training begins.
  • 4.2 FURTHER DISCUSSIONS: The paper proposes studying connected interiors of level sets rather than relying exclusively on classical basin-of-attraction or wide-versus-narrow-basin descriptions.It links this perspective to the possibility that over-parameterization creates larger connected regions at the bottom of the landscape.

5 CONCLUSION

The conclusion emphasizes Hessian singularity and flat directions as central features of neural-network loss landscapes. It argues that large- and small-batch solutions can lie in the same basin, with their generalization gap not explained by finding different wider basins.

  • 5 CONCLUSION: Hessian singularity and clusters of zero eigenvalues should be included in theoretical accounts of practical neural-network loss landscapes.The paper uses the generalized Gauss–Newton decomposition to argue that these zero eigenvalues are expected.
  • 5 CONCLUSION: Large- and small-batch methods can reach the same basin even when their solutions differ in generalization.The paper reports paths between the solutions that remain in the same level set and do not cross a barrier.
  • 5 CONCLUSION: The observed generalization gap is not attributed to small-batch training finding a different, better, wider basin.The conclusion presents this as a shift away from conventional energy-landscape interpretations.

A OUTLIER EIGENVALUES

The outlier-eigenvalue experiments vary network and data settings to examine how the Hessian spectrum changes. The accompanying table organizes outlier counting around cluster structure and heuristic spectral thresholds.

  • A OUTLIER EIGENVALUES: The experiments use feed-forward networks with 100-dimensional inputs, two 30-unit hidden layers, and k-class softmax outputs.They vary k over {2, 5, 10, 20, 50}, compare GD and SGD, and use tanh or ReLU nonlinearities.
  • A OUTLIER EIGENVALUES: The data consist of k Gaussian clusters sampled in input space and globally normalized before training.The number of clusters is varied across the experimental settings.
  • A OUTLIER EIGENVALUES: Table 1 counts spectral outliers in relation to the number of data blobs using covariance scale, eigenvalue gaps or ratios, and a heuristic cutoff.Its element dictionary defines nc as the number of clusters and heur/cutoff as the thresholding rule and value.

B THE SPECTRUM OF THE GENERALIZED GAUSS-NEWTON MATRIX

The generalized Gauss–Newton spectrum is analyzed through a gradient-covariance representation, yielding a bulk–outlier decomposition shaped by covariance structure and model/data dimensions.

  • G can be written as a transformed sample covariance matrix of model-output gradients, with Σ defined as its expected covariance.For linear models, the scaled gradient is expressed as g = Tx, leading to G = (1/N)TXX^T T^T and Σ = E(G).
  • When gradients are uncorrelated, the nontrivial eigenvalues follow the Marčenko–Pastur law, whose width depends on α = M/N.If M > N, the remaining M − N eigenvalues are zero.
  • M − K eigenvalues are exactly zero when M > N, while the remaining order-K eigenvalues form the bulk.Here K = min{M, N}.
  • Large covariance eigenvalues generate right outliers, whereas covariance eigenvalues near zero generate left outliers below the bulk.The corresponding outlier eigenvectors are close to those of Σ.
  • For logistic regression, the Hessian reduces to the gradient-covariance term because the model-output Hessian is zero.With a single sigmoid neuron and linear score f(w, x) = ⟨w, x⟩, the decomposition applies directly.
  • Two Gaussian blobs produce one large outlier eigenvalue and a bulk close to zero in the logistic-regression spectrum.Figure 9 reports outlier eigenvalues 454.4, 819.5, and 92.7 for α = 1, 2, and 0.02, respectively.
Loading 1706.04454v3…