Source-linked AI summary
Eigenvalues of the Hessian in Deep Learning: Singularity and Beyond
Levent Sagun, Leon Bottou, Yann LeCun
TL;DR
The paper examines Hessian eigenvalues before and after training to characterize how their spectrum depends on network architecture and input data. Empirically, the spectrum separates into a near-zero bulk shaped by model size and isolated eigenvalues shaped by data, with the Hessian becoming highly singular.
Problem
The paper addresses limited applicability of theories and methods that assume a non-singular Hessian, given observations that deep-learning Hessians are highly singular.
Method
The authors calculate exact Hessians for networks trained on MNIST or random-pattern inputs, varying hidden-layer size and comparing spectra before and after training.
Results
The spectrum has a model-size-dependent bulk concentrated around zero and isolated data-dependent eigenvalues away from it; increasing network size sharpens the zero concentration while preserving the data-dependent component.
Takeaways & Limitations
The observed degeneracy suggests flat regions may extend across weight space, while top-eigenvalue and near-zero directions can be studied with separate methods.
Takeaways & Limitations
The paper identifies implications and possible low-energy-path analyses but leaves finding such paths for subsequent work.
Abstract
from arXiv · showhide
We look at the eigenvalues of the Hessian of a loss function before and after training. The eigenvalue distribution is seen to be composed of two parts, the bulk which is concentrated around zero, and the edges which are scattered away from zero. We present empirical evidence for the bulk indicating how over-parametrized the system is, and for the edges that depend on the input data.
1 INTRODUCTION
The paper examines how Hessian eigenvalues reflect the loss landscape and how data and architecture shape the spectrum. It finds a data-dependent discrete component alongside an architecture-dependent bulk concentrated near zero, making the Hessian highly singular.
- 1 INTRODUCTION: Second-order loss information can reveal the optimization landscape and guide modifications intended to improve training.Accessing this information is a major challenge in deep learning.
- 1 INTRODUCTION: The top discrete Hessian eigenvalues depend on the data, while the bulk depends on the architecture.As networks grow, the data-dependent discrete component remains similar while concentration around zero sharpens.
- 1 INTRODUCTION: The Hessian is very singular, so theory and methods assuming a non-singular Hessian require appropriate modification.The paper connects this issue to non-degeneracy assumptions in convergence theory for non-convex systems.
2 THE CASE OF THE FULLY-CONNECTED NETWORK
The experiments compute exact Hessian spectra for fully connected networks while varying network size and input data. Training produces highly singular spectra, and changing the inputs changes both initial and final eigenvalue profiles.
- 2.1 MNIST WITH INCREASING SIZES OF HIDDEN LAYERS: The post-training Hessian is extremely singular, and increasing hidden-layer width intensifies this singularity.The experiment uses 1000 randomly selected 28 ×28 MNIST examples and gradient descent until convergence.
- 2.1 MNIST WITH INCREASING SIZES OF HIDDEN LAYERS: Figure 1 compares the full Hessian after convergence with eigenvalue profiles for increasingly larger networks.For k hidden networks, the number of eigenvalues is (784 + 1) ∗k + (k + 1) ∗ k + (k + 1) ∗10.
- 2.2 VARYING THE DATA: Keeping the architecture fixed while replacing MNIST inputs with random patterns tests whether the eigenvalue distribution depends on the data.The exact Hessian is computed both before training and after the gradient norm becomes close to zero.
- 2.2 VARYING THE DATA: Figure 2 shows that random inputs and MNIST produce very different initial eigenvalue profiles and different final profiles relative to Figure 1.The first histograms use MNIST, while the last two use random inputs.
3 A SIMPLER CASE
In a two-dimensional Gaussian-blob setting, enlarging the network sharpens the eigenvalue concentration around zero while preserving data-dependent isolated eigenvalues; greater data complexity substantially increases the top eigenvalues.
- Data and setup: Two Gaussian blobs centered at (1, 1) and (−1, −1) provide the simple two-dimensional dataset.
- Network-size experiment: The tested systems increase from 18 to 434 parameters, with an additional network trained using mean square loss.
- Network-size experiment: Increasing network size adds eigenvalues around zero while retaining two isolated eigenvalues away from the bulk.
- Loss-function comparison: Mean square loss preserves the same discrete, data-dependent component and the component concentrated at zero.
- Training dynamics: The training procedure concentrates eigenvalues at zero as optimization progresses.
- Data complexity: Increasing Gaussian-blob standard deviation causes the top two eigenvalues to grow significantly beyond initialization fluctuations.
4 CONCLUSION
The Hessian spectrum separates into a model-size-dependent bulk concentrated near zero and isolated data-dependent eigenvalues. This degeneracy suggests broader flat regions and motivates treating top-eigenvalue directions separately from low-eigenvalue directions.
- 4 CONCLUSION: The Hessian is degenerate, requiring revisions to theoretical work whose premises assume non-degenerate systems.The paper notes that existing theory and methodology based on non-singular Hessians may need modification.
- 4 CONCLUSION: Training stops at a point with a small but nonzero gradient, while small negative eigenvalues can remain.Thus, the endpoint is not technically a critical point and the local curvature need not be fully nonnegative.
- 4 CONCLUSION: The spectrum has a bulk concentrated around zero that depends on model size and isolated eigenvalues away from the bulk that depend on data.The two phases are presented as distinct components of the Hessian spectrum.
- 4 CONCLUSION: The observed degeneracy may indicate flat regions extending across weight space beyond the classical notion of wide basins.The paper frames this as a possible global-scale implication rather than a definitive conclusion.
- 4 CONCLUSION: Separate methods may target top-eigenvalue directions, while low-eigenvalue directions may support searches for low-energy paths in weight space.Initial interpolation experiments report completely flat cost profiles even as trained points diverge, though further path exploration is deferred.