Source-linked AI summary
Energy-entropy competition and the effectiveness of stochastic gradient descent in machine learning
Yao Zhang, Andrew M. Saxe, Madhu S. Advani, Alpha A. Lee
TL;DR
SGD often generalizes well despite being unlikely to find the global minimum, leaving its effectiveness unexplained. The paper connects parameter inference to free-energy minimization and finds that undersampling can favor wide, shallow minima, while SGD’s correlated noise biases it toward wider ones.
Problem
SGD can generalize well despite being highly unlikely to find the global minimum, and the reason its solutions generalize remains mysterious.
Source: INTRODUCTION · p. 1Method
The paper analyzes SGD noise and uses a Bayesian Laplace approximation to connect parameter inference with free-energy minimization, treating undersampling as temperature.
Source: INTRODUCTION · p. 2, STOCHASTIC GRADIENT DESCENT AND… · pp. 2–3Results
In the undersampled, high-temperature regime, Bayes-optimal parameters balance training error against basin width, and correlated, anisotropic SGD noise favors wide minima.
Source: STOCHASTIC GRADIENT DESCENT AND… · p. 3, CONCLUSION · p. 7Takeaways & Limitations
The energy–entropy account provides a physical justification for how SGD can achieve low out-of-sample error without finding the global loss minimum.
Source: CONCLUSION · p. 7The harmonic log-determinant entropy approximation is limited by singular Hessians, and the framework leaves generalization at saddle points open.
Source: CONCLUSION · pp. 7–8
Abstract
from arXiv · showhide
Finding parameters that minimise a loss function is at the core of many machine learning methods. The Stochastic Gradient Descent algorithm is widely used and delivers state of the art results for many problems. Nonetheless, Stochastic Gradient Descent typically cannot find the global minimum, thus its empirical effectiveness is hitherto mysterious. We derive a correspondence between parameter inference and free energy minimisation in statistical physics. The degree of undersampling plays the role of temperature. Analogous to the energy-entropy competition in statistical physics, wide but shallow minima can be optimal if the system is undersampled, as is typical in many applications. Moreover, we show that the stochasticity in the algorithm has a non-trivial correlation structure which systematically biases it towards wide minima. We illustrate our argument with two prototypical models: image classification using deep learning, and a linear neural network where we can analytically reveal the relationship between entropy and out-of-sample error.
INTRODUCTION
The introduction frames SGD’s strong generalization despite its likely failure to find the global minimum as an unresolved puzzle. It motivates an account based on minimum width and undersampling, linking correlated SGD noise and an energy–entropy trade-off to the paper’s proposed explanation.
INTRODUCTION
- SGD is widely effective at producing accurate predictions outside training data, yet is highly unlikely to find the global minimum, leaving its generalisability puzzling.The introduction notes that routinely used models can have O(10^7) parameters and non-convex losses, making the global minimum computationally intractable to find.Source: INTRODUCTION · p. 1
- Topology-based accounts attribute high-loss critical points to saddle points and, in some architectures, exclude nonglobal local minima [15].These accounts also motivate the view that many local minima are “good enough”.Source: INTRODUCTION · p. 1
- Loss minimization alone poorly captures SGD’s effectiveness because generalization, not training fit, is the ultimate metric, and deeper minima can generalize worse than early-stopped solutions.With many more parameters than samples, training data can be fit completely [16].Source: INTRODUCTION · p. 1
- Prior work links flatter minima with lower generalization error and reports that smaller batches or added Gaussian noise yield wider minima and better generalization [22] [23] [24].Phenomenological width measures are not reparameterization-invariant, since rescaling can make wide minima appear arbitrarily sharp [27].Source: INTRODUCTION · pp. 1–2
- The paper proposes that undersampling makes wide but shallow minima optimal and that correlated SGD noise systematically biases the search toward wide minima.It develops a free-energy correspondence and applies the insight to image classification and a linear neural network.Source: INTRODUCTION · p. 2
STOCHASTIC GRADIENT DESCENT AND ENTROPY-ENERGY COMPETITION
The paper links SGD’s correlated, anisotropic noise and Bayesian parameter inference to an energy–entropy competition: under undersampling, wide minima can be preferred even when their training error is higher. The strength of that preference depends on undersampling and approximation noise.
STOCHASTIC GRADIENT DESCENT AND ENTROPY-ENERGY COMPETITION
- SGD noise variance increases as O(1/b), and its anisotropy gives greater noise to high-curvature directions, helping the algorithm escape narrow basins.The paper contrasts this with methods that take smaller steps in stiffer directions; SGD can instead become trapped in wide basins.Source: STOCHASTIC GRADIENT DESCENT AND… · p. 2
- The statistical-physics treatment rewrites the posterior in intensive quantities that do not scale with the sample count P or parameter count N.The mean error is interpreted as an internal energy in this formulation.Source: STOCHASTIC GRADIENT DESCENT AND… · p. 2
- The analysis uses a high-dimensional regime with P→∞ and fixed P/N=α, where the paper says the Laplace approximation is justified.For the regression model, it also assumes Gaussian approximation error and independently sampled data points.Source: STOCHASTIC GRADIENT DESCENT AND… · pp. 2–3
- The entropy s(θq) is related to minimum width through the Hessian eigenvalues, but its definition assumes nonzero eigenvalues and may not capture full basin volume.The authors say a full basin-volume analysis is required; for singular Hessians or saddle points, their experiments suggest qualitatively similar arguments after truncating to large positive eigenvalues.Source: STOCHASTIC GRADIENT DESCENT AND… · p. 3
- The free-energy mapping treats undersampling 1/α as temperature, mean error as internal energy, and minimum width as entropy.As P grows with α fixed, the dominant minimum is the one with lowest effective free energy, not necessarily the one with lowest error.Source: STOCHASTIC GRADIENT DESCENT AND… · p. 3
- In the high-dimensional limit, wide minima can be preferred despite higher in-sample error, and SGD noise biases its search toward them.How strongly width is preferred depends on the degree of undersampling and approximation noise.Source: STOCHASTIC GRADIENT DESCENT AND… · pp. 2–3
NUMERICAL EXPERIMENTS
The experiments test whether SGD favors wider minima and whether higher-entropy solutions generalize better, using a simplified CIFAR-10 image-classification task. SGD found higher-entropy minima on average than same-noise Langevin dynamics, while entropy was negatively correlated with test error.
NUMERICAL EXPERIMENTS
- The CIFAR-10 experiment used airplane-versus-automobile classification with 10 × 10 downsampled images because Hessian computation at industrial scale is challenging.Source: NUMERICAL EXPERIMENTS · pp. 3–4
- The three-hidden-layer network had 10 units per layer and 1,320 parameters; 500 of 12,000 images trained it to zero in-sample error.Source: NUMERICAL EXPERIMENTS · p. 4
- To test the effect of noise correlations, the authors compared SGD with Langevin dynamics matched in noise magnitude but lacking SGD's correlation structure.Source: NUMERICAL EXPERIMENTS · p. 4
- SGD found higher-entropy minima on average than same-noise Langevin dynamics, supporting the predicted bias toward wider minima.Figure 1 summarizes 50 independent runs.Source: Figure 1 · p. 4, NUMERICAL EXPERIMENTS · p. 5
- Among solutions with zero in-sample error, entropy showed a discernible negative correlation with test error on 2,000 held-out images.Figure 2 shows results from 50 independent SGD runs.Source: NUMERICAL EXPERIMENTS · p. 5, Figure 2 · p. 5
ENTROPY IN DEEP LINEAR NETWORKS
In underdetermined deep linear networks, higher entropy is associated with lower test error, with lower weight norms providing an analytic explanation for the relationship. The strength of the entropy and Hessian-trace correlations depends on how the weights are initialized.
ENTROPY IN DEEP LINEAR NETWORKS
- On the global-minimum manifold, extreme scalar weight choices can make entropy diverge even though every point implements the same function and generalizes identically.This illustrates that Hessian-based entropy can depend on parameterization within this simple network.Source: ENTROPY IN DEEP LINEAR NETWORKS · p. 6
- In the linear chain, entropy equals the negative log of the Hessian trace, which closely tracks entropy and correlates about equally well with generalization performance.The equivalence is stated for the linear chain considered in the analysis.Source: ENTROPY IN DEEP LINEAR NETWORKS · p. 6, Figure 5 · p. 7
- Balanced weight initialization strengthens the entropy-generalization correlation; with imbalanced initial weights, entropy and Hessian-trace correlations decrease while the weight-norm correlation remains unchanged.Under batch gradient descent, the difference between layer weight norms is invariant; the paper notes this balance approximately carries to SGD at small learning rates.Source: ENTROPY IN DEEP LINEAR NETWORKS · p. 6, Figure 4 · p. 6
- In the underdetermined regime, higher-entropy solutions correlate with lower test error, and Figure 3 reports this relationship for deep linear networks.When P < Ni, training-error minimizers can generalize differently despite all attaining zero training error.Source: ENTROPY IN DEEP LINEAR NETWORKS · pp. 5, 7, Figure 3 · p. 5
- The linear model’s entropy-test-error relationship is explained by higher-entropy solutions tending to have lower weight norms, which correlate strongly with test error.The predicted linear relationship between generalization performance and weight norm fits the simulations.Source: Figure 3 · p. 5, ENTROPY IN DEEP LINEAR NETWORKS · p. 7
CONCLUSION
The paper interprets parameter inference through free-energy minimization, with undersampling acting like temperature, and reports that SGD’s correlated noise favors wide minima. It also identifies limits of Hessian-based entropy and leaves the generalization of saddle points as an open question.
CONCLUSION
- The paper’s framework treats training error and the log determinant of the Hessian as energy and entropy, with undersampling playing the role of temperature.In the undersampled regime, Bayes-optimal parameters balance training error against basin width.Source: CONCLUSION · p. 7
- SGD preferentially finds wide minima because its noise is correlated and anisotropic; experiments report higher entropy than for white-noise Langevin dynamics or noise-free gradient descent.At equal training error, higher-entropy solutions have lower testing error; the linear-network relationship is analytically explained.Source: CONCLUSION · p. 7
- The harmonic approximation’s Hessian-log entropy is limited by singular Hessians in overparameterized systems and by its omission of higher-order basin geometry.The paper calls for a more nuanced metric for flat minima with singular Hessians and points to basin-volume methods as a possible source of insight.Source: CONCLUSION · p. 7
- Because neural networks seldom converge to minima, relating saddle-point geometry to out-of-sample performance remains an open question.The paper notes that saddle points proliferate in high-dimensional energy landscapes.Source: CONCLUSION · p. 8