Source-linked AI summary
Entropy-SGD: Biasing Gradient Descent Into Wide Valleys
Pratik Chaudhari, Anna Choromanska, Stefano Soatto, Yann LeCun, Carlo Baldassi, Christian Borgs, Jennifer Chayes, Levent Sagun, Riccardo Zecchina
TL;DR
Deep-network optimization can reach low training loss without reliably finding solutions that generalize well. Entropy-SGD biases optimization toward wide, flat regions by estimating a local-entropy gradient with inner-loop Langevin dynamics, and the authors report favorable generalization behavior across deep-network experiments.
Problem
Good generalization remains challenging despite deep networks being easy to optimize to near-zero training loss.
Method
Entropy-SGD uses a local-entropy objective and inner-loop stochastic gradient Langevin dynamics to bias updates toward wide, flat valleys.
Results
Entropy-SGD obtains comparable generalization error to SGD while achieving lower training cross-entropy loss, and applies to large convolutional and recurrent networks.
Takeaways & Limitations
The results suggest that geometric properties of deep-network energy landscapes are important for generalization and can be exploited algorithmically.
Takeaways & Limitations
The theoretical generalization analysis assumes the Hessian has no eigenvalues in [−2γ−c,c], although observed spectra contain many near-zero eigenvalues.
Abstract
from arXiv · showhide
This paper proposes a new optimization algorithm called Entropy-SGD for training deep neural networks that is motivated by the local geometry of the energy landscape. Local extrema with low generalization error have a large proportion of almost-zero eigenvalues in the Hessian with very few positive or negative eigenvalues. We leverage upon this observation to construct a local-entropy-based objective function that favors well-generalizable solutions lying in large flat regions of the energy landscape, while avoiding poorly-generalizable solutions located in the sharp valleys. Conceptually, our algorithm resembles two nested loops of SGD where we use Langevin dynamics in the inner loop to compute the gradient of the local entropy before each update of the weights. We show that the new objective has a smoother energy landscape and show improved generalization over SGD using uniform stability, under certain assumptions. Our experiments on convolutional and recurrent neural networks demonstrate that Entropy-SGD compares favorably to state-of-the-art techniques in terms of generalization error and training time.
1 INTRODUCTION
The paper argues that well-generalizing solutions lie in wide, nearly flat valleys and introduces Entropy-SGD to actively seek such regions through a local-entropy objective.
- ≈94% of Hessian directions have near-zero eigenvalues below 10^-4, while the largest positive and negative eigenvalues are almost 40 and −0.46, respectively.
- Across architectures, sizes, datasets, and optimizers, well-generalizing minima tend to occupy wide valleys rather than sharp, isolated minima.Almost-flat regions are described as robust to parameter, activation, and data perturbations.
- Entropy-SGD replaces loss minimization with a local-entropy objective that measures both valley depth and flatness.
- The algorithm uses stochastic gradient Langevin dynamics in an inner loop to estimate the local-entropy gradient before outer-loop parameter updates.This creates two nested loops resembling SGD.
- The modified objective produces a smoother landscape controlled by γ, whose scope selects valleys of particular widths.The paper reports experiments on fully connected, convolutional, and recurrent networks across MNIST, CIFAR-10, and PTB.
2 RELATED WORK
The related work connects Entropy-SGD to theories of neural-network landscapes, flat-minimum methods, and local-entropy approaches, while distinguishing its scalable Langevin-based implementation.
- Prior theory and empirical work predicts many nearly equivalent minima and easy optimization, but good generalization remains challenging and depends on initialization, learning rates, and landscape-modifying techniques.
- Baldassi et al. (2015; 2016b;c) identify dense regions of minima that generalize well and use robust-ensemble weighting with belief propagation to access them.
- Entropy-SGD generalizes this local-entropy idea to continuous-weight deep networks without replicated-training hardware or belief-propagation estimation.
- Its enabling technique is Langevin dynamics, which estimates local-entropy gradients efficiently with mini-batch updates in large networks.
- Unlike hard constraints on loss and minimum width, the local-entropy objective avoids tightly coupled hyperparameters; unlike generic smoothing, it favors wide minima even when they are shallower than the global minimum.
3 LOCAL ENTROPY
Local entropy favors broad, low-loss neighborhoods over isolated minima by combining valley depth with flatness. Its scope parameter controls how locally the modified landscape is evaluated, producing a smoother objective that can prioritize wide valleys.
- A wider but shallower minimum can have higher marginalized likelihood than a sharp global minimum under a local prior.
- The modified Gibbs distribution uses γ to focus mass near x, with large γ enforcing locality and small γ allowing the energy term to dominate.
- The negative local-entropy landscape has a global minimum near the wider robust valley rather than the sharp non-robust minimum.
- Increasing γ makes the local-entropy landscape approach the original landscape, whereas very small γ makes it nearly uniform.
- Classical entropy alone can favor broad, high-loss regions, whereas local entropy distinguishes wide regions that are also deep enough to generalize.
4 ENTROPY-GUIDED SGD
Entropy-SGD replaces direct loss minimization with local-entropy optimization, estimating its gradient through an inner SGLD process before each outer weight update. The resulting objective is smoother and is theoretically associated with improved generalization under stated assumptions, while scoping complicates learning-rate annealing and increases computational work.
- 4.1 GRADIENT OF LOCAL ENTROPY: Entropy-SGD estimates the local-entropy gradient with L Langevin iterations inside each outer parameter update.The inner SGLD process uses mini-batch updates and produces an estimate used by the outer weight update.
- 4.1 GRADIENT OF LOCAL ENTROPY: A locally averaged Gibbs gradient points toward the wider of two similarly good minima, steering optimization toward wider valleys.
- 4.2 ALGORITHM AND IMPLEMENTATION DETAILS: Scoping increases γ during training to explore progressively finer scales, but it interferes with the learning-rate annealing commonly used in deep learning.
- 4.4 THEORETICAL PROPERTIES: The objective F(x,γ; Ξ) is smoother than the original objective, with smoothness scaled by 1+γ^-1 c.
- 4.4 THEORETICAL PROPERTIES: Under the stated Hessian-spectrum assumption, Entropy-SGD has better stability-based generalization than SGD when both converge after T passes and T > α.
5 EXPERIMENTS
Experiments across MNIST, CIFAR-10, and recurrent-network settings evaluate Entropy-SGD against SGD or Adam, with results covering Hessian structure, generalization, optimization smoothness, and training efficiency.
- Hessian structure: The Hessian analysis is constrained by computational cost: exact Hessians require smaller networks, while the largest network uses a diagonal Fisher-information proxy.The paper notes that Hessian-vector products could provide an alternative for large networks.
- MNIST: Entropy-SGD achieves comparable MNIST generalization to Adam: 1.37 ± 0.03% versus 1.39±0.03% for mnistfc, and 0.50 ± 0.01% versus 0.51±0.01% for LeNet.It is slightly faster for LeNet but marginally slower for the small mnistfc network.
- CIFAR-10: On CIFAR-10, Entropy-SGD performs as well as SGD on a large CNN, with training and validation curves showing almost no plateauing.The authors associate this behavior with a smoother local-entropy landscape and an appropriate scope schedule for γ.
- Recurrent networks: Entropy-SGD reaches 77.656±0.171 test perplexity on PTB-LSTM and trains faster than SGD, using 25 effective epochs versus 55 while achieving slightly better generalization perplexity.The validation perplexity is 80.116±0.069.
6 DISCUSSION
Entropy-SGD favors wide, flat regions that achieve lower training loss while maintaining generalization error comparable to SGD. Its wall-clock advantage is substantial for recurrent networks but modest for small feedforward networks.
- Comparable generalization error but consistently lower training cross-entropy than SGD suggests Entropy-SGD favors deeper wide valleys.The authors interpret these valleys as lying deeper in the landscape than SGD solutions, while noting that this contrasts with theories predicting equivalent minima.
- The results support treating deep-network energy-landscape geometry as important for generalization, although broad geometric theory remains underdeveloped.
- Significantly faster training than SGD for recurrent networks yields only minor wall-clock gains in the reported small-network setting.The discussion contrasts recurrent-network speedups with the limited gains observed for small networks.
7 CONCLUSIONS
The paper introduces Entropy-SGD, which uses local entropy to bias optimization toward flat regions with low generalization error. Experiments show applicability to large convolutional and recurrent networks.
- Entropy-SGD uses local entropy to bias optimization toward flat regions associated with low generalization error.
- Experiments demonstrate that Entropy-SGD applies to large convolutional and recurrent deep networks used in practice.
A STOCHASTIC GRADIENT LANGEVIN DYNAMICS (SGLD)
SGLD approximates posterior sampling with noisy gradient updates and mini-batches, making Langevin dynamics applicable to large datasets. In Entropy-SGD, this machinery estimates local-entropy gradients, while alternative MCMC methods remain possible.
- SGLD replaces full-dataset gradients with mini-batch averages, enabling Langevin updates to scale to large datasets.The full-data gradient is prohibitive, while mini-batch updates provide the scalable alternative.
- Langevin dynamics injects Gaussian noise into MAP updates, helping prevent over-fitting of the solution.
- As the learning rate approaches zero, SGLD can omit the Metropolis-Hastings rejection step, avoiding another full-dataset computation.
- With a uniform parameter prior, the prior-gradient term vanishes; the paper uses SGLD for ease of implementation despite other available MCMC methods.
B PROOFS
The proof links the modified objective’s smoothness to the Hessian spectrum. Under an eigenvalue-separation assumption, decreasing γ makes the energy landscape smoother, except near very flat directions.
- The proof approximates −∇F(x) through a saddle-point value that minimizes the exponent, using a Hessian-dependent quadratic form.
- A uniform smoothness bound requires that Hessian eigenvalues avoid the interval [−2γ−c,c] for a small constant c > 0.
- Decreasing γ makes the energy landscape smoother and reduces the Lipschitz constant, except at locations with very flat directions.
C CONNECTION TO VARIATIONAL INFERENCE
The paper distinguishes local entropy from standard variational inference: both construct weight-space distributions, but local entropy uses a parameter-dependent moving prior and is not generally equivalent to ELBO. Entropy-SGD estimates the local-entropy gradient with SGLD in an inner loop before updating the parameters.
- Local entropy and stochastic variational inference both construct distributions over weights to support generalization, but their objectives differ in general.
- Local entropy cannot generally be interpreted as an ELBO through a choice of prior or variational family.
- The two objectives coincide only formally for a flat variational family with an x-dependent distribution, which is not a genuine parameter-free prior.The corresponding distribution is uniform-like around x and depends on x itself.
- Unlike variational inference, local entropy uses a prior that moves with the current parameter iterate x.This moving prior is identified as a crucial feature of the local-entropy objective.
C.1 COMPARISON WITH SGLD
Compared with vanilla SGLD, Entropy-SGD achieved lower reported test error on LeNet and All-CNN-BN, while local-entropy results also improved on recurrent-network benchmarks. The paper attributes this advantage to transforming the energy landscape to exploit entropic effects rather than relying on conventional MCMC dynamics alone.
- 0.50±0.01% versus 0.63±0.1% on LeNet, and 7.81±0.09% versus 9.89±0.11% on All-CNN-BN, favored Entropy-SGD over SGLD.The comparisons were reported after 300 epochs for LeNet and 500 epochs for All-CNN-BN.
- 77.656±0.171 versus 94.03 test perplexity on PTB, and 1.213±0.007 versus 1.3375 on War and Peace, favored local entropy over reported SGLD results.
- Training deep networks with SGLD or more sophisticated MCMC methods to SGD-like error levels is difficult in the paper’s experiments and literature comparison.
- The paper explains local entropy’s effectiveness as operating on a transformed energy landscape that exploits entropic effects.