PaperlayerView exact revision on arXiv

Source-linked AI summary

Entropy-SGD: Biasing Gradient Descent Into Wide Valleys

Pratik Chaudhari, Anna Choromanska, Stefano Soatto, Yann LeCun, Carlo Baldassi, Christian Borgs, Jennifer Chayes, Levent Sagun, Riccardo Zecchina

arXiv:1611.01838v5cs.LGstat.ML

TL;DR

Deep-network optimization should account for energy-landscape geometry because well-generalizing solutions are associated with wide, flat regions. The paper introduces Entropy-SGD, which optimizes a local-entropy objective using nested Langevin dynamics, and reports favorable generalization behavior across deep-network experiments, with computational cost measured explicitly.

  • Problem

    The paper addresses how to modify deep-network optimization to favor wide, flat regions associated with better generalization rather than sharp valleys.

  • Method

    Entropy-SGD maximizes a local-entropy objective and estimates its gradient with inner-loop stochastic gradient Langevin dynamics before outer-loop parameter updates.

  • Results

    Entropy-SGD is applicable to large convolutional and recurrent networks and produces generalization error comparable to SGD while achieving lower training cross-entropy loss.

  • Takeaways & Limitations

    The results support using energy-landscape geometry and local entropy to bias optimization toward wide valleys in practical deep networks.

  • Takeaways & Limitations

    Entropy-SGD requires L SGLD steps before each parameter update, making its effective dataset passes L times those of SGD or Adam for the same number of updates.

Abstract

from arXiv · show

This paper proposes a new optimization algorithm called Entropy-SGD for training deep neural networks that is motivated by the local geometry of the energy landscape. Local extrema with low generalization error have a large proportion of almost-zero eigenvalues in the Hessian with very few positive or negative eigenvalues. We leverage upon this observation to construct a local-entropy-based objective function that favors well-generalizable solutions lying in large flat regions of the energy landscape, while avoiding poorly-generalizable solutions located in the sharp valleys. Conceptually, our algorithm resembles two nested loops of SGD where we use Langevin dynamics in the inner loop to compute the gradient of the local entropy before each update of the weights. We show that the new objective has a smoother energy landscape and show improved generalization over SGD using uniform stability, under certain assumptions. Our experiments on convolutional and recurrent networks demonstrate that Entropy-SGD compares favorably to state-of-the-art techniques in terms of generalization error and training time.

1 INTRODUCTION

The paper argues that well-generalizing solutions lie in wide, nearly flat valleys of deep-network energy landscapes and proposes modifying SGD to seek them. Entropy-SGD uses a local-entropy objective approximated through nested Langevin-dynamics updates.

  • Hessian geometry: The largest positive Hessian eigenvalue is almost 40, whereas the largest negative eigenvalue is only −0.46.The negative directions correspond to descent directions missed by the optimizer.
  • Wide valleys: Across architectures, network sizes, datasets, and optimization algorithms, well-generalizing minima are associated with wide valleys rather than sharp isolated minima.Almost-flat regions are described as robust to data, activation, and parameter perturbations.
  • Objective: The paper asks whether SGD can be modified to actively seek wide valleys instead of minimizing only the original loss f(x).This motivates maximizing a local-entropy-based objective.
  • Algorithm: Entropy-SGD uses stochastic gradient Langevin dynamics in an inner loop to approximate local-entropy gradients before outer-loop parameter updates.The resulting objective is described as producing a smoother energy landscape controlled by γ, a scope parameter for valley widths.

2 RELATED WORK

The related work connects flat minima and local entropy to prior theoretical and optimization approaches. Entropy-SGD extends local-entropy ideas to modern deep networks with continuous weights using Langevin-based gradient estimation.

  • Landscape theory: Prior theory predicts saddle points with many negative directions at high training error and local minima with training error near the global minimum.These results arise from analyses relating deep-network losses to high-dimensional random fields and statistical-physics models.
  • Landscape theory: Existing analyses also predict multiple almost-equivalent local minima, though their assumptions may be unrealistic for practical deep networks.Related results extend across deep networks, matrix factorization, and tensor factorization.
  • Optimization challenges: Good generalization remains difficult because complex architectures are sensitive to initialization and learning rates, while even linear networks can have hard-to-escape saddle points.Adaptive or annealed learning rates, momentum, dropout, and batch normalization address related optimization challenges.
  • Local-entropy approaches: For shallow networks with discrete weights, dense clusters of nearby minima can generalize well and be reached using a robust-ensemble measure.These results motivate seeking dense or wide regions rather than isolated minima.
  • Entropy-SGD: Entropy-SGD generalizes local-entropy methods to modern deep networks with continuous weights without replicated-training hardware or belief-propagation estimates.Langevin dynamics estimates local-entropy gradients efficiently through mini-batch updates.
  • Comparison with prior methods: Unlike hard constraints on loss and valley width, local entropy naturally balances training loss against valley width through γ.The paper describes γ as a focusing parameter with computational advantages.
  • Comparison with smoothing: Local-entropy optimization differs from loss smoothing because it can favor a wider, shallower minimum, whereas smoothing may create an artificial minimum between sharp valleys.Continuation methods also require successive smoothed minimizers to remain close, which is unclear for deep networks.

3 LOCAL ENTROPY

Local entropy reweights optimization toward parameter neighborhoods that combine low loss with broad support, favoring wide valleys over sharp minima. Its scope parameter controls how locally the modified distribution explores the energy landscape.

  • Local-entropy intuition: A wider but shallower minimum can have higher marginalized likelihood than a sharper global minimum under a fixed-variance Bayesian prior.The paper uses x_robust and x_non-robust to illustrate this distinction.
  • Local-entropy intuition: Local entropy is large near wide minima because it aggregates contributions from many good parameters, whereas sharp minima contribute fewer such parameters.This makes valley width part of the optimization objective rather than an external diagnostic.
  • Modified Gibbs distribution: The modified Gibbs distribution balances the loss f(x′) against proximity to the current location x through the scope parameter γ.Large γ concentrates probability near x; small γ lets the energy term dominate and permits broader exploration.
  • Definition: Local entropy F(x,γ) is defined as the log-partition function of the modified Gibbs distribution.The objective therefore measures valley depth together with flatness within the chosen scope.
  • Effect on the landscape: Negative local entropy produces a smoother landscape whose global minimum can lie near a wide valley, especially for low γ.This differs from classical entropy, which can favor broad regions with very high training loss.

4 ENTROPY-GUIDED SGD

Entropy-SGD minimizes negative local entropy by estimating its gradient with an inner SGLD process and updating weights in an outer optimization loop. The resulting objective is theoretically smoother and is reported to improve generalization under stated assumptions, while its inner iterations increase effective data passes.

  • 4 ENTROPY-GUIDED SGD: Entropy-SGD uses nested optimization loops: inner SGLD estimates the local-entropy gradient, while outer SGD updates the network parameters.The inner expectation is difficult to compute directly and is approximated with mini-batch stochastic gradient Langevin dynamics.
  • 4.1 GRADIENT OF LOCAL ENTROPY: The locally focused Gibbs average points toward the wider of two similarly low-loss minima, steering updates toward wider valleys.Unrolling the SGLD steps also makes the method resemble averaged SGD.
  • 4.2 ALGORITHM AND IMPLEMENTATION DETAILS: Increasing γ during training progressively narrows the explored neighborhood, a schedule the paper calls scoping.Large γ restricts SGLD near the current parameters, whereas small γ permits exploration farther away.
  • 4.4 THEORETICAL PROPERTIES: The local-entropy objective is smoother than the original objective under β-smoothness and a Hessian spectral-gap assumption.Under these assumptions, the paper derives Lipschitz and smoothness bounds for F(x,γ; Ξ).
  • 4.4 THEORETICAL PROPERTIES: Entropy-SGD is shown to generalize better than SGD for T > α when both algorithms converge after T passes over the samples.This conclusion follows from combining the smoothness result with a uniform-stability bound.
  • 4.2 ALGORITHM AND IMPLEMENTATION DETAILS: Entropy-SGD can require substantially more effective dataset passes because it performs L SGLD steps before each parameter update.The paper reports similar CNN passes but half as many RNN passes as SGD in its experiments.
  • 4.4 THEORETICAL PROPERTIES: The theoretical Hessian assumption is acknowledged as unrealistic because practical minima can contain many nearly zero eigenvalues.A rigorous bound without this assumption would require a dynamical analysis of SGD that the paper says is currently out of reach.

5 EXPERIMENTS

The experiments examine Hessian geometry across architectures and evaluate Entropy-SGD against SGD or Adam on image and text prediction tasks. Entropy-SGD achieves comparable or better generalization, with faster training reported on the PTB-LSTM experiment, while its effective dataset passes scale with the inner-loop length.

  • Experimental setup: The experiments cover Hessian spectra and Entropy-SGD comparisons on MNIST, CIFAR-10, PTB, and War and Peace using convolutional, fully connected, and recurrent networks.The study includes small and large networks, including a PTB-LSTM with about 66 million weights.
  • Universality of the Hessian: Hessian spectra at SGD optima are mostly concentrated near zero, with long positive-curvature tails and very small negative-curvature tails.The reported pattern indicates mostly flat optima with few negative-curvature directions of small magnitude.
  • MNIST: 1.37 ± 0.03% and 0.50 ± 0.01%: Entropy-SGD obtains these MNIST errors for mnistfc and LeNet, respectively, compared with Adam baselines of 1.39±0.03% and 0.51±0.01%.Entropy-SGD trains slightly faster for LeNet but is marginally slower for the small mnistfc network.
  • Computational complexity: Entropy-SGD performs L SGLD steps before each parameter update, so its effective number of dataset passes is L times that of SGD or Adam for the same update count.The paper plots against effective epochs as a hardware-agnostic proxy for wall-clock time, modulo parameter-update costs.
  • Penn Tree Bank: Entropy-SGD uses 5 inner SGLD steps on PTB-LSTM, producing 25 effective epochs and slightly better generalization perplexity than SGD while training significantly faster.The validation perplexity is 80.116 ± 0.069 and test perplexity is 77.656 ± 0.171.
  • Recurrent networks: Entropy-SGD and Entropy-Adam experiments on recurrent networks require tuned momentum, with Entropy-SGD using 0.5 momentum on PTB-LSTM and Entropy-Adam using β1 = 0.5 on char-LSTM.The corresponding baselines use no momentum for PTB-LSTM and β1 = 0.9 for char-LSTM Adam.

6 DISCUSSION

Entropy-SGD favors wide valleys and achieves generalization error comparable to SGD while consistently reducing training cross-entropy. Its training-speed advantage is clearer for recurrent networks than in wall-clock time overall.

  • Empirical findings: Entropy-SGD achieves generalization error comparable to SGD while always obtaining lower training cross-entropy loss.The authors interpret this as evidence that its favored wide valleys lie deeper than SGD’s solutions.
  • Interpretation: The authors connect Entropy-SGD’s wide-valley preference to the importance of energy-landscape geometry for generalization.They contrast this interpretation with models predicting multiple equivalent minima having the same loss.
  • Open questions: The paper identifies the lack of general results on deep-network loss geometry, particularly for convolutional networks, as an open direction.This is presented as a limitation of the current literature rather than a failure established for Entropy-SGD.
  • Training time: Entropy-SGD trains significantly faster than SGD for recurrent networks, but yields relatively minor wall-clock-time gains in the reported broader setting.The passage preserves the distinction between faster recurrent-network training and limited overall wall-clock improvement.

7 CONCLUSIONS

The paper introduces Entropy-SGD to bias deep-network optimization toward flat regions associated with low generalization error. Experiments show applicability to large convolutional and recurrent networks used in practice.

  • Conclusion: Entropy-SGD is motivated by observations that SGD-discovered minima are nearly flat across architectures, datasets, and training methods.The conclusion connects this observation to local entropy as the optimization biasing mechanism.
  • Conclusion: The method uses local entropy to bias optimization toward flat regions that have low generalization error.This states the paper’s central connection between local geometry and generalization.
  • Conclusion: Experiments show that Entropy-SGD applies to large convolutional and recurrent deep networks used in practice.The conclusion states applicability rather than a broader claim about all architectures.

A STOCHASTIC GRADIENT LANGEVIN DYNAMICS (SGLD)

SGLD approximates the difficult local-entropy gradient through MCMC sampling while scaling updates with mini-batches. The paper uses SGLD for implementation simplicity despite several more sophisticated alternatives.

  • Motivation and setup: Local entropy averages over the entire configuration space, so its gradient is approximated with MCMC techniques such as SGLD.SGLD is designed to draw samples from a Bayesian posterior and scale to large datasets with mini-batch updates.
  • Langevin dynamics: SGLD injects Gaussian noise into MAP updates, helping prevent over-fitting of the parameter solution.The passage introduces Langevin dynamics as the stochastic sampling mechanism used in the method’s inner loop.
  • Scalability: Mini-batch average gradients replace full-dataset gradients, avoiding prohibitive computation for large datasets.This substitution is explicitly attributed to the scalable SGLD formulation.
  • Sampling properties: As the learning rate tends to zero, the Metropolis-Hastings rejection step can be omitted, and generated samples converge to the correct posterior under the cited argument.The resulting samples can be used to compute statistics of parameter functions.
  • Prior choice: With a uniform parameter prior, the prior-gradient term vanishes from the SGLD update.The paper therefore considers only the remaining update contributions in this setting.
  • Algorithm choice: The authors use SGLD for ease of implementation, while noting alternatives including SGHMC and stochastic annealing thermostats.These alternatives are presented as applicable MCMC methods rather than evaluated replacements here.

B PROOFS

The proof analysis establishes that decreasing γ smooths the modified energy landscape, but uniform bounds require an eigenvalue-separation assumption.

  • Proof strategy: The Hessian-based proof approximates −∇F(x) through a saddle-point choice of s that minimizes the exponent.This connects the gradient calculation to the local curvature analysis.
  • Proof strategy: The displayed proof step plugs the saddle-point approximation into the smoothness condition.It is an intermediate algebraic step in deriving the smoothness result.
  • Assumption: The proof notes that uniform bounds require no Hessian eigenvalue to lie in [−2γ−c,c] for a small constant c>0.This is the explicit spectral assumption used to obtain the bound.
  • Smoothness result: A smaller γ produces a smoother energy landscape except in very flat directions.The proof also states that the Lipschitz constant decreases by the same factor.

C CONNECTION TO VARIATIONAL INFERENCE

The paper compares local entropy with stochastic variational inference and concludes that they are generally different because local entropy uses a parameter-dependent moving prior. Their objectives coincide only under a formal flat variational family.

  • Connection to Variational Inference: Local entropy and variational inference both construct distributions over weight space to pursue good generalization, but their objectives differ in general.The paper frames the comparison through the local-entropy objective and the Evidence Lower Bound (ELBO).
  • Connection to Variational Inference: The ELBO uses a parameter-free prior p(z), while local entropy contains a data-likelihood term and a prior that moves with the current iterate x.This moving prior is identified as a crucial feature of local entropy and is absent from variational inference.
  • Connection to Variational Inference: Local entropy cannot generally be interpreted as ELBO because no choice of prior or variational family makes the two formulations equivalent.The paper derives this distinction using Jensen’s inequality.
  • Connection to Variational Inference: Formally, equivalence holds only when qx(z) belongs to a flat variational family with a uniform distribution centered at x and a parameter-dependent prior.The paper emphasizes that this parameter dependence means the distribution is not a genuine parameter-free prior derived from ELBO.
  • Connection to Variational Inference: Gradient averaging through weight or activation perturbations computes only the data-likelihood term and therefore differs from local entropy, which also includes the KL-related structure of the objective.The paper contrasts perturbation-based averaging with local entropy rather than treating them as equivalent methods.

C.1 COMPARISON WITH SGLD

The paper compares Entropy-SGD with vanilla SGLD on convolutional networks and reports lower test error for Entropy-SGD. It attributes the difference to Entropy-SGD’s transformation of the energy landscape through entropic effects, whereas conventional MCMC methods struggle to reach comparable errors.

  • Comparison with SGLD: SGLD achieved test errors of 0.63±0.1% on LeNet and 9.89 ± 0.11% on All-CNN-BN after 300 and 500 epochs, respectively.These results followed a hyper-parameter search.
  • Comparison with SGLD: Entropy-SGD achieved test errors of 0.50 ± 0.01% on LeNet and 7.81 ± 0.09% on All-CNN-BN.The paper presents these results as substantially better generalization than the corresponding SGLD experiments.
  • Comparison with SGLD: On recurrent networks, local entropy obtained lower test perplexity than SGLD on both PTB and War and Peace.The reported values were 77.656 ± 0.171 versus 94.03 on PTB, and 1.213 ±0.007 versus 1.3375 on War and Peace.
  • Comparison with SGLD: The paper reports that training deep networks with SGLD or other MCMC methods to SGD-like errors is difficult.It connects this difficulty to the absence of comparable results in the literature and to its experimental experience.
  • Comparison with SGLD: Entropy-SGD is presented as effective because it operates on a transformed energy landscape that exploits entropic effects, unlike conventional MCMC techniques.The comparison concerns the optimization landscape rather than only the sampling procedure.
Loading 1611.01838v5…