PaperlayerView exact revision on arXiv

Source-linked AI summary

Preconditioned Stochastic Gradient Langevin Dynamics for Deep Neural Networks

Chunyuan Li, Changyou Chen, David Carlson, Lawrence Carin

arXiv:1512.07666v1stat.ML

TL;DR

Deep neural networks are difficult to train because of pathological curvature and overfitting, while standard SGLD can be inefficient in this setting. The paper combines adaptive preconditioning with SGLD, yielding improved sampling efficiency and state-of-the-art performance on feedforward and convolutional neural networks.

  • Problem

    Pathological curvature makes deep neural networks difficult to optimize, while standard SGLD can be inefficient and overfitting remains a concern.

  • Method

    The paper develops preconditioned SGLD using adaptive preconditioners, with theoretical analysis of finite-time error, risk, and asymptotic convergence.

  • Results

    pSGLD improves effective sampling rates and achieves state-of-the-art performance on feedforward and convolutional neural networks, including 0.45% test error on MNIST.

  • Takeaways & Limitations

    Adaptive preconditioning enables SGLD to use local geometry while providing sample-based uncertainty for deep neural networks.

  • Takeaways & Limitations

    The RMSspectral comparison is omitted because it uses larger batch sizes and is difficult to compare on the reported scale.

Abstract

from arXiv · show

Effective training of deep neural networks suffers from two main issues. The first is that the parameter spaces of these models exhibit pathological curvature. Recent methods address this problem by using adaptive preconditioning for Stochastic Gradient Descent (SGD). These methods improve convergence by adapting to the local geometry of parameter space. A second issue is overfitting, which is typically addressed by early stopping. However, recent work has demonstrated that Bayesian model averaging mitigates this problem. The posterior can be sampled by using Stochastic Gradient Langevin Dynamics (SGLD). However, the rapidly changing curvature renders default SGLD methods inefficient. Here, we propose combining adaptive preconditioners with SGLD. In support of this idea, we give theoretical properties on asymptotic convergence and predictive risk. We also provide empirical results for Logistic Regression, Feedforward Neural Nets, and Convolutional Neural Nets, demonstrating that our preconditioned SGLD method gives state-of-the-art performance on these models.

Introduction

The introduction identifies pathological curvature and overfitting as key challenges in training deep neural networks, motivating a scalable Bayesian sampling method that combines adaptive preconditioning with SGLD. The proposed pSGLD method has theoretical risk and convergence guarantees and improves sampling efficiency through a simple, low-overhead procedure.

  • Motivation: Deep neural networks are difficult to train because their objectives are hard to optimize and their parameter spaces exhibit pathological curvature and saddle points.Adaptive preconditioning methods address local geometry and can improve optimization performance with low per-iteration cost.
  • Motivation: Bayesian learning can reduce overfitting by incorporating uncertainty, but exact Bayesian inference in deep neural networks is generally intractable.Approximate alternatives include stochastic variational inference, expectation propagation, and stochastic-gradient MCMC methods.
  • Motivation: SGLD provides scalable posterior sampling using gradients from small data mini-batches, but rapidly changing neural-network curvature makes existing SG-MCMC methods inefficient.Geometry-aware SG-MCMC methods can incur significant computational overhead and lack the scalability needed for deep neural networks.
  • Contribution: pSGLD combines adaptive optimization preconditioners with SGLD to improve sampling efficiency while adding trivial per-iteration overhead.The method is designed for deep neural networks, where existing SG-MCMC methods are hindered by pathological curvature and saddle points.
  • Contribution: The paper establishes risk bounds and asymptotic convergence properties for pSGLD and demonstrates an improved bias-variance tradeoff for small problems.These results support the method’s theoretical validity and empirical sampling efficiency.

Related Work

Prior work addresses neural-network overfitting through regularization and Bayesian uncertainty, while optimization research improves training using higher-order information and adaptive parameter rescaling. These lines of work motivate efficient posterior sampling and curvature-aware optimization.

  • Related Work: Regularization methods including early stopping, weight decay, dropout, and dropconnect have been developed to prevent neural-network overfitting, while Bayesian methods capture uncertainty during learning.MCMC methods produce Monte Carlo approximations to the posterior with asymptotic consistency.
  • Related Work: Higher-order gradient information can improve neural-network training, but computing it is often cumbersome in models of interest.Quasi-Newton and second-order approximations have shown promising results.
  • Related Work: Rescaling parameters to equalize loss curvature across directions improves performance in Adagrad, Adadelta, Adam, and RMSprop.This approach provides an alternative to full quasi-Newton methods; RMSprop has also been explained as a diagonal preconditioner.

Preliminaries

The preliminaries frame Bayesian prediction through posterior parameter uncertainty and contrast MAP optimization with SGLD-based posterior sampling. They also state the decreasing-step-size conditions required for convergence and asymptotic posterior consistency.

  • Bayesian formulation: The prior acts as a parameter penalty, the likelihood as the optimization loss, and MAP estimation maximizes the posterior log density.The optimization objective is θ_MAP = argmax log p(θ|D).
  • Stochastic optimization: SGD updates parameters using randomly selected mini-batches and step sizes, with convergence established for the stochastic optimization procedure.At iteration t, D_t is a random subset of n < N data items.
  • Bayesian prediction: Bayesian prediction averages p(y|x, θ) over the posterior, whereas MAP prediction substitutes θ_MAP and ignores parameter uncertainty.The Bayesian predictive estimate is p(y|x, D) = E_p(θ|D)[p(y|x, θ)], while MAP uses p(y|x, θ_MAP).
  • Posterior sampling: SGLD samples parameters from the posterior using mini-batch gradient steps, enabling Monte Carlo approximation of Bayesian predictive distributions.Given T samples, prediction is approximated by 1/T Σ_t p(y|x, θ_t).
  • Step-size assumptions: Decreasing step sizes are required for optimization convergence and asymptotic consistency of stochastic samples with the true posterior.Without suitable step sizes, gradient noise is not eliminated; for sampling, approximation error is dominated by Langevin stochasticity.

Preconditioned Stochastic Gradient Langevin Dynamics

The method preconditions SGLD to address slow mixing caused by heterogeneous curvature in deep neural networks. It uses an RMSprop-style diagonal preconditioner based on recent mini-batch gradients to equalize curvature and adapt step sizes.

  • Preconditioned Stochastic Gradient Langevin Dynamics: The preconditioner addresses standard SGLD’s slow mixing when neural-network parameters have different curvature.Deep neural networks can have ill-conditioned landscapes, with flat and highly curved directions that impede sampling near saddle points.
  • Preconditioned Stochastic Gradient Langevin Dynamics: Any positive definite preconditioning matrix defines a valid Riemannian metric, so exact expected Fisher information is not required.The proposed local transform aims to make curvature rates more equal across directions.
  • Preconditioned Stochastic Gradient Langevin Dynamics: Preconditioned SGLD uses an RMSprop-style diagonal preconditioner updated sequentially from current mini-batch gradient information.The running estimate uses exponential weighting of historical and current squared gradients, with element-wise operations.
  • Preconditioned Stochastic Gradient Langevin Dynamics: RMSprop-based preconditioning equalizes gradients and makes step sizes adaptive, assigning larger steps to flatter directions and smaller steps to more curved directions.The approach uses recent gradient magnitudes because flatter landscapes tend to have smaller gradients while curved landscapes have larger gradients.
  • Preconditioned Stochastic Gradient Langevin Dynamics: The method has two tuning parameters: λ controls curvature extremes, while α balances historical and current gradients.The stated defaults are λ=10^-5 and α=0.99, producing an exponentially decaying gradient sequence.

Preconditioned SGLD Algorithms in Practice

The section establishes finite-time error bounds and asymptotic consistency for preconditioned SGLD, then develops practical techniques that trade small bias for more effective samples and lower computational cost. It shows that omitting the correction term and thinning samples preserve controlled MSE behavior while improving sampling efficiency.

  • Method framework: The theoretical analysis extends prior SGLD convergence results to adaptive preconditioners and approximates posterior expectations with weighted MCMC sample averages.The method uses numerical integration of Langevin dynamics to estimate otherwise intractable posterior integrals.
  • Finite-time convergence: Decreasing-step-size pSGLD is asymptotically consistent with true posterior expectations because its finite-time approximation-error terms approach zero.The finite-time MSE bound includes stochastic-gradient estimation and numerical-integration errors.
  • Predictive risk: Predictive risk decomposes into squared bias and variance, with effective sample size and autocorrelation characterizing the variance under limited computation time.The section frames practical sampling as a bias–variance tradeoff: more effective samples can reduce total risk even when they introduce bias.
  • Excluding Γ(θt) term: Ignoring Γ(θt) adds a bias-controlled MSE term that is negligible when α is close to 1, while generating more samples per unit time and reducing predictive variance.The additional bound term is proportional to (1 − α)^2/α^3, which is approximately zero for α near 1.
  • Thinning samples: Thinning pSGLD samples reduces the ensemble size while lowering autocorrelation and potentially preserving a similar effective sample size.The MSE retains the same form under the thinning scheme.

Experiments

Experiments evaluate pSGLD through simulation, Bayesian logistic regression, and neural-network benchmarks. Across these settings, pSGLD improves sampling efficiency, predictive error, convergence, and test performance relative to established methods.

  • Simulation: In the 2D Gaussian simulation, pSGLD consistently produced lower reconstruction error and autocorrelation time than SGLD, especially at larger stepsizes.At sufficiently small stepsizes, both samplers performed similarly because the sampler moved little.
  • Bayesian Logistic Regression: For BLR on the Australian dataset, pSGLD generated larger effective sample sizes and smaller weight-estimation error than SGLD, reducing overall risk.The ground truth was obtained from 10^6 HMC samples with Metropolis-Hastings.
  • Bayesian Logistic Regression: On the a9a BLR dataset, both pSGLD and SGLD outperformed doubly stochastic variational Bayes in test error.The methods were compared using test errors in Table 1 and learning curves in Figure 3.
  • Feedforward Neural Networks: pSGLD achieved state-of-the-art FNN test accuracy and improved performance as network size increased, unlike BPB and SGD dropout.Stochastic sampling methods also outperformed their corresponding stochastic optimization methods on MNIST.
  • Feedforward Neural Networks: On MNIST learning curves, pSGLD and SGLD converged in fewer iterations and more stably than optimization methods, while pSGLD converged faster and to a better point than SGLD.RMSspectral was omitted from the plotted comparison because its larger batch sizes made it difficult to compare on that scale.
  • Convolutional Neural Networks: pSGLD reduced CNN test error to 0.45%, compared with 0.82% for traditional SGD on the same MNIST architecture.The comparison was intended to evaluate methods under the same model architecture rather than establish overall state-of-the-art results.

Conclusion

The paper develops RMSprop-based preconditioned SGLD with controllable finite-time approximation error, improving sampling and performance in deep neural networks. Experiments show state-of-the-art performance on feedforward and convolutional neural networks while providing sample-based uncertainty.

  • Conclusion: RMSprop-based preconditioned SGLD achieves state-of-the-art performance on FNN and CNN models while providing sample-based uncertainty.The method adapts to local geometry and addresses overfitting and pathological curvature in DNNs.
  • Conclusion: The developed preconditioned SGLD method has controllable finite-time approximation error and improves effective sampling rates by adapting to local geometry.The approach is based on RMSprop and is applied to deep neural networks.

Supplementary Material of Preconditioned Stochastic Gradient Langevin Dynamics for … Proof of Theorem 1

The supplementary material establishes Theorem 1 for pSGLD by adapting general SG-MCMC convergence techniques to a first-order Euler integrator with a preconditioner. It specifies Poisson-equation regularity assumptions and shows that the resulting error terms vanish under these conditions.

  • A. The proof for main theorem: The proof follows Chen, Ding, and Carin’s general SG-MCMC framework while treating the first-order numerical integrator and preconditioner used by pSGLD.The supplementary analysis specifically extends existing convergence techniques to the preconditioned setting.
  • Details on the assumption: The analysis defines the posterior average and approximates it with a sample average from pSGLD trajectories.The posterior average is represented through the invariant measure ρ(θ), while samples (θ_t) are used for approximation.
  • Details on the assumption: A Poisson-equation solution functional ψ characterizes the difference between each test-function value and its posterior average.Under elliptic or hypoelliptic settings, ψ is typically unique and at least as smooth as the test function.
  • Details on the assumption: Assumption 2 requires ψ and its derivatives through third order to be controlled by a smooth function V with bounded expectations along the trajectory.The stated bounds cover derivative orders k = 0, 1, 2, 3 and impose uniform moment and interpolation conditions on V.
  • Proof of Theorem 1: The proof compares the stochastic-gradient local generator with the true Langevin generator through the stochastic-gradient error term ΔV_t.The true generator replaces the stochastic gradient with the full gradient, isolating minibatch-gradient effects.
  • Proof of Theorem 1: Using the Euler integrator and the Poisson equation, the proof decomposes the test-function error, sums over iterations, and normalizes by S_T.The decomposition relies on the first-order integrator and the relation between the stochastic and true generators.
  • Proof of Theorem 1: Under the assumptions, the resulting bound is finite and all terms approach zero, completing the first part of Theorem 1.The Gaussian-noise contribution is bounded by O(√ϵ_t), while higher-order terms can be ignored after simplification.

B. The proof for Corollary 2

The proof of Corollary 2 establishes a bound on the diagonal preconditioner under bounded first- and second-order gradients, then uses it to derive the corollary’s mean-squared-error bound. The argument expands the exponential moving average, modifies the local generator by dropping Γ(θ_t) terms, and follows the proof of Theorem 1.

  • B. The proof for Corollary 2: Under bounded first- and second-order gradients, Lemma 4 establishes a bound for each diagonal component of Γ(θ_t).The proof treats one diagonal element because Γ(θ) is diagonal.
  • B. The proof for Corollary 2: The proof expresses the exponential moving average as a function of gradients from all previous timesteps.
  • B. The proof for Corollary 2: Dropping Γ(θ_t) terms yields a modified local generator for the pSGLD stochastic differential equation, with Δ˜V_t = ΔV_t + Γ(θ_t) · ∇θ.
  • B. The proof for Corollary 2: Following Theorem 1, the proof bounds (φ̂ − φ̄)^2 by the corresponding earlier expression plus an additional term, then relaxes that term using Lemma 4.
  • B. The proof for Corollary 2: Taking expectations produces the mean-squared-error bound for some C > 0.

C. The proof for Corollary 3 … Variance term in risk of estimator

The proof shows that thinning pSGLD samples yields a first-order integrator with a modified generator in the original SDE form and preserves the MSE derivation. The variance analysis relates estimator risk to autocorrelation, integrated autocorrelation time, and effective sample size, while omitting typically small long-lag terms.

  • C. The proof for Corollary 3: The proof constructs thinned subsamples from the original pSGLD trajectory and applies the first-order Euler integrator and Kolmogorov operators.The retained sequence has m ≤ n and is indexed by a subsequence of the original samples.
  • C. The proof for Corollary 3: Thinning pSGLD samples produces a first-order local integrator with stepsize S_ij and a modified generator retaining the original SDE form.The modified quantities are defined over the interval between retained samples θ_i and θ_j.
  • C. The proof for Corollary 3: The thinned-sample derivation yields the same mean squared error as Theorem 1 in the main text.
  • Variance term in risk of estimator: Terms with |t|>2T are omitted because autocovariance properties usually make their contribution small.
  • Variance term in risk of estimator: The autocovariance function describes correlation between samples separated by time lag t, while its normalized form is the autocorrelation function.
  • Variance term in risk of estimator: Integrated autocorrelation time measures the interval between independent samples, and effective sample size is defined from these dependence quantities.
  • Variance term in risk of estimator: The variance derivation substitutes the effective-sample-size definition into the estimator-risk calculation.

E. More results on simulation · Feedforward Neural Networks · Convolutional Neural Networks

Across simulations and neural-network experiments, pSGLD adapts to changing geometry, converges more quickly and stably than comparison methods, and achieves the lowest reported CNN test errors. These results span Gaussian sampling, feedforward networks on MNIST, and convolutional networks on MNIST and CIFAR-10.

  • E. More results on simulation: In the Gaussian simulation, pSGLD adapts its stepsizes to manifold geometry across dimensions and different stepsizes.The comparison varies a and ϵ, with results shown in Figure 1.
  • E. More results on simulation: When covariance scaling changes a from 0.5 to 2, pSGLD does not require a new stepsize, whereas standard SGLD needs fine-tuning for decent samples.The passage identifies ϵ = 0.1 as suitable for SGLD at a = 0.5 but unsuitable at a = 2.
  • Feedforward Neural Networks: For feedforward networks of sizes 400-400 and 800-800 on MNIST, pSGLD consistently converges faster and better than SGLD and other methods.Stochastic sampling methods also require fewer iterations and produce more stable results than optimization methods.
  • Convolutional Neural Networks: pSGLD achieves the lowest CNN test error on MNIST: 0.56% versus 0.76% for SGLD, 0.64% for RMSprop, and 0.81% for SGD.The comparison uses the same network architecture; the result is shown in the test-error table and described in the text.
  • Convolutional Neural Networks: On CIFAR-10, pSGLD converges faster and reaches a lower error than SGLD in the reported CNN learning-curve comparison.The experiment uses a three-layer CNN with 32-32-64 channels and no data augmentation.
Loading 1512.07666v1…