Variational Dropout and the Local Reparameterization Trick

Diederik P. Kingma; Tim Salimans; Max Welling

Source-linked AI summary

Variational Dropout and the Local Reparameterization Trick

Diederik P. Kingma, Tim Salimans, Max Welling

arXiv:1506.02557v2Jun 8, 2015stat.MLcs.LGstat.CO

TL;DR

Bayesian inference for neural networks is attractive for controlling overfitting, but existing variational methods suffer from high-variance gradients and have not clearly surpassed dropout. The paper introduces local reparameterization, which converts global parameter uncertainty into independent local noise for efficient SGVB, and extends Gaussian dropout by learning dropout rates. The resulting approach supports full Bayesian analysis and more flexible posteriors, with reported encouraging empirical results.

Problem
Existing variational Bayesian methods for neural networks have been limited by high-variance gradients and have not shown clear improvement over dropout-based regularization.
Method
The paper translates global parameter uncertainty into independent local noise for SGVB and interprets dropout variationally to derive variational dropout with data-inferred rates.
Results
The proposed local-reparameterization approach achieves optimization speed on the same level as fast dropout while retaining full Bayesian analysis and greater posterior flexibility.
Takeaways & Limitations
Local reparameterization provides an efficient, parallelizable SGVB estimator, while variational dropout makes dropout rates adaptive to the data.
Takeaways & Limitations
Directly sampling separate weight matrices for each training example is computationally inefficient and prevents standard optimized matrix operations.

Abstract

from arXiv · show

We investigate a local reparameterizaton technique for greatly reducing the variance of stochastic gradients for variational Bayesian inference (SGVB) of a posterior over model parameters, while retaining parallelizability. This local reparameterization translates uncertainty about global parameters into local noise that is independent across datapoints in the minibatch. Such parameterizations can be trivially parallelized and have variance that is inversely proportional to the minibatch size, generally leading to much faster convergence. Additionally, we explore a connection with dropout: Gaussian dropout objectives correspond to SGVB with local reparameterization, a scale-invariant prior and proportionally fixed posterior variance. Our method allows inference of more flexibly parameterized posteriors; specifically, we propose variational dropout, a generalization of Gaussian dropout where the dropout rates are learned, often leading to better models. The method is demonstrated through several experiments.

1 Introduction

The paper targets overfitting in flexible neural networks by developing efficient Bayesian posterior inference that can compete with dropout while offering greater flexibility. It introduces local reparameterization as a faster variational-inference approach and connects Gaussian dropout to SGVB.

Motivation: Neural networks scale to millions of parameters and datapoints but can overfit by fitting spurious training patterns.Dropout is described as a popular and empirically effective regularization technique.
Motivation: Bayesian posterior inference offers a theoretically attractive approach to controlling overfitting, but exact inference is computationally intractable.Approximate methods such as MCMC and variational inference are therefore needed.
Research gap: Stochastic variational inference for neural networks has been limited by high gradient variance and has not yet shown clear gains over simpler dropout regularization.The paper frames this as a limitation of existing Bayesian approaches despite their theoretical appeal.
Contribution: Local reparameterization translates global parameter uncertainty into independent local noise across minibatch datapoints, improving SGVB efficiency and reducing gradient variance.The resulting optimization speed is reported to be on the same level as fast dropout.
Contribution: The method retains full Bayesian analysis while allowing more flexible posterior parameterizations than standard dropout.Gaussian dropout is presented as a special case of the proposed approach.

2 Efficient and Practical Bayesian Inference

The paper develops local reparameterization for SGVB by moving global weight uncertainty into independently sampled local activations. This removes minibatch covariance contributions, yields 1/M gradient-variance scaling, and reduces sampling cost while preserving efficient matrix operations.

Variational inference: Variational inference approximates the intractable posterior qφ(w) by optimizing the variational lower bound and minimizing DKL(qφ(w)||p(w|D)).The expected log-likelihood and KL terms form the optimization objective.
SGVB: SGVB reparameterizes sampled weights as w = f(ϵ, φ), producing an unbiased differentiable minibatch estimator for the expected log-likelihood.Stochastic gradient ascent on the bound then updates the variational parameters.
Variance of SGVB: The SGVB estimator’s individual variance decreases with minibatch size M, but covariance contributions do not, so they can dominate for moderately large M.This covariance term limits practical progress when gradient variance remains high.
Local reparameterization: The local reparameterization estimator enforces Cov[Li, Lj] = 0, making stochastic-gradient variance scale as 1/M.It samples intermediate variables rather than the original global noise directly.
Local reparameterization: Local reparameterization translates global weight uncertainty into independent local uncertainty in intermediate activations, yielding computationally and statistically efficient gradient estimates.The approach applies when global noise can be moved into local intermediate states.
Computational constraint: Sampling separate weight matrices per example would remove covariance but require M million random numbers and prevent optimized matrix-matrix operations.The same issue affects convolutional networks because optimized convolution libraries cannot use separate filters per example.
Activation sampling: Sampling activations directly requires M thousand random variables instead of M million for the example layer, a thousand-fold savings.The activation distribution remains factorized Gaussian conditional on the input under a factorized Gaussian weight posterior.
Gradient estimation: Local reparameterization also produces lower-variance gradients than drawing random weights, because each local noise variable has a more direct influence on its gradient.With random weights, many noise variables influence each gradient term and their individual effects are obscured.

3 Variational Dropout

Variational dropout interprets continuous-noise dropout as variational inference and extends it by learning noise parameters under a scale-invariant prior.

Dropout interpretation: Dropout adds multiplicative noise during training, while Gaussian dropout directly samples approximately Gaussian activation marginals.The Gaussian construction can be interpreted through local reparameterization, although it ignores dependencies between activation noises.
Posterior parameterization: Variational dropout treats dropout posteriors as a mean parameter θ multiplied by noise governed by α.The framework includes independent weight uncertainty and correlated weight noise, corresponding to two Gaussian-dropout variants.
Prior and objective: Dropout training is variationally justified when the prior is scale-invariant and log-uniform, making the KL divergence independent of θ.Under this prior, the KL term also has an interpretation related to controlling the number of significant digits represented by the posterior.
Adaptive regularization: The variational lower bound permits α, including dropout rates, to be optimized rather than fixed as hyperparameters.Adaptive rates can be learned per layer, neuron, or individual weight, but constraining α ≤ 1 avoids difficult large-variance-gradient optima and corresponds to a dropout rate of 0.5.
Variational objective: For a factorized Gaussian posterior, the negative KL divergence is not analytically tractable and is instead approximated accurately using a fitted expression.The same approximation is used for the correlated scale-variable posterior.

4 Related Work

Prior work established practical stochastic variational inference and related dropout interpretations, but these approaches remained connected to broader questions about efficient Bayesian regularization.

Variational inference: Reparameterization-based SGVB estimators provide efficient and unbiased stochastic-gradient inference for many continuous latent variables or parameters.Earlier variational methods for neural networks had introduced practical estimators, while later work generalized the reparameterization approach.
Dropout connections: Dropout and DropConnect can be interpreted as variational inference with different noise structures and corresponding posterior assumptions.Related work also includes learned dropout-rate mechanisms such as Standout.

5 Experiments

Experiments compare variational dropout with standard and Gaussian dropout on gradient variance, optimization speed, and classification error across neural-network settings.

Experimental setup: The experiments use standard binary dropout, two Gaussian-dropout variants, and two variational-dropout variants on MNIST and CIFAR-10 network settings.The MNIST fully connected architecture has three hidden layers, and results are averaged over five runs in Figure 1(a).
Variance: The local reparameterization trick yields the lowest variance among tested variational-dropout gradient estimators across all conditions.Its variance scales as 1/M, and drawing fewer random numbers provides an additional variance reduction of about a factor of 2.
Speed: 1635 seconds per epoch versus 7.4 seconds gives the local-reparameterization estimator an over 200-fold wall-clock speedup over the naïve estimator.The comparison uses a modern GPU implementation and contrasts separate weight samples per datapoint with local reparameterization.
Classification error: Adaptive variational Gaussian-dropout methods perform equal to or better than non-adaptive Gaussian dropout and standard dropout under all tested conditions.The advantage is especially noticeable for smaller networks, where inferred dropout rates are lower than in larger networks.
Classification error: A threefold downscaled KL-divergence term prevents underfitting and beats all other tested dropout methods in the reported models.This result is identified as Variational (A2) in Figure 1.

6 Conclusion

The paper presents local reparameterization as an efficient SGVB technique and dropout as a special case, then proposes variational dropout with learned rates.

Conclusion: Local reparameterization translates global parameter uncertainty into independent local noise per datapoint, yielding low-variance, parallelizable SGVB estimators.The method is described as having low computational complexity and variance that decreases with minibatch size.
Conclusion: Dropout is a special case of SGVB with local reparameterization, while variational dropout infers dropout rates from data instead of fixing them in advance.The paper reports encouraging empirical results for this extension.
Conclusion: The log-uniform prior gives the KL divergence a natural interpretation as controlling the number of significant digits under the variational posterior.This connects the variational objective to the representation of floating-point parameters.

B Derivation of dropout’s implicit variational posterior

The appendix interprets Gaussian dropout as uncertainty over weights, showing that stochastic linear operations can be represented through a multiplicative variational posterior whose dropout rates may be optimized.

Dropout replaces each linear operation b_i = a_iW with a stochastic operation involving the input vector for datapoint i.
The resulting dropout outputs are characterized by their expected values and variances with respect to the dropout rate.
Ignoring dependencies between output elements, dropout can be approximated by a Gaussian distribution.
Equivalently, Gaussian dropout can be parameterized as uncertainty over the weights rather than as explicit output noise.
Treating p(V_jk|W_jk, α_jk) as a variational posterior allows each α_jk, the dropout rate for an individual parameter, to be optimized.

C Negative KL-divergence for the log-uniform prior

The derivation computes the negative KL term for a factorized posterior under a log-uniform prior, then specializes it to multiplicative and Gaussian dropout posteriors with a tractable approximation.

The variational lower bound includes an expected log likelihood term and a negative KL-divergence penalty that must be evaluated for the approximate posterior.
The log-uniform prior combines a Bernoulli sign distribution with a uniform distribution over log(|w_i|).
The KL divergence separates into contributions from the sign bit and the conditional posterior and prior on the log scale.
For multiplicative noise posteriors, the terms depending on θ cancel exactly, making the KL divergence independent of the mean parameters.
The log-uniform prior is the only prior identified as consistent with dropout because additional terms involving w_i would not cancel with the entropy term.
For Gaussian dropout, the remaining expectation over log(|ε_i|) is approximated numerically by a third-degree polynomial, which is reported as extremely close to the exact value.
The figure reports indistinguishable exact and approximate negative KL values for dropout rates between p = 0.05 and p = 0.5.

D Variance reduction of local parameterization compared to separate weight samples

The local reparameterization analysis explains why moving uncertainty from shared weights to local activation noise reduces the variance of stochastic gradient estimates.

The analysis compares gradient estimators formed from separate sampled weight matrices with estimators using the local reparameterization trick.
Both estimators share the same first variance term, but differ in an additional term arising from the sampling representation.
With separate weight samples per training example, conditioning on the minibatch output leaves an additional positive variance contribution.
With local reparameterization, the corresponding noise variable is uniquely determined by the output, so the additional variance term vanishes.

E Variance of stochastic gradients for variational dropout with correlated weight noise

For correlated weight noise, the empirical comparison finds similar relative estimator performance to the independent-noise analysis, but generally higher noise levels.

The experiment measures minibatch stochastic-gradient estimator variance for variational dropout with correlated weight noise.
The relative performance of the different estimators is similar to that reported for the corresponding analysis in section 5.
The noise level is generally much higher with correlated weight noise than in the referenced comparison.
For the table’s specification, separate weight samples and local reparameterization have identical stochastic-gradient variance when dropout rates are not optimized.

F Variance of SGVB estimator with minibatches of datapoints without replacement

The minibatch estimator’s variance combines datapoint-selection effects with covariance induced by using one shared noise sample across the minibatch. The selection contribution decreases with minibatch size, while shared-noise covariance can dominate for moderately large minibatches.

The estimator rewrites minibatch sampling with indicator variables for training-example inclusion and per-example log-likelihood terms.The indicators denote minibatch membership, while Li(ϵ, φ) abbreviates each example’s log likelihood under reparameterized weights.
Ignoring negative correlations between indicators yields an upper-bound variance expression for the minibatch estimator.The bound uses the expected value of each inclusion indicator.
The variance from random datapoint selection is inversely proportional to minibatch size M.
Shared noise ϵ contributes covariance across datapoint losses, and this contribution does not decrease with M.A single noise sample is used for the entire minibatch, with the cross-example covariance positive on average for i.i.d. data.
For moderately large M, the estimator variance can be dominated by the shared-noise contribution.