Source-linked AI summary
Variational Dropout Sparsifies Deep Neural Networks
Dmitry Molchanov, Arsenii Ashukha, Dmitry Vetrov
TL;DR
Deep neural networks contain many parameters, creating regularization and resource challenges, while prior Variational Dropout formulations restricted dropout rates and lacked experimental results for per-weight rates. The paper extends Variational Dropout to unbounded rates, reduces gradient-estimator variance, and evaluates individual dropout rates per weight. It obtains highly sparse fully-connected and convolutional networks with negligible performance loss, while identifying pre-training and unreported compression as important scope boundaries.
Problem
Variational Dropout theoretically permits individual dropout rates but restricts α ≤1 and had no experimental results for per-weight rates, while deep networks remain parameter- and resource-intensive.
Method
The paper introduces Sparse Variational Dropout with unbounded dropout rates, a tight full-domain KL approximation, and a lower-variance stochastic gradient estimator.
Results
Experiments produce highly sparse fully-connected and convolutional DNN layers, with state-of-the-art sparsity on LeNet and negligible performance drops on larger VGG-like networks.
Takeaways & Limitations
The method provides an Automatic Relevance Determination effect and can restrict model complexity without overfitting randomly labeled data like Binary Dropout networks.
Takeaways & Limitations
Random initialization can prune useful weights early and cause large accuracy drops, while the paper measures sparsity rather than actual network compression.
Abstract
from arXiv · showhide
We explore a recently proposed Variational Dropout technique that provided an elegant Bayesian interpretation to Gaussian Dropout. We extend Variational Dropout to the case when dropout rates are unbounded, propose a way to reduce the variance of the gradient estimator and report first experimental results with individual dropout rates per weight. Interestingly, it leads to extremely sparse solutions both in fully-connected and convolutional layers. This effect is similar to automatic relevance determination effect in empirical Bayes but has a number of advantages. We reduce the number of parameters up to 280 times on LeNet architectures and up to 68 times on VGG-like networks with a negligible decrease of accuracy.
1. Introduction
The paper extends Bayesian Variational Dropout to individually tuned, unbounded dropout rates and uses this framework to induce sparsity in deep neural networks. It reports sparse solutions across fully-connected and convolutional layers with little accuracy loss.
- Motivation: DNNs often have more parameters than training instances, making regularization necessary and motivating methods that reduce model size.Parameter sparsity can also reduce computational and memory requirements.
- Motivation: Dropout rates are typically shared across layers because optimizing multiple rates by grid search has exponential complexity.Bayesian interpretations of dropout make individual rates per weight, neuron, or layer theoretically tunable.
- Contributions: Sparse Variational Dropout extends Variational Dropout to all dropout-rate values using a tight approximation of the KL-divergence term.The method also introduces a lower-variance stochastic gradient estimator intended to improve convergence and the objective value.
- Contributions: Sparse Variational Dropout provides an Automatic Relevance Determination effect in linear models while overcoming certain disadvantages of empirical Bayes.The resulting linear-model solutions can be sparse.
- Results: Experiments find high sparsity in fully-connected and convolutional DNN layers, with state-of-the-art sparsity on LeNet and negligible performance drops on larger VGG-like networks.The method also fails to overfit randomly labeled data unlike Binary Dropout networks.
2. Related Work
Related work develops regularization and sparsification methods for deep networks, including Bayesian approaches and Sparse Bayesian Learning. Variational Dropout provides a Bayesian interpretation of Gaussian Dropout, but prior approaches had limitations in posterior flexibility, gradient estimation, or scalability.
- Sparse neural networks: Deep-network sparsification has been pursued through pruning, elastic-net regularization, tensor decompositions, and composite techniques.These approaches aim to reduce the number of parameters in deep models.
- Sparse Bayesian methods: Sparse Bayesian Learning offers principled sparse-model training without manual hyperparameter tuning, while Automatic Relevance Determination has been applied to neural and linear models.The Relevance Vector Machine is a representative linear-model application.
- Bayesian deep networks: Bayesian deep-network methods support scalable stochastic training and can address overfitting or provide model uncertainty.Related work also includes efficient training of Sparse Bayesian Models.
- Bayesian sparsification: Soft Weights Sharing combines sparse Bayesian ideas with flexible priors to obtain sparse and quantized Bayesian deep networks.
- Variational Dropout: Variational Dropout interprets Gaussian Dropout as Bayesian regularization and theoretically permits individual dropout rates, but its posterior family excludes the ARD effect by restricting α ≤1.Generalized Dropout uses a biased gradient estimator, while Spike-and-Slab networks do not scale well with model width and depth.
3. Preliminaries
The preliminaries formulate stochastic variational inference and explain dropout as multiplicative noise with a Bayesian posterior interpretation. Variational Dropout optimizes per-weight distribution parameters, while reparameterization methods provide stochastic gradient estimators and can reduce their variance.
- Stochastic Variational Inference: Variational inference approximates the posterior p(w | D) with a parametric distribution qφ(w), optimized through a variational lower bound.The bound combines expected log-likelihood with a KL-divergence regularization term.
- Stochastic Variational Inference: Because expectations are intractable in complex models, the variational lower bound and its gradients are estimated by sampling and optimized stochastically.
- Stochastic Variational Inference: The Reparameterization Trick expresses weight noise as a differentiable function of variational parameters and non-parametric noise, enabling unbiased minibatch gradient estimates.
- Stochastic Variational Inference: The Local Reparameterization Trick reduces gradient-estimator variance by sampling per-data-point weight matrices through efficiently moved activation noise.
- Layer formulation: For a fully connected layer, inputs A and weights W produce outputs B through B = AW, with dimensions determined by the input and output neuron counts.
- Variational Dropout: Dropout injects multiplicative noise into layer inputs; Gaussian noise is equivalent to Gaussian noise on weights and supports a posterior distribution over weights.Variational Dropout tunes the posterior parameters θ and α through stochastic variational inference.
- Variational Dropout: In Variational Dropout, α is a variational parameter rather than a hyperparameter and can theoretically be assigned separately to each weight, neuron, or layer.The original formulation manually restricted the approximate posterior to α ≤1 and reported difficulties with larger dropout rates.
4. Sparse Variational Dropout
Sparse Variational Dropout extends Variational Dropout to unbounded, per-weight dropout rates by improving KL-divergence approximation and gradient estimation. The resulting posterior can drive weights toward zero, enabling sparse fully connected and convolutional networks while retaining the method’s Bayesian regularization advantages.
- 4.1. Additive Noise Reparameterization: Large dropout rates αij are difficult to train because multiplicative noise creates high-variance stochastic gradients.The original formulation was limited to α≤1, whereas αij→∞ corresponds to weights that are effectively always ignored.
- 4.1. Additive Noise Reparameterization: The additive-noise reparameterization replaces θij(1 + √αij·ϵij) with θij + σij·ϵij while preserving the same weight distribution and objective.Optimizing with respect to (θ, σ) greatly reduces gradient variance and permits training across αij∈(0,+∞).
- 4.2. Approximation of the KL Divergence: The new KL-divergence approximation is tight across the full range of α, unlike earlier approximations accurate only for α≤1.Its maximum absolute deviation is less than 0.009 over log αij∈(−∞,+∞), compared with 0.04 for the original approximation over log αij∈(−∞,0].
- 4.3. Sparsity: As αij grows, the regularization favors large dropout rates, which can make the corresponding variational distribution effectively a delta function at zero.The paper explains this sparsity effect as weights being almost always dropped and then set to zero at test time.
- 4.3. Sparsity: In linear regression, the optimal coefficient θi tends to zero as αi grows, demonstrating the sparsifying behavior analytically.The result assumes the i-th feature is not identically zero.
- 4.4. Implementation: Sparse Variational Dropout combines the KL approximation with additive-noise and local-reparameterization techniques for fully connected and convolutional layers.The implementation is available for Lasagne and PyTorch, and both forward and backward passes take twice as much time as the original layers.
- 4.5. Relation to RVM: Unlike empirical Bayes, Sparse Variational Dropout obtains the automatic relevance determination effect through variational inference with a fixed prior.The authors state that this avoids additional overfitting from model selection.
5. Experiments
The experiments evaluate Sparse Variational Dropout across neural architectures, initialization strategies, parameterizations, and random-label settings. The method produces highly sparse models, with accuracy usually preserved, but initialization substantially affects complex networks.
- Sparse Variational Dropout produces extremely sparse models across fully-connected and convolutional neural networks.
- Pre-training avoids the high accuracy drop observed when random initialization prunes weights before useful representations can form.Warm-up offers an alternative optimization strategy by gradually increasing the KL-divergence weight.
- Initialization matters little for LeNet but significantly affects more complex VGG-like models, with the final variational lower bound used to select a strategy.
- Additive Noise Reparameterization yields faster convergence, a better variational lower bound, and higher sparsity than the original parameterization.
- LeNet experiments achieve the highest sparsity among compared methods with similar accuracy, while the study reports sparsity rather than final compression ratio.
- The method achieves over 65x sparsification on CIFAR-10 without accuracy drop and up to 41x on CIFAR-100 with a moderate accuracy drop.
- On randomly labeled data, Sparse Variational Dropout drops every weight and produces constant predictions, unlike Binary Dropout networks that fit the labels.The paper notes that learning random labels remains possible after pre-training, but with a lower variational lower bound than the fully sparse solution.
6. Discussion
The discussion interprets Sparse Variational Dropout as a sparsity-inducing form of regularization related to automatic relevance determination. It also identifies missing compression measurements and structured sparsity as important boundaries and future directions.
- Allowing Variational Dropout to remove irrelevant weights cuts most model weights, restricting model complexity as a form of regularization.
- Unlike standard dropout networks in the cited random-label setting, the model favors a zero-weight network with a higher objective than a trained memorizing network.
- The study measures sparsity but not actual network compression, leaving the resulting compression ratio unreported.
- Future work includes combining sparsification with quantization and Huffman coding and obtaining structured sparsity for acceleration.