Source-linked AI summary
Structured Bayesian Pruning via Log-Normal Multiplicative Noise
Kirill Neklyudov, Dmitry Molchanov, Arsenii Ashukha, Dmitry Vetrov
TL;DR
Existing Bayesian dropout can produce highly sparse but unstructured networks whose evaluation may not accelerate. This paper proposes Structured Bayesian Pruning, a Bayesian dropout-like layer for structured sparsity, reporting substantial acceleration with negligible accuracy loss.
Problem
Bayesian dropout can create highly sparse networks, but unstructured sparsity may leave evaluation time unchanged and is difficult to exploit for acceleration.
Method
Structured Bayesian Pruning injects multiplicative noise into layer outputs, tunes noisy masks by stochastic variational inference, and uses truncated log-uniform and log-normal distributions.
Results
Experiments show high structured sparsity and significant convolutional-network acceleration with negligible or no significant accuracy drop across multiple architectures and classification tasks.
Takeaways & Limitations
SBP removes redundant neurons or convolutional filters while preserving compatibility with existing software implementations through a separate dropout-like layer.
Takeaways & Limitations
With an improper log-uniform prior, variational optimization is ill-posed and can collapse the posterior to the prior, preventing information extraction.
Abstract
from arXiv · showhide
Dropout-based regularization methods can be regarded as injecting random noise with pre-defined magnitude to different parts of the neural network during training. It was recently shown that Bayesian dropout procedure not only improves generalization but also leads to extremely sparse neural architectures by automatically setting the individual noise magnitude per weight. However, this sparsity can hardly be used for acceleration since it is unstructured. In the paper, we propose a new Bayesian model that takes into account the computational structure of neural networks and provides structured sparsity, e.g. removes neurons and/or convolutional channels in CNNs. To do this we inject noise to the neurons outputs while keeping the weights unregularized. We establish the probabilistic model with a proper truncated log-uniform prior over the noise and truncated log-normal variational approximation that ensures that the KL-term in the evidence lower bound is computed in closed-form. The model leads to structured sparsity by removing elements with a low SNR from the computation graph and provides significant acceleration on a number of deep neural architectures. The model is easy to implement as it can be formulated as a separate dropout-like layer.
1 Introduction
Bayesian dropout can tune noise magnitudes automatically, but weight-level sparsity is difficult to accelerate. Structured Bayesian Pruning instead targets neurons and convolutional filters, producing structured sparsity and practical acceleration.
- Motivation: Bayesian dropout methods automatically tune dropout rates, extending regularization beyond fixed-magnitude Binary and Gaussian Dropout.These methods provide theoretical justification for dropout variants while adapting noise magnitudes during training.
- Motivation: Unstructured weight sparsity can compress neural networks while leaving evaluation time unchanged because common software handles sparse matrices inefficiently.Structured sparsity instead removes redundant neurons or convolutional filters to support acceleration.
- Proposed approach: Structured Bayesian Pruning uses a dropout-like layer with parametric multiplicative noise and stochastic variational inference to induce structured sparsity.Its proper sparsity-inducing log-uniform prior avoids problems associated with an improper prior.
- Results: Experiments on LeNet and VGG-like architectures using MNIST and CIFAR-10 show high group sparsity and significant CNN acceleration with negligible accuracy drop.The reported evaluations cover both compact and deeper architectures.
2 Related Work
Prior acceleration methods exploit structured groups, intermediate representations, filters, layers, skipped computation, reduced precision, or low-rank approximations. These approaches aim to obtain practical speedups with limited software changes.
- Structured acceleration: Structured sparsity methods remove convolutional filters, intermediate matrix rows, or entire layers rather than individual weights.These structures are designed to produce practical acceleration with little to no modification of existing software.
- Alternative approaches: Other approaches reduce computation through skipped convolutions, adaptive computation time, one-bit weights, or low-rank filter approximations.The related methods target different sources of computational and storage cost.
3 Stochastic Variational Inference
The paper uses variational inference to approximate Bayesian treatment of noisy masks while learning ordinary neural-network weights by maximum likelihood. Reparameterization enables differentiable stochastic optimization with low computational overhead.
- Variational inference: Variational inference approximates the posterior p(θ | D) with a parametric distribution qφ(θ) by minimizing KL divergence.This is equivalent to maximizing the variational lower bound L(φ).
- Optimization: The expected log-likelihood is generally intractable for complex models, so the reparameterization trick provides an unbiased differentiable minibatch Monte Carlo estimator.Samples are generated as a deterministic function of variational parameters and non-parametric noise.
- Optimization: Stochastic gradient ascent optimizes the variational parameters and can be applied efficiently to deep neural networks with usually small computational overhead.The procedure solves the variational optimization problem using stochastic gradients.
- Model parameters: The model separates Bayesian noisy-mask parameters θ from neural-network weights and biases w, which are learned by maximum likelihood.The approximate posterior is formed for the masks while the remaining parameters use their maximum-likelihood estimates.
4 Group Sparsity with Log-normal Multiplicative Noise
The model induces group-wise sparsity by applying Bayesian multiplicative noise to neuron outputs rather than weights, using truncated log-uniform and log-normal distributions. It prunes low-SNR neurons or filters while retaining a tractable variational objective and the same training procedure.
- Model construction: A dropout-like layer places multiplicative noise on intermediate outputs, while neural-network weights and other parameters remain maximum-likelihood parameters.Noise variables receive the Bayesian treatment; weights, biases, and batch-normalization parameters are optimized conventionally.
- Model construction: Sharing noise variables across arbitrary groups enforces group-wise sparsity on parameters or intermediate tensors.The grouping mechanism can target structured elements such as neurons, convolutional filters, or other computational groups.
- Variational inference: The variational posterior is fully factorized log-normal because it matches the multiplicative-noise formulation and can approach the log-uniform prior as σ goes to infinity.The log-normal posterior corresponds to Gaussian additive noise in logarithmic space and avoids a prior gap in the no-data limit.
- Limitations and correction: The improper log-uniform prior makes optimization ill-posed, whereas the truncated replacement avoids posterior collapse and supports correct stochastic variational inference.With the improper prior, the optimum occurs as σ goes to infinity, causing the posterior to collapse into the prior and preventing data information extraction.
- Variational inference: Truncating both prior and posterior over a shared finite support makes the probabilistic model proper and the KL divergence computable for all valid variational parameters.The reparameterization trick remains available through the inverse CDF of the truncated normal, and the final loss and training procedure remain unchanged.
- Structured pruning: Neurons or filters with SNR below 1 are removed by setting their outputs to zero, with no accuracy drop reported across the experiments.SBP layers remove convolutional filters after convolutional layers and input neurons before fully connected layers.
5 Experiments
Experiments evaluate Structured Bayesian Pruning across architectures and datasets, showing structured sparsity, acceleration, and little accuracy loss. They also examine truncation settings, variational-parameter optimization, and Bayesian ensembling.
- Experiment overview: Structured Bayesian Pruning achieved high structured sparsity in convolutional filters and neurons without significant accuracy drop across supervised tasks and architectures.The evaluation includes deep VGG-like architectures with batch normalization layers and reports retained neurons, filters, and acceleration.
- Pruning criterion: SNR < 1 identified redundant neurons and filters whose removal caused no accuracy drop in all reported experiments.SBP layers were placed after convolutional layers and before fully connected layers to remove filters and input neurons.
- LeNet comparisons: Table 1 compares SSL, SparseVD, and the proposed method on LeNet-5-Caffe and LeNet-500-300 using accuracy, group-wise sparsity, and CPU, GPU, and FLOPs acceleration.The caption states that the proposed method provides the highest speed-up with similar accuracy.
- Variational optimization: Optimizing both variational parameters µ and σ produced a tighter variational lower bound, higher sparsity, and more efficient sparsification than tuning only noise variance.The comparison used a fully connected MNIST network with two hidden layers of 1000 neurons each.
6 Conclusion
The paper presents Structured Bayesian Pruning as a dropout-like Bayesian layer for adaptive structured sparsity and practical neural-network acceleration without changing existing software implementations.
- 6 Conclusion: Structured Bayesian Pruning injects multiplicative random noise over preceding-layer outputs and tunes its distribution using stochastic variational inference.The layer uses a sparsity-inducing prior over noise variables.
- 6 Conclusion: SBP can induce arbitrary structured sparsity patterns over inputs and cut down neurons and convolutional filters in neural networks.The paper reports significant practical acceleration without modifying existing software implementations.
A KL divergence for the truncated log-normal distribution
This appendix develops the KL divergence for a truncated log-normal variational distribution and a bounded-support log-uniform prior, including the truncated-normal entropy term.
- Notation: The notation tN(x | µ, σ^2, a, b) represents a normal distribution with mean µ and standard deviation σ truncated between thresholds a and b.The truncation notation is introduced for deriving the KL divergence.
- KL divergence: The KL divergence is calculated between truncated log-normal q(θ_i) and log-uniform p(θ_i) distributions with support θ_i ∈ [e^a, e^b].The bounded support enables the appendix’s closed-form treatment.
- Entropy: The derivation requires the entropy of the truncated normal distribution.The appendix explicitly introduces the entropy expression as part of the calculation.
B Sampling from the truncated log-normal distribution
Sampling from the truncated log-normal distribution proceeds by sampling a truncated normal log-value and exponentiating it, using inverse-CDF reparameterization.
- Sampling procedure: To sample θ_i, the method first samples log θ_i from a truncated normal distribution and then applies the exponential function.The truncated normal uses parameters µ_i, σ_i, a, and b.
- Reparameterization: Inverse-CDF sampling converts uniform random values into samples from the truncated normal distribution.The appendix derives the truncated-normal CDF and its inverse before presenting the reparameterization trick.
C Mean of the truncated log-normal ditribution
This section derives the probability density and expectation of the truncated log-normal distribution. The obtained density is described as very similar to the log-normal PDF.
- The derivation begins by finding the PDF of the truncated log-normal distribution.
- The truncated log-normal PDF is stated to be very similar to the log-normal distribution PDF.
- Using the derived formula, the section obtains the expectation Ex.
D Signal-to-noise ratio of the truncated log-normal distribution
This section derives the signal-to-noise ratio for the truncated log-normal distribution by calculating its variance and combining it with the expectation.
- The signal-to-noise ratio Ex/Var(x) is calculated to investigate sparsity in the resulting layer.
- The variance Var(x) is calculated as a required component of the SNR derivation.
- The derivation introduces a new variable t and uses a variable substitution for the remaining summands.
- Combining the expression for Ex with the derived terms yields the signal-to-noise ratio.
E Stable Computation of Statistics
This section addresses numerical instability when computing truncated log-normal statistics at high σ values. It uses the scaled complementary error function to stabilize the calculations.
- Direct computation of the SNR and mean can produce indeterminate values such as 0 · ∞ when σ is high.
- The scaled complementary error function is defined as erfcx(x) = exp(x2)erfc(x) for stable calculations.
- Equations (19) and (20) are rewritten using the scaled-function formulation.