PaperlayerView exact revision on arXiv

Source-linked AI summary

Bayesian Compression for Deep Learning

Christos Louizos, Karen Ullrich, Max Welling

arXiv:1705.08665v4stat.MLcs.LG

TL;DR

Deep networks remain costly to deploy because their energy, speed, communication, and hardware requirements challenge real-world use. The paper applies variational Bayesian compression with group sparsity and uncertainty-based precision selection. It reports theoretically principled compression, reduced computation, improved energy efficiency, and competitive compression and accuracy outcomes, while noting limitations from mean-field variance estimates and finetuning error.

  • Problem

    Deep networks are difficult to deploy efficiently because energy consumption, prediction speed, communication bandwidth, and hardware constraints remain substantial.

  • Method

    The method uses variational Bayesian inference with hierarchical sparsity-inducing priors to prune hidden units and posterior uncertainty to determine fixed-point weight precision.

  • Results

    Bayesian compression yields smaller architectures and weight precisions below the standard 32 bits per layer, while achieving competitive compression and accuracy outcomes.

  • Takeaways & Limitations

    Bayesian methods provide a unified approach to theoretically principled compression, reduced computation, improved energy efficiency, and learned weight bit precisions.

  • Takeaways & Limitations

    Mean-field variational approximations can underestimate variance and thereby lead to higher weight bit precisions; finetuning with Sparse VD increased error by around 3%.

Abstract

from arXiv · show

Compression and computational efficiency in deep learning have become a problem of great significance. In this work, we argue that the most principled and effective way to attack this problem is by adopting a Bayesian point of view, where through sparsity inducing priors we prune large parts of the network. We introduce two novelties in this paper: 1) we use hierarchical priors to prune nodes instead of individual weights, and 2) we use the posterior uncertainties to determine the optimal fixed point precision to encode the weights. Both factors significantly contribute to achieving the state of the art in terms of compression rates, while still staying competitive with methods designed to optimize for speed or energy efficiency.

1 Introduction

Deep networks are difficult to deploy efficiently because their energy, speed, communication, and hardware demands remain substantial. The paper proposes Bayesian compression using node-level sparsity and posterior uncertainty to reduce structure and weight precision.

  • Motivation: Deep networks remain difficult to deploy in real-world scenarios because prediction energy, speed, communication, and hardware constraints are substantial.The motivation includes billions of daily predictions, real-time latency, band-limited channels, and hardware-limited devices.
  • Motivation: Compression and performance optimization can diverge: AlexNet convolutional layers contain 4% of parameters but account for 91% of computation.Compressing these layers therefore has limited effect on overall memory footprint.
  • Existing approaches: Existing approaches commonly reduce network structure and effective fixed-point precision, including pruning unnecessary connections and student-teacher learning.These approaches are motivated by significant parameter redundancy in neural networks.
  • Bayesian perspective: A Bayesian perspective connects sparsity-induced pruning and reduced bit precision with model selection and posterior uncertainty.The paper relates this connection to the bits-back argument and the MDL principle.
  • Contribution: The proposed variational Bayesian method uses sparsity-inducing priors for hidden units, pruning neurons together with their ingoing and outgoing weights.This avoids more complicated coding schemes for pruning or vector-quantizing individual weights.
  • Contribution: Posterior uncertainty is used to identify insignificant bits and derive a practical optimal fixed-point precision for each layer.Weights that fluctuate too much under approximate posterior sampling contribute fewer significant bits.

2 Variational Bayes and Minimum Description Length

The paper frames variational Bayesian inference as an MDL-compatible compression procedure. Its objective balances data fit, prior-based model cost, and posterior entropy while enabling stochastic optimization and group-wise pruning.

  • Minimum Description Length: The MDL principle selects hypotheses by minimizing the bits needed to communicate model complexity and data misfit.Variational inference can be reinterpreted from this minimum-description-length perspective.
  • Variational Bayes: Variational inference approximates the intractable posterior p(w|D) with a fixed-form posterior qφ(w) by optimizing variational parameters φ.The model likelihood is p(D|w), with parameters w governed by prior p(w).
  • Variational objective: The ELBO combines a communication cost for targets under an agreed prior and model with an entropy term for the approximate posterior.The entropy is denoted H(·), and L(φ) is the evidence-lower-bound or negative variational free energy.
  • Compression mechanisms: Sparsity-inducing priors on groups of weights feeding a neuron prune hidden units that are unnecessary for prediction.Group-wise pruning achieves compression by removing units rather than independently encoding every weight.
  • Compression mechanisms: Noisy weight encodings can exploit the bits-back argument through posterior entropy, unlike infinitely precise weights with H(δ(w)) = −∞.The practical weight-precision term is treated separately from the intractable neural-network data-misfit term.
  • Optimization: The reparameterization trick produces unbiased stochastic gradients of the ELBO, while local reparameterization reduces estimator variance by sampling pre-activations.The experiments use local reparameterizations to marginalize weights locally at each layer.

3 Related Work

Related work addresses efficiency through pruning and quantization, but weight-level sparsity can incur storage and computation overhead. Quantization methods differ in practicality because some require matrix reconstruction or specialized hardware support.

  • Pruning: Pruning is an early direct efficiency strategy, with prior work reporting removal of up to 99,5% of parameters in common architectures.Empirical Bayesian approaches have also explored weight pruning.
  • Pruning: Weight pruning can be inefficient for compression because matrix storage requires the Compressed Sparse Column format.Conventional CNN computation is dominated by convolution, motivating schemes that account for computational structure.
  • Quantization: Quantization reduces the accessible weight set or the bit size per weight after an optimal architecture has been selected.Prior work includes binary, tertiary, k-means, product, residual, fixed-point, and hashing quantization.
  • Quantization: Set quantization is impractical at test time because each feature map requires reconstructing the original weight matrix from indexes and a codebook.Precision quantization avoids this reconstruction requirement by reducing bits per weight.

4 Bayesian compression with scale mixtures of normals

The paper uses hierarchical scale-mixture priors and variational inference to induce group sparsity, pruning neurons or feature maps rather than individual weights. Posterior uncertainty then guides weight bit precision, while log-uniform and horseshoe priors provide alternative shrinkage mechanisms.

  • Scale-mixture priors: Scale-mixture priors place random scales over weights, producing heavier-tailed marginal priors with more mass at zero and biasing posteriors toward sparsity.The paper identifies scale mixtures of normals as a general family containing several sparsity-inducing distributions.
  • Alternative shrinkage priors: The log-uniform prior and half-Cauchy prior offer continuous alternatives to spike-and-slab shrinkage, with the latter inducing a horseshoe prior and global-local shrinkage.The horseshoe’s global scale pulls variables toward zero, while heavy-tailed local scales allow some weights to escape.
  • Group sparsity: Sharing one scale across weights in a group couples their shrinkage, allowing entire neurons or feature maps and their associated parameters to be pruned together.This group-level approach avoids coding schemes designed for individually pruned or vector-quantized weights.
  • Variational dropout: Variational inference uses a factorized approximate posterior over weights and scales, with group pruning triggered by thresholding the corresponding variational dropout rate.The threshold is expressed through the group’s log variational dropout rate and a pruning threshold.
  • Inference at test time: At test time, the stochastic weight distribution is replaced by a single masked variational posterior mean for one feedforward pass.The mask is determined by group variational dropout rates, while the matrix entries use the means of the variational posterior.
  • Precision coding: Posterior marginal variances determine significant bits for each weight matrix, after which three exponent bits and one sign bit are added to obtain final precision.The method uses mean variance across a weight matrix to compute the unit roundoff needed for its representation.
  • Limitation: Mean-field variational approximations can underestimate posterior variance, potentially producing higher weight bit precisions than richer posterior approximations.The paper leaves more involved posterior families for future work.

5 Experiments

The experiments evaluate Bayesian Compression on LeNet and VGG, measuring learned architectures, bit precisions, compression, speed, and energy. The method produces smaller architectures and low-bit weights while remaining competitive in compression and execution efficiency.

  • 5.1 Architecture learning & bit precisions: Bayesian Compression infers smaller architectures for LeNet-300-100 and LeNet-5-Caffe than Sparse VD, Generalized Dropout, and Group Lasso.For VGG, the large 512-feature-map layers are reduced to around 10 feature maps while initial layers remain mostly intact.
  • 5.1 Architecture learning & bit precisions: All evaluated Bayesian methods use fewer than the standard 32 bits per layer, sometimes reaching 5-bit weight precisions.Table 1 reports the neurons remaining after pruning together with average per-layer weight bit precisions.
  • 5.2 Compression Rates: The compression comparison covers pruning alone, pruning with reduced per-layer weight precision, and maximum compression using k-means clustering with k=32.The k-means alternative stores weight indices and a codebook but requires restoring the original matrix for each layer at test time.
  • 5.2 Compression Rates: Bayesian Compression is competitive with state-of-the-art compression on LeNet-300-100 and achieves significantly better compression on LeNet-5-Caffe without accuracy loss.For VGG, random initialization yielded around 1%-2% less accuracy than initializing the posterior means from a pretrained network.
  • 5.3 Speed and energy consumption: Both the proposed models and the best competing models reach a speed-up factor of around 8×, while the proposed architecture saves about 3× GPU energy relative to the original.The measurement uses batches of 8192 samples and averages 104 forward passes; the reported speed result uses the pruned architecture only.

6 Conclusion

The paper presents Bayesian compression as a unified approach to deep-network efficiency and compression. It links principled compression, reduced computation, energy efficiency, and learned weight bit precisions, while reporting a finetuning caveat for Sparse VD.

  • 6 Conclusion: Bayesian Compression addresses deep-network compression and efficiency in a unified and principled way.The conclusion frames the approach as theoretically principled compression with reduced computation.
  • 6 Conclusion: The proposed methods reduce computation and energy while naturally learning bit precisions for individual weights.The authors present these properties as an argument for Bayesian methods when compression and speed-up matter.
  • 6 Conclusion: Finetuning the same network with Sparse VD increased error by around 3%, so those results were not reported.This caveat concerns the attempted finetuning experiment.

A. Detailed experimental setup

The detailed setup specifies optimization, initialization, standard-deviation constraints, preprocessing, and how pruning propagates across layers. It also identifies floating-point formats as part of the experimental context.

  • A. Detailed experimental setup: The methods were implemented in TensorFlow and optimized with Adam using default hyperparameters.Variational parameters were optimized directly in the implementation.
  • A. Detailed experimental setup: The variational parameters were initialized using a published scheme for Gaussian means and sampled log standard deviations from N(−9, 1e −4).The group-scale parameters were initialized with mean approximately 1 and variance approximately 1e −8 so all groups began active.
  • A. Detailed experimental setup: Table 3 summarizes floating-point formats used as the reference context for precision choices.The supplied table passage identifies the topic but does not provide its entries.
  • A. Detailed experimental setup: Standard-deviation constraints vary by architecture, with tighter limits assigned to selected early layers in LeNet and VGG.The remaining standard deviations were left unconstrained where specified.
  • A. Detailed experimental setup: MNIST digits were rescaled to [−1, 1], while CIFAR-10 used the preprocessing supplied with its dataset.No additional MNIST preprocessing is reported.
  • A. Detailed experimental setup: Pruning a convolutional filter also removes the corresponding next-layer feature-map parameters, and dropping a fully connected input neuron removes related previous-layer weights.This reflects group-level structural pruning across adjacent layers.

B. Standards for Floating-Point Arithmetic

The paper reviews binary floating-point standards and explains how exponent and significand bits determine representable values and precision. It then motivates tailored formats with fewer exponent bits for network parameters.

  • B. Standards for Floating-Point Arithmetic: IEEE 754-2008 defines binary16, binary32, binary64, and binary128 formats, commonly called half, single, double, and quadruple precision.Each format contains one sign bit, exponent bits, and precision bits.
  • B. Standards for Floating-Point Arithmetic: The sign bit determines sign, the exponent is a signed w-bit integer, and the mantissa contains significant bits with an implicit leading one.These components determine the floating-point representation of a value.
  • B. Standards for Floating-Point Arithmetic: Floating-point formats are characterized by overflow, underflow, and unit round off, which depend on exponent and significant-bit counts.The paper uses these quantities when selecting a custom format.
  • B. Standards for Floating-Point Arithmetic: The proposed custom format uses significantly fewer exponent bits because network parameters usually lie between [-10,10].The unit round off is set equal to the precision to compute the significant bits needed for a weight.
  • B. Standards for Floating-Point Arithmetic: Mixed-precision deep learning can assign different precisions to activations and weights, such as high-precision activations with low-precision weights.This is presented as related work beyond standard floating-point formats.

C. Shrinkage properties of the normal-Jeffreys and horseshoe priors

The normal-Jeffreys and horseshoe priors both strongly shrink near zero, but differ in their treatment of intermediate shrinkage and tails. The log-uniform prior concentrates mass near pruning or retaining parameters, whereas horseshoe retains intermediate shrinkage flexibility.

  • Both priors behave similarly near zero, while the normal-Jeffreys prior has an extremely heavy, non-normalizable tail.
  • The shrinkage coefficient λ provides a continuous relaxation of spike-and-slab behavior, ranging from no shrinkage at λ = 0 to exact zero at λ = 1.
  • The log-uniform prior concentrates almost all probability near λ ≈ 0 or λ ≈ 1, effectively favoring parameter retention or pruning.
  • The horseshoe prior preserves probability mass at intermediate λ values and can potentially provide better regularization and generalization.

D. Negative KL-divergences for log-normal approximating posteriors

The appendix derives negative KL-divergence expressions for log-normal approximating posteriors under inverse-gamma and gamma priors. These expressions use log-normal moments and entropy to obtain tractable variational objectives.

  • The approximating posterior q(z) is modeled as log-normal with parameters μ and σ^2.
  • For inverse-gamma priors, the negative cross-entropy is expressed using E_q(z)[log z] and E_q(z)[z^-1].
  • The log-normal identities E_q(z)[log z] = μ and E_q(z)[z^-1] = exp(-μ + 0.5σ^2) substitute moments into the inverse-gamma expression.
  • For gamma priors, the negative cross-entropy and negative KL-divergence are written in terms of α, β, μ, σ^2, and the log-normal expectation E_q(z)[z].
  • The resulting scale-prior divergence is combined with the weight-distribution divergence given by equation 8 in the main paper.

E. Visualizations

The visualizations show how Bayesian Compression separates signal from noise through group thresholds and assigns weights substantially fewer than 32 bits. The accompanying algorithms implement stochastic forward passes for group normal-Jeffreys and group horseshoe layers.

  • Threshold distributions: BC-GNJ and BC-GHS threshold distributions usually form two separable groups, making signal-versus-noise threshold selection easy.Smaller thresholds indicate signal, whereas larger thresholds indicate noise or useless groups.
  • Bit precisions: All compared methods usually require far fewer than 32 bits to represent weights in the three-layer LeNet-300-100 architecture.
  • Forward-pass algorithms: The forward-pass algorithms use local reparameterization with approximate posteriors for BC-GNJ and BC-GHS in fully connected and convolutional layers.
  • Forward-pass algorithms: Fully connected BC-GNJ and BC-GHS layers compute activation means and variances from H, M_w, and Σ_w before sampling output noise.
  • Forward-pass algorithms: Convolutional BC-GNJ layers apply convolution to weight means and variances, then modulate the mean and variance using sampled scale variables.
  • Forward-pass algorithms: The convolutional BC-GHS algorithm is identified as a separate layer procedure alongside the convolutional BC-GNJ procedure.
Loading 1705.08665v4…