Source-linked AI summary
Compressing Neural Networks using the Variational Information Bottleneck
Bin Dai, Chen Zhu, David Wipf
TL;DR
The paper addresses how to reduce neural-network memory and computation while preserving accuracy, focusing on pruning individual neurons. It uses a variational information bottleneck whose learned sparse structure removes redundant neurons with a single compression–accuracy tuning parameter. Across standard benchmarks, VIBNet achieves comparable accuracy with improved compression, although theoretical pruning guarantees rely on a local data-fit condition and the bound can become loose in certain activation regimes.
Problem
Neural networks can be over-parameterized, producing unnecessarily large computational and memory footprints even when predictive performance is high.
Method
VIBNet applies a tractable variational information bottleneck that reduces inter-layer redundancy and prunes neurons through learned Gaussian variances, using one compression–accuracy tuning parameter.
Results
Across common compression benchmarks, VIBNet achieves comparable accuracy with improved compression against numerous competing neuron-pruning methods.
Takeaways & Limitations
The framework simultaneously targets model size, FLOPs, and run-time memory through intrinsic sparse neuron pruning without additional prior or constraint hyper-parameters.
Takeaways & Limitations
The pruning sufficiency guarantee requires the data term to increase locally with neuron variance, and the variational bound can become arbitrarily loose in some activation regimes.
Abstract
from arXiv · showhide
Neural networks can be compressed to reduce memory and computational requirements, or to increase accuracy by facilitating the use of a larger base architecture. In this paper we focus on pruning individual neurons, which can simultaneously trim model size, FLOPs, and run-time memory. To improve upon the performance of existing compression algorithms we utilize the information bottleneck principle instantiated via a tractable variational bound. Minimization of this information theoretic bound reduces the redundancy between adjacent layers by aggregating useful information into a subset of neurons that can be preserved. In contrast, the activations of disposable neurons are shut off via an attractive form of sparse regularization that emerges naturally from this framework, providing tangible advantages over traditional sparsity penalties without contributing additional tuning parameters to the energy landscape. We demonstrate state-of-the-art compression rates across an array of datasets and network architectures.
1. Introduction
The paper targets redundant neural-network capacity that increases computational and memory footprints, proposing an information-bottleneck approach to compress networks while preserving accuracy. Its variational formulation promotes sparse neuron usage and reports improved compression across standard benchmarks.
- Over-parameterized networks can preserve predictive performance while incurring unnecessarily large computational and memory footprints.
- The information bottleneck penalizes redundancy between adjacent network layers by squeezing input-related information irrelevant to label prediction.
- A variational bound aggregates useless information into expendable neurons whose activations can be shut off and pruned.
- The proposed compression energy requires only one tuning parameter for the compression–accuracy trade-off.
- Useful information accumulates in a sparse neuron subset while other activations move toward zero, offering advantages over traditional sparsity penalties.
- Experiments across common compression benchmarks report improvement over numerous state-of-the-art approaches.
2. Related Network Compression Work
Prior compression work uses architectural redesign, quantization, decompositions, connection pruning, Bayesian estimators, sparse priors, and deterministic regularizers. The paper emphasizes activation pruning because changing activation dimensionality directly supports computational and memory reductions.
- Compression methods redesign architectures, quantize weights, decompose tensors or matrices, or prune existing network structures.
- Removing small-magnitude connections often leaves neuron and activation dimensionality unchanged, limiting computation and memory gains without specialized processing.
- The paper targets activations for pruning as its primary focus to address practical efficiency limitations.
- Bayesian activation-pruning methods use sparse priors and optimize a variational free-energy approximation or bound.
- Deterministic alternatives apply group Lasso, ℓ1, or smoothed ℓ0 penalties to induce pruning effects.
3. Model Development
The model treats hidden layers as probabilistic transformations and applies a variational information bottleneck objective combining information compression with data fidelity. Learned Gaussian variances create an intrinsic mechanism for pruning neurons while retaining a tractable optimization problem.
- Hidden layers form successive representations in a Markov-chain view, extracting information from earlier layers while the output approximates p(y|hL).
- The objective minimizes inter-layer mutual information while maximizing information about the output label, balancing compression against prediction accuracy.
- A variational upper bound replaces the intractable information-bottleneck objective with a tractable surrogate optimized over network and distribution parameters.
- The loss combines a KL term measuring extracted information with a data-fit term reflecting fidelity to the data distribution.
- Gaussian conditional layers use learnable means and scales with injected Gaussian noise, reducing to a regular feed-forward network when µi = 1 and σi = 0.
- The learned variances ξi can collapse to zero, pressuring corresponding conditional coordinates toward a Dirac delta and effectively pruning those neurons.
- Gaussian assumptions yield a closed-form KL approximation, while removing ψi,j simplifies the model after its contribution proved unnecessary for excellent compression.
- Layer-specific γi values tailor compression strength, and the objective uses a weighting factor L to balance KL penalties with prediction accuracy in deeper networks.
4. Reduced Redundancy via Intrinsic Sparsity
VIBNet uses a variational information-bottleneck objective whose KL-based sparsity mechanism concentrates redundancy in neurons that can be pruned. The resulting pruning criterion is theoretically linked to zero mutual information and downstream ineffectiveness, while the objective’s practical justification remains partly conditional.
- The stochastic sampling scheme multiplies layer activations by noise parameterized by learned means and variances.
- The VIBNet objective combines a KL-based regularizer with a data-fit term over latent hidden states.
- The concavity of log(1 + u) makes the KL-derived regularizer favor pushing some neuron-related quantities exactly to zero rather than shrinking all of them partially.
- Proposition 1 links αi,j = 0 to zero or bounded mutual information, with exact zero mutual information requiring a local increase of the data term in σi,j.
- A neuron with zero mutual information about its preceding layer contains no information about the label and is redundant under the network’s Markov structure.
- At an objective minimum, zero outgoing weights require αi,j = 0, aligning the pruning indicator with neurons whose information cannot reach subsequent layers.
5. Analysis of Tractable Upper Bounds
The analysis uses tractable upper bounds to study VIBNet’s complex energy surface and explain its adaptive sparsification behavior. These bounds connect local curvature to regularizer shape, balancing sparse solutions against optimization risk.
- Tractable upper bounds: Tractable upper bounds make it possible to analyze VIBNet’s otherwise difficult deep-network energy surface and high-dimensional integrals.The bounds are inspired by sparse Bayesian methods and describe local neighborhoods of the objective.
- Tractable upper bounds: The bound is constructed from a positive semi-definite quadratic upper bound on the network loss with respect to stochastic latent activations.For fixed W = W′, the construction uses z_i(ϵ_i; θ_i) = µ_i + σ_i ⊙ ϵ_i and can be exact at some points under conditions such as Lipschitz-continuous gradients.
- Adaptive sparsification: VIBNet’s implicit sparsification adapts its shape to the local curvature of the network objective, approximating the ideal ℓ0 norm while reducing the risk of bad local minima.This adaptive effect contrasts with fixed deterministic sparsity penalties.
- Comparison with traditional penalties: The comparison frames weighted ℓ0 minimization as maximally sparse but NP-hard, while weighted ℓ1 regularization is convex yet can over-shrink large coefficients and reduce compressibility.VIBNet is presented as an adaptive alternative between these behaviors.
- Adaptive sparsification: When the data term is relatively smooth and unconstrained, small a_i,j values produce a smoother, flatter regularizer that avoids aggressive or premature sparsity dominance.Small a_i,j values make ω_i,j large.
- Adaptive sparsification: When the local region is highly curved and constrained, large a_i,j values produce an ℓ0-norm-like regularizer, allowing stronger sparsity pressure with limited risk of rapid descent to distant spurious optima.In this regime ω_i,j becomes small.
6. Experiments and Discussion
Across MNIST and CIFAR benchmarks, VIBNet achieves strong compression while maintaining comparable accuracy, and its information-bottleneck training reduces mutual information during compression.
- MNIST Results: VIBNet achieves the smallest FLOPs and rN on LeNet-5-Caffe, with the second-best rW and similar accuracy across methods.The comparison covers model size, floating-point operations, and run-time feature-map memory.
- CIFAR10 and CIFAR100 Results Using VGG-16: On CIFAR10 VGG-16 benchmarks, VIBNet produces the best compression by a significant margin while accuracy measures remain similar.Comparisons adopt the training protocols of the competing methods, including protocols involving fine-tuning and repeated pruning.
- Discussion: VIBNet provides a single compression-versus-accuracy control through γ, whereas some competing pruning procedures require multiple training and pruning iterations.Increasing γ can produce lower accuracy with higher compression.
- Redundancy Reduction Example: During training, VIBNet mutual information first increases while learning a predictive model and later decreases as network compression begins.The regular network’s mutual information increases while its compression ratio remains 1.
Appendix A. Derivation of the Variational Upper Bound (3)
The appendix derives a variational upper bound for the information bottleneck objective by rewriting its expectation and bounding the predictive term. The derivation identifies a KL-divergence component and applies Jensen’s inequality.
- The information bottleneck objective is rewritten as an expectation involving information compression and label-prediction terms.
- The derivation drops the constant p(y) log p(y) term and uses Jensen’s inequality to obtain an upper bound.
- The first expectation factor is a KL divergence between p(h_i|h_{i−1}) and q(h_i), evaluated analytically or stochastically.
- The predictive term is expanded using distributions over later hidden layers and the variational label distribution q(y|h_L).
- The resulting expression is identified as the final upper bound of L_i.
Appendix B. KL Term Derivation
The appendix derives the KL term by substituting the variational distributions into the objective and applying the closed-form Gaussian KL formula. It then solves the resulting stationarity condition algebraically.
- The KL term is obtained by plugging the variational distributions into the objective and applying the standard Gaussian KL-divergence formula.
- Taking the gradient with respect to the relevant parameters and setting it to zero yields a stationarity equation.
- Solving the stationarity equation produces the parameter expression used in the subsequent derivation.
- Substituting the solution back into the preceding equation gives the simplified KL-term result.
Appendix C. Proof of Proposition 1
The appendix proves that the objective favors solutions in which neurons carrying no information from the preceding layer have zero activation scale. It compares alternative parameterizations with unchanged data loss and reduced or equal KL cost.
- The objective is decomposed to analyze individual neuron information I(h_{i,j}; h_{i−1}) and the corresponding KL contribution.
- Jensen’s inequality and the appendix derivations establish the sufficient condition used for zero information.
- When σ^2_{i,j}=0, the neuron has p(h_{i,j}|h_{i−1})=p(h_{i,j}) and hence I(h_{i,j};h_{i−1})=0.
- When a neuron is independent of the preceding layer, its conditional function is constant and its marginal distribution is Gaussian.
- An alternative parameterization preserves p(h_{i+1}|h_{i−1}) and therefore leaves the data loss unchanged after integrating out h_i.
- Because the original parameters minimize the KL term, the alternative solution cannot have a lower KL cost than the original one.
Appendix D. Proof of Proposition 2
The appendix argues that at any minimum, a neuron with zero mutual information from the preceding layer must have α_{i,j}=0. This follows because the objective becomes independent of that neuron’s mean and variance parameters.
- After integrating out p(h_{i,j}|h_{i−1}), the objective is independent of μ_{i,j} and σ_{i,j}.
Appendix E. Proof of Proposition 3
The appendix derives the induced penalty on µ and uses its concavity and monotonicity to establish sparsity properties of local minima. It then shows that the relevant support relationship holds by analyzing limiting cases for µ and σ.
- Penalty derivation: Optimizing each σ_i,j separately yields a penalty ρ(µ_i,j; γ_i, a_i,j) after eliminating σ from the objective.The derivation defines a_i,j from the corresponding diagonal element of A⊤A and substitutes the unique optimal σ_i,j into the objective.
- Penalty derivation: The resulting penalty ρ is concave and non-decreasing in |µ_i,j| because γ_i/a_i,j is non-negative.This property follows from the cited concavity and monotonicity result for the auxiliary function used in the derivation.
- Reduced optimization: Any local or global minimizer of the original objective has µ* as at least a local minimum of the reduced objective and a local feasible solution to the associated constrained problem.The appendix transfers the minimizer analysis from the joint variables θ={µ,σ} to the reduced formulation.
- Sparsity bound: Local minima of the separable concave objective under linear constraints have at most rank[eA] nonzero elements.The appendix invokes Rao et al. (2003) for this sparsity property, implying a bound on the number of nonzero elements.
- Limiting cases: The support relation is completed by showing that µ_i,j=0 must hold when µ_i,j^2=0, yielding ∥µ*∥_0 ≥ ∥(µ*)^2 ⊙ (σ*)^-2∥_0.This is the final step of the appendix’s limiting-case argument.