Source-linked AI summary
Training Sparse Neural Networks
Suraj Srinivas, Akshayvarun Subramanya, R. Venkatesh Babu
TL;DR
Large neural networks use dense computations and are difficult to sparsify tractably with standard regularizers. The paper learns weights and gate variables for parameter selection, interprets the resulting objective through a spike-and-slab prior, and achieves state-of-the-art compression results across several architectures.
Problem
Dense large-scale vision networks use many parameters, while standard sparsity-inducing regularization is often insufficient for deep non-convex training.
Method
The method learns network weights together with gate variables for parameter selection, with the resulting regularization equivalent to a spike-and-slab prior.
Results
State-of-the-art compression results are achieved on LeNet-5, AlexNet, and VGG-16, with about 96% of LeNet-5 parameters removed at negligible accuracy loss.
Takeaways & Limitations
Learning weights and pruning simultaneously enables the method to obtain the optimal level of sparsity for deep-neural-network compression.
Takeaways & Limitations
Convolutional layers are less susceptible to compression than fully connected layers because they contain fewer parameters.
Abstract
from arXiv · showhide
Deep neural networks with lots of parameters are typically used for large-scale computer vision tasks such as image classification. This is a result of using dense matrix multiplications and convolutions. However, sparse computations are known to be much more efficient. In this work, we train and build neural networks which implicitly use sparse computations. We introduce additional gate variables to perform parameter selection and show that this is equivalent to using a spike-and-slab prior. We experimentally validate our method on both small and large networks and achieve state-of-the-art compression results for sparse neural network models.
Introduction
Large vision networks rely on dense computations, motivating sparse alternatives that reduce storage, evaluation time, and potentially overfitting. The paper targets tractable parameter-count regularization, validates its behavior experimentally, and reports state-of-the-art compression on several architectures.
- Motivation: Sparse computations can reduce parameter storage from O(mn) to O(k) and feedforward evaluation time from O(mnp) to O(kp).The comparison assumes a matrix with k non-zero elements and matrix-vector multiplication with a dense vector of size p.
- Problem: Restricting total parameter count requires a different regularizer from magnitude penalties such as ℓ2 or ℓ1.The paper notes that ℓ1 regularization is often insufficient for inducing sparsity in large non-convex deep-network training.
- Contribution: The paper proposes a novel regularizer that directly restricts the total number of network parameters.This is presented as the paper’s primary contribution in the introduction.
- Experiments: Experimental analysis examines the behavior of the proposed method.The experiments are identified as a dedicated contribution in Section 4.
- Results: Applications to LeNet-5, AlexNet, and VGG-16 achieve state-of-the-art network-compression results.The introduction reports this outcome across the three named architectures.
Problem Formulation
The paper formulates sparsification as jointly learning network weights and a binary parameter index set, then relaxes that set through trainable Bernoulli gate variables. Its objective combines task loss, a bimodal gate regularizer, and variance control, while admitting an equivalent spike-and-slab interpretation.
- Problem Formulation: Network complexity is defined as the total number of parameters represented by the layers’ binary index vectors.A zero indicates an absent parameter and one indicates a present parameter.
- Problem Formulation: The optimization jointly learns neural-network weights and a binary index set, making the problem difficult because index variables lie in {0, 1}.The index set permits direct penalization of the network’s total parameter count.
- Gate Variables: Pointwise multiplication of the index set with the weight matrix produces an effectively sparse matrix when the index set contains many zeros.The resulting sparse matrix is formed by multiplying the binary gate matrix with W.
- Gate Variables: Gate variables are real-valued Bernoulli parameters whose sampled binary matrices correspond exactly to the learned index set.This introduces weights and gates as two jointly learned variable sets.
- Promoting Sparsity: Deterministic thresholding at 0.5 is used instead of unbiased Bernoulli sampling, while a bimodal regularizer encourages gate values near zero and one.The overall gate regularizer combines the bimodal term with an ℓ1 or ℓ2 penalty on individual gates.
- An Alternate Interpretation: The objective adds a variance penalty for sampled gates, derived by approximating the stochastic loss expectation with a Monte-Carlo average.Reducing gate variance reduces the variance of the loss term and yields the stated objective when one evaluation is used.
- Relation to Spike-and-Slab priors: The formulation implicitly uses a spike-and-slab prior: its first term counts nonzero parameters, while its second term is an ℓ2 weight regularizer.Spike-and-slab priors mix a low-variance spike with a higher-variance slab to encourage sparsity.
- Estimating gradients for gate variables: The straight-through estimator backpropagates through Bernoulli gates as if sampling were an identity function, with clipping keeping gate parameters valid.The resulting sampling function is Bernoulli sampling from clipped gate values.
Related Work
Prior work compresses neural networks through pruning, reparameterization, factorization, quantization, or neuron removal. This paper instead learns weight pruning jointly with training and targets total network weights.
- Earlier pruning methods used hand-crafted rules or alternated between weight pruning and network training.
- The paper learns weight pruning during network training rather than applying hand-crafted pruning rules.
- Parameter-reduction methods predicted weights, reparameterized layers, factorized tensors, or quantized matrices to reduce storage and computation.
- Unlike Architecture Learning, which minimizes neurons, this work minimizes the total number of weights.
- Table 1 reports compression results for the LeNet-5 architecture.
Experiments
Experiments evaluate the method’s behavior, hyper-parameter sensitivity, initialization robustness, and compression across LeNet-5, AlexNet, and VGG-16. The method achieves high sparsity with similar accuracy and favorable compression results.
- Experimental setup: Experiments cover LeNet-5, AlexNet, and VGG-16, including behavior analysis and compression evaluation.LeNet-5 analysis uses MNIST, while AlexNet compression uses ILSVRC-2012; VGG-16 is also evaluated.
- Hyper-parameter effects: Both thresholding and sampling methods are similarly sensitive to λ1 and λ2.The experiments compare their accuracy and sparsity under different regularization-constant combinations.
- Method behavior: The thresholding method is deterministic, while thresholding and sampling produce almost equal sparsity and very similar accuracies.This determinism motivates the paper’s primary use of thresholding.
- Hyper-parameter effects: λ1 primarily controls gate variance and can stabilize training by reducing sparsity, whereas increasing λ2 increases sparsity.The observed variance and mean-sparsity patterns support these respective roles.
- Initialization: The method is robust to gate initialization, with thresholding consistently giving higher sparsities and no particular advantage from stochastic initialization.This reduces the need to identify good initial gate values.
- Practical considerations: Training uses twice the typical parameter count because weights and gates are both learned, but only their sparse element-wise product is needed at test time.For sparse storage, parameter indices are included in reported counts; ASIC implementations can encode indices in circuit structure.
Conclusion
The paper introduces a learning-based method that jointly learns neural-network weights and sparse connections. It reports state-of-the-art deep-network compression by learning sparsity while pruning.
- Conclusion: The method learns neural-network weights and performs pruning simultaneously.This learning-based approach targets the optimal level of sparsity.
- Conclusion: The approach achieves state-of-the-art results on compression of deep neural networks.