Source-linked AI summary
Data-free parameter pruning for Deep Neural Networks
Suraj Srinivas, R. Venkatesh Babu
TL;DR
The paper asks how to compress trained neural networks without relying on training data or changing their overall structure. It prunes redundant neurons by identifying similar weight-sets and merging their outgoing coefficients. The method removes parameters while retaining a structure suitable for fine-tuning, with reported compression results on MNIST and AlexNet.
Problem
Large neural networks have many parameters despite lower effective complexity, motivating compression methods that reduce storage and evaluation costs.
Method
The method finds similar or redundant weight-sets, estimates their output effect, and prunes neurons without using training or validation data.
Results
The method removes parameters from fully connected layers while preserving performance, including 21.3 million weights on AlexNet with only a small reported accuracy reduction.
Takeaways & Limitations
The approach can be applied to most network architectures containing fully connected layers while preserving the overall structure for fine-tuning.
Takeaways & Limitations
The complete procedure for choosing how many neurons to prune in large networks is not fully data-free because validation data selects among pruning levels.
Abstract
from arXiv · showhide
Deep Neural nets (NNs) with millions of parameters are at the heart of many state-of-the-art computer vision systems today. However, recent works have shown that much smaller models can achieve similar levels of performance. In this work, we address the problem of pruning parameters in a trained NN model. Instead of removing individual weights one at a time as done in previous works, we remove one neuron at a time. We show how similar neurons are redundant, and propose a systematic way to remove them. Our experiments in pruning the densely connected layers show that we can remove upto 85\% of the total parameters in an MNIST-trained network, and about 35\% for AlexNet without significantly affecting performance. Our method can be applied on top of most networks with a fully connected layer to give a smaller network.
1 Introduction
The introduction frames model compression as removing unnecessary complexity from large trained networks, contrasting weight pruning with data-driven and approximation-based alternatives. The proposed method requires no training or validation data and preserves the network’s overall structure for fine-tuning.
- Motivation: Large networks can contain far fewer effective degrees of freedom than parameters, motivating smaller models that are faster to evaluate and easier to store.These advantages are especially relevant for real-time and embedded applications.
- Prior approaches: Prior pruning methods remove unnecessary weights, ranging from naive magnitude-based removal to theoretically motivated but computationally intensive approaches such as OBS.This line of work operates on individual weights in trained models.
- Prior approaches: Knowledge Distillation and related methods train smaller networks to mimic larger ones, including shallower-wide and deeper-thinner architectures.FitNets apply distillation at several layers and report high compression on trained models.
- Prior approaches: Other compression methods use sparsity, parameter prediction, random connectivity, low-rank approximations, or product quantization to reduce model size.Approximation-based methods can alter network structure and make later fine-tuning difficult.
- Contribution: The proposed pruning method needs no training or validation data and only removes parameters, preserving the network’s overall structure for on-the-fly fine-tuning.It is positioned as a data-free alternative to structure-changing compression methods.
2 Wiring similar neurons together
The paper identifies equal weight vectors as a form of neuron redundancy: one of two neurons can be removed when its outgoing coefficient is added to the other’s. This extends the zero-weight intuition to redundant learned features.
- Redundancy: The motivating question is when weights can be removed from a neural network without affecting its accuracy, given that neural nets contain redundant parameters.The section focuses on redundancy arising from similar neurons rather than only zero-valued weights.
- Redundancy: Zero-valued weights can be removed without changing the network, motivating magnitude-based pruning.The paper then considers equal weight vectors as another redundancy pattern.
- Network formulation: For a single-hidden-layer network, each hidden neuron uses a weight-set vector including bias terms, and the output sums their activated responses weighted by scalar coefficients.The input is represented by X with the bias term absorbed, and h is applied to each weighted input.
- Neuron surgery: When two weight-sets are equal, one neuron can be removed by replacing its coefficient with the sum of the two coefficients.The paper calls this coefficient-merging operation the “surgery” step.
- Neuron surgery: Equal weight vectors therefore create redundant neurons whose outputs can be combined without changing the represented computation.The construction is related to the stated Hebbian intuition that neurons firing together can be wired together.
3 The case of dissimilar neurons
The method estimates which neuron removal changes network output least, using weight-set similarity and downstream coefficients, then iteratively removes neurons with a surgery update.
- 3 The case of dissimilar neurons: Neuron removal is analyzed by comparing similar weight sets and measuring the resulting squared output difference.Exact equality preserves the output after combining downstream coefficients; approximate similarity motivates an upper-bound analysis.
- 3 The case of dissimilar neurons: A monotonicity lemma for nonlinearities such as sigmoid and ReLU simplifies the bound on output change.The lemma is applied after computing the squared difference between the original and reduced outputs.
- 3 The case of dissimilar neurons: The pruning criterion selects the neuron pair minimizing an upper bound on expected squared output change.The expectation assumes inputs come from the training-data distribution, but the common input-norm factor need not be computed, making the criterion dataset independent.
- 3 The case of dissimilar neurons: The saliency criterion combines near-zero downstream coefficients with similar incoming weight sets rather than relying on either signal alone.The paper contrasts this with OBD, which removes individual weights using a Hessian-based training-error criterion.
- 3 The case of dissimilar neurons: For multiple output neurons, saliency averages the corresponding quantity across outputs, enabling pruning in intermediate layers.The procedure computes all pairwise saliencies, removes the neuron from the minimum-saliency pair, updates its partner coefficient, and updates the saliency matrix.
- 3 The case of dissimilar neurons: The saliency matrix is computed once before pruning, while only affected columns are updated after each neuron removal.This makes matrix initialization the most computationally intensive step of the algorithm.
2. Note that both contain a2
The paper refines neuron similarity through normalization and compares its data-free output-preservation objective with existing pruning and distillation approaches.
- 2. Note that both contain a2: OBD removes individual weights using a Hessian-based training-error criterion, whereas this method removes neuron-associated weight sets based on output-activation change.The approaches need not remove the same weights because training-error change is not generally proportional to output-activation change.
- 2. Note that both contain a2: Knowledge distillation minimizes squared differences between large- and small-network outputs over training data, while this method minimizes an upper bound without training data.The proposed minimization is restricted to output-layer neurons rather than all weights.
- 3.3 Weight normalization: Weight normalization identifies proportional weight sets as similar by scaling each set to unit norm and adjusting the corresponding next-layer coefficient.This addresses cases where similar features have different magnitudes.
- 3.4 Some heuristics: The paper uses heuristics to improve performance, including separate treatment of non-bias weights and balanced contributions from weights and biases.The stated motivation is that bias weights typically have much higher magnitude because they are not weight-decay regularized.
- 3.4 Some heuristics: The authors report that their new similarity measure performs much better empirically than using weight differences alone.They hypothesize that it may provide a tighter upper bound on expected output difference.
- 3.4 Some heuristics: The experiments consider fully connected layers, although similar heuristics could be defined for convolutional layers.The paper explicitly limits its treatment in this work to fully connected layers.
4 How many neurons to remove?
The method estimates how many neurons to remove using saliency, with the saliency distribution providing a data-free cutoff. This cutoff works best when baseline accuracy is high, while complete data-free selection for large networks remains unresolved.
- A saliency curve can serve as a proxy for the increase in test error, which rises rapidly near the curve’s exponential region.The method compares saliency during removal with the increase in test error from baseline levels.
- The histogram mode of saliency provides a data-free cutoff for selecting the number of neurons to remove.The cutoff uses the saliency value corresponding to the mode of the approximately Gaussian-like histogram.
- The histogram-based cutoff works well when baseline accuracy is high but can substantially decrease accuracy when baseline accuracy is low.
- A complete data-free method for choosing removals in large networks requires further investigation because the experiments use fractions of the predicted cutoff and validation data.
5 Experiments and Results
Experiments compress fully connected layers by pruning neurons in small and large networks, comparing the method with existing pruning approaches and evaluating cutoff selection. The results show substantial parameter removal with limited accuracy loss in the reported settings, while surgery is essential for the method’s performance.
- Experimental setup: Fully connected layers contain most parameters in large-scale networks, so pruning them can considerably compress the network.The experiments therefore focus on fully connected layers.
- 5.1 Comparison with OBS and OBD: In a SpamBase toy network, the method maintains low test error as more weights are removed, and surgery improves performance relative to OBD.
- 5.1 Comparison with OBS and OBD: Without surgery, the method breaks down completely, whereas OBS performs worse because it presumably removes important weights early.
- 5.1 Comparison with OBS and OBD: The method prunes many more weights than OBD and OBS at little or no increase in test error.
- 5.1 Comparison with OBS and OBD: < 0.1 seconds versus 7 minutes for OBD and > 5 hours for OBS shows substantially lower runtime in the toy comparison.
- 5.2 Experiments on LeNet: On MNIST, the data-free cutoff predicts removing 440 neurons, while the data-driven cutoff predicts 420 neurons for a 1% decrease in accuracy.
- 5.3 Experiments on AlexNet: In CaffeNet, removing 9.3 million parameters by pruning 700 FC6 neurons reduces base accuracy by 0.2%, while the best method removes up to 21.3 million weights.
6 Conclusion
The paper presents a data-free method for compressing neural networks by pruning neurons based on logit differences and saliency.
- The proposed method performs neural-network model compression without training data.
- Minimizing expected squared logit differences avoids using training data during compression.
- The method uses an exponentially increasing saliency curve to determine how many neurons to prune.
- The approach applies to most existing architectures that contain fully connected layers.
Appendix
The appendix proof uses monotonicity and positivity to transform an inequality by squaring both sides.
- Monotonic increase of h(·) is used in the proof of Lemma 1.
- Because both h(a)−h(b) and a−b are positive, the proof squares both sides of the inequality.
- The proof step relies on positivity of both differences before applying the squaring operation.