Source-linked AI summary
NISP: Pruning Networks using Neuron Importance Score Propagation
Ruichi Yu, Ang Li, Chun-Fu Chen, Jui-Hsin Lai, Vlad I. Morariu, Xintong Han, Mingfei Gao, Ching-Yung Lin, Larry S. Davis
TL;DR
Existing CNN pruning methods often consider only local layer statistics, overlooking error propagation across deep networks. NISP measures important final-response features, formulates pruning as binary integer optimization, and propagates importance throughout the network. Across multiple datasets and CNN architectures, it achieves full-network acceleration and compression with small accuracy losses, including 1.43% accuracy loss with 67.85% FLOP reduction on AlexNet.
Problem
Existing pruning methods often use individual-layer or consecutive-layer statistics, while deep-network error propagation motivates a unified network-wide pruning objective.
Method
NISP ranks final-response neurons, formulates pruning as binary integer optimization with a closed-form relaxed solution, propagates importance backward, prunes low-importance neurons, and fine-tunes the network.
Results
NISP achieves full-network acceleration and compression with small accuracy loss across evaluated CNNs, including 1.43% accuracy loss and 67.85% FLOP reduction on AlexNet.
Takeaways & Limitations
The framework provides generic, hardware-independent pruning that outputs a smaller network of the original model type while retaining predictive capability through fine-tuning.
Takeaways & Limitations
Selecting per-layer pruning ratios for a FLOPs budget remains a challenging open problem with a large search space, so the experiments use uniform or baseline-replicated ratios.
Abstract
from arXiv · showhide
To reduce the significant redundancy in deep Convolutional Neural Networks (CNNs), most existing methods prune neurons by only considering statistics of an individual layer or two consecutive layers (e.g., prune one layer to minimize the reconstruction error of the next layer), ignoring the effect of error propagation in deep networks. In contrast, we argue that it is essential to prune neurons in the entire neuron network jointly based on a unified goal: minimizing the reconstruction error of important responses in the "final response layer" (FRL), which is the second-to-last layer before classification, for a pruned network to retrain its predictive power. Specifically, we apply feature ranking techniques to measure the importance of each neuron in the FRL, and formulate network pruning as a binary integer optimization problem and derive a closed-form solution to it for pruning neurons in earlier layers. Based on our theoretical analysis, we propose the Neuron Importance Score Propagation (NISP) algorithm to propagate the importance scores of final responses to every neuron in the network. The CNN is pruned by removing neurons with least importance, and then fine-tuned to retain its predictive power. NISP is evaluated on several datasets with multiple CNN models and demonstrated to achieve significant acceleration and compression with negligible accuracy loss.
1. Introduction
NISP addresses error propagation in deep CNN pruning by measuring final-response importance and propagating it throughout the network. The method prunes low-importance neurons and fine-tunes the resulting model, achieving acceleration and compression with small accuracy losses.
- CNN pruning reduces redundancy, parameters, and computational cost, but many existing methods assess neurons using only one layer or two consecutive layers.
- NISP prunes lower-scoring neurons using pre-defined per-layer ratios and fine-tunes the pruned network to recover predictive accuracy.
- Greedy layer-by-layer pruning can propagate reconstruction errors through deep networks because early neurons may contribute substantially to important later responses.
- NISP defines early-neuron importance by minimizing reconstruction error for important responses in the final response layer, the second-to-last layer before classification.
- The method formulates pruning as binary integer optimization, derives a closed-form relaxed solution, and propagates importance scores from final responses to earlier neurons.
- 1.43% accuracy loss accompanies a 67.85% FLOP reduction on AlexNet, while ResNet-56 achieves almost zero accuracy loss with 43.61% FLOP reduction.
2. Related Work
Prior CNN compression methods use diverse parameter, filter, matrix, sparsity, and approximation strategies. NISP differs by measuring importance using network-wide information, input data, and propagation from the final response layer rather than greedy local pruning.
- Prior work reduces CNN cost through structured matrices, weight sparsity, quantization, Huffman coding, low-rank filters, channel scaling, and spatial operation skipping.
- Fully connected layers have been compressed by pruning similar neurons, reparameterizing matrix multiplication, random pruning, and hashing connection weights.
- Exact neuron-importance solutions are difficult because of neural-network nonlinearity, motivating approximations based on Taylor expansions or Lipschitz continuity.
- NISP differs from magnitude-based and greedy layer-by-layer pruning by incorporating input data, other network neurons, and importance propagation across the entire network.
3. Our Approach
NISP prunes a trained CNN by ranking final-response features, propagating their importance backward through the network, and removing low-importance neurons before fine-tuning. Its optimization minimizes the effect of pruning on important final responses while supporting general network structures and connections.
- 3. Our Approach: NISP first ranks neurons in the final response layer, whose responses directly serve the classification task, then propagates those scores throughout the network.The method can use feature-selection procedures such as Inf-FS to score final responses according to classification power.
- 3. Our Approach: The pruning objective minimizes weighted differences between original and pruned final responses, prioritizing responses with greater importance.This objective is formulated for a selected layer and accumulated over dataset samples.
- 3. Our Approach: The pruning problem is formulated as binary integer programming, then approximated by minimizing an analytically derived upper bound.The resulting solution retains the neurons with the highest inferred importance scores under the layer’s keep-count constraint.
- 3. Our Approach: A lower-layer neuron’s importance is computed recursively as a weighted sum of directly connected subsequent neurons’ importance scores.The weights are derived from absolute network weights and the propagation follows the network’s connectivity structure.
- 3. Our Approach: NISP performs this propagation in a single backward pass, accommodates normalization, pooling, and branch connections, and then prunes and fine-tunes the network.Pruning indicators retain neurons among the top N_l importance values in each layer; pruning ratios are application-specific hyperparameters.
4. Experiments
Experiments across datasets and CNN architectures compare NISP with random, scratch, magnitude-based, layer-by-layer, and existing pruning methods. NISP generally preserves accuracy while accelerating and compressing networks, with lower reconstruction error and faster convergence than key baselines.
- Experimental Setup: Experiments cover MNIST, CIFAR10, and ImageNet using five CNN architectures and multiple pruning baselines.The evaluation includes LeNet, Cifar-net3, AlexNet, GoogLeNet, and ResNet, with comparisons to random pruning, training from scratch, magnitude-based pruning, layer-by-layer pruning, and prior methods.
- Random Pruning and Training-from-scratch Baselines: At 50% pruning, NISP converges faster and retains more accuracy than random pruning and training-from-scratch baselines.For LeNet on MNIST, NISP loses only 0.02% top-1 accuracy; for Cifar-net on CIFAR10, the loss is below 1% at the same per-layer pruning ratio.
- Feature Selection v.s. Magnitude of Weights: NISP-FS yields smaller accuracy loss than NISP-Mag, while NISP-Mag still outperforms random pruning and training from scratch.This isolates the contribution of feature selection while retaining the broader importance-propagation procedure.
- NISP v.s. Layer-by-Layer Pruning: NISP achieves lower final accuracy loss than layer-by-layer pruning, whose reconstruction error increases sharply with network depth and pruning ratio.NISP consistently reduces WARE on retained final-response neurons, providing a better fine-tuning initialization.
- Comparison with Existing Methods: On AlexNet, NISP-A reduces 67.85% FLOPs with 1.43% accuracy loss, while NISP-C reduces 53.70% FLOPs with 0.54% loss.On ImageNet ResNet models, pruning 15% or 25% of filters per layer yields 27–44% FLOPs and parameter reduction with tiny top-1 accuracy loss.
- Additional Analysis: Selecting per-layer pruning ratios under a FLOPs budget remains a challenging open problem with a large search space.The experiments therefore use a single ratio across layers or replicate baseline ratios because of time limitations.
5. Conclusion
The paper presents NISP as a generic framework that propagates neuron importance throughout CNNs to guide pruning, acceleration, and compression while retaining predictive capability.
- NISP obtains importance scores in a layer of interest, usually before classification, through feature ranking.
- The pruning problem is formulated as a binary integer program with a closed-form solution for a relaxed version.
- NISP efficiently propagates final-response importance to neurons throughout the network.
- The network is pruned by removing less important neurons and then fine-tuned to retain predictive capability.
- Experiments show reduced CNN redundancy with full-network acceleration and compression.
6. Supplementary Material
The supplementary material describes how NISP propagates importance backward through network layers using weight-proportional mappings while simplifying the treatment of bias.
- NISP identifies the previous-layer positions feeding a neuron and propagates importance to them proportionally to the weights.Only selected feature extractors are propagated to previous layers; pruned ones are ignored.
- The method covers mappings between one-way tensors and between one-way and three-way tensors, including fully connected, convolutional, and pooling layers.
- NISP simplifies propagation by ignoring bias propagation.
- For a fully connected layer, input-neuron importance is computed from output importance and the element-wise absolute values of the layer weights.The formulation uses an M-by-N weight matrix for M input and N output neurons.
- For a fully connected layer receiving a three-way tensor, the input importance is represented as a flattened vector over the spatial positions and channels.The input tensor has size X × X × C and connects to N output neurons.
6.4. NISP: from 3-way tensor to 3-way tensor
For three-way tensor layers, NISP maps output-response importance back to input positions using propagation matrices shaped by layer connectivity, kernels, strides, and local operations.
- Convolutional layer: NISP derives convolutional-layer propagation as the most complicated three-way tensor case and provides an algorithm for it.The supplementary material labels this procedure Algorithm 1.
- Convolutional layer: Convolutional importance propagation decomposes between fixed input and output channels into propagation between two-dimensional feature maps.The example uses X = 4, Y = 2, k = 3, and s = 1.
- Convolutional layer: The convolutional propagation matrix maps flattened output-channel importance to input-layer importance using receptive-field structure and kernel weights.Its blocks are formed from row-mapping matrices whose nonzeros correspond to convolutional-kernel rows, with offsets determined by stride.
- Pooling layer: Pooling propagation maps output importance to input positions involved in each pooling operation, with structure determined by filter size and stride.Pooling matrices use normalized ones to distribute importance across covered input positions.
- Local Response Normalization layer: For cross-channel LRN, NISP approximates propagation by assuming kernel maps at one spatial position contribute equally to the output response.The method uses a symmetric matrix to map each output importance vector to the corresponding input vector, with local size l and N channels determining its structure.
6.6. PCA Accumulated Energy Analysis
PCA accumulated energy analysis is used to guide pruning-ratio selection by estimating how many principal components capture most response variance in a pre-pruned layer.
- PCA accumulated energy analysis indicates how many neurons or kernels may be retained by measuring the principal components needed to capture response variance.
- At a variance threshold of 0.95, LeNet requires 120 of 500 principal components to capture the variance.
- At a variance threshold of 0.95, AlexNet requires 2234 of 4096 principal components to capture the variance.
6.7. Experiments on AlexNet: Convolutional Layers v.s. FC Layers
The AlexNet experiments examine how pruning fully connected and convolutional layers affects learning behavior across feature-extractor levels.
- FC layers significantly influence accuracy loss, model size, and memory usage.
- The experiments prune half of the FC neurons together with selected convolutional layers.
- The five convolutional layers are grouped into low-level Conv1–Conv2, middle-level Conv3, and high-level Conv4–Conv5 feature extractors.
6.8. Experiments on GoogLeNet
On GoogLeNet, importance-based pruning is evaluated at a 50% pruning ratio while retaining reduction layers in inception modules, and it learns more effectively than the compared setting.
- At a 50% pruning ratio, all layers except reduction layers in inception modules are pruned.
- Importance-based pruning converges much faster and achieves the smallest accuracy loss.
- Figure 12 presents GoogLeNet learning curves on ImageNet for this pruning setup.
6.9. Layer-wise Improvements
The layer-wise analysis considers intermediate-layer computation and reports theoretical and empirical acceleration when half of convolutional kernels and fully connected neurons are pruned.
- Intermediate layers account for 17% of AlexNet’s empirical testing time and therefore must be included in acceleration design.
- Pruning reduces data volume or feature dimension at every layer, allowing all network layers to be sped up.
- With a 50% pruning ratio, convolutional layers after the first theoretically achieve 4× speedup, while intermediate layers achieve around 2×.
- For pooling, non-linearity, and normalization layers, theoretical and empirical speedups are almost the same.