Source-linked AI summary
Structured Pruning of Deep Convolutional Neural Networks
Sajid Anwar, Kyuyeon Hwang, Wonyong Sung
TL;DR
Real-time deep learning is hindered by computational complexity, memory accesses, and irregular pruning patterns that are difficult to exploit in parallel hardware. The paper introduces structured pruning with particle-filter candidate selection, retraining, and fixed-point optimization, reporting reduced computational complexity and storage requirements while maintaining performance in the described settings.
Problem
Irregular pruning reduces connections but requires extra sparse representations and does not fit well with parallel computation, limiting direct computational savings.
Method
The paper combines channel-wise, kernel-wise, and intra-kernel strided structured pruning with particle-filter candidate selection, retraining, and fixed-point optimization.
Results
The work reports that intra-kernel strided sparsity with convolution lowering significantly reduces convolution complexity, while 4 or 5 bit representations maintain the same performance and reduce storage.
Takeaways & Limitations
Structured sparsity and reduced word length support direct computational and storage savings for embedded, parallel, hardware-based, and on-chip implementations.
Abstract
from arXiv · showhide
Real time application of deep learning algorithms is often hindered by high computational complexity and frequent memory accesses. Network pruning is a promising technique to solve this problem. However, pruning usually results in irregular network connections that not only demand extra representation efforts but also do not fit well on parallel computation. We introduce structured sparsity at various scales for convolutional neural networks, which are channel wise, kernel wise and intra kernel strided sparsity. This structured sparsity is very advantageous for direct computational resource savings on embedded computers, parallel computing environments and hardware based systems. To decide the importance of network connections and paths, the proposed method uses a particle filtering approach. The importance weight of each particle is assigned by computing the misclassification rate with corresponding connectivity pattern. The pruned network is re-trained to compensate for the losses due to pruning. While implementing convolutions as matrix products, we particularly show that intra kernel strided sparsity with a simple constraint can significantly reduce the size of kernel and feature map matrices. The pruned network is finally fixed point optimized with reduced word length precision. This results in significant reduction in the total storage size providing advantages for on-chip memory based implementations of deep neural networks.
1 Introduction
The paper addresses the computational and memory costs of deep networks by proposing structured pruning at multiple granularities, combined with particle-filter candidate selection and fixed-point optimization. These constraints are intended to make sparsity directly exploitable by parallel and embedded hardware.
- Motivation: Irregular pruning can require sparse-location representations and is difficult to exploit efficiently, despite reducing parameters and multiply-accumulate operations.The paper therefore focuses on architecture, layer type, and connectivity patterns as determinants of computational complexity.
- Structured pruning: Structured sparsity constrains nonzero parameters to well-defined locations that CPUs and GPUs can exploit directly.The work examines channel-wise, kernel-wise, and intra-kernel strided sparsity.
- Structured pruning: Channel pruning removes all connections to and from feature maps, kernel pruning removes complete k×k kernels, and intra-kernel pruning removes weights within kernels.Kernel-level pruning is described as a special case of intra-kernel sparsity with 100% pruning.
- Candidate selection: Particle filtering searches for likely connection combinations by assigning particles to candidate masks and pruning less likely combinations.The pruned network is retrained to compensate for losses caused by pruning.
- Fixed-point optimization: Fixed-point optimization represents weights and signals with 4 or 5 bits while maintaining the same level of performance.The reduced word length further lowers storage requirements for on-chip implementations.
2 Convolutional Neural Network
CNNs combine convolutional, pooling, and fully connected layers. Convolutional and pooling layers extract features, while fully connected layers near the output perform classification.
- CNN architecture: CNNs contain convolutional, pooling, and fully connected layers, giving them more diverse layer types than standard deep neural networks.The paper presents CNNs in the context of pruning.
- CNN architecture: Convolutional and pooling layers act as feature extractors, while fully connected layers near the output classify the extracted features.The paper refers to a sample CNN network shown in Fig. 1.
3 Neural Networks Pruning
The paper introduces structured pruning at channel, kernel, and intra-kernel strided granularities, then uses evolutionary particle filtering to select pruning candidates. Strided sparsity constrains offsets and strides to reduce convolution matrix sizes while preserving hardware-friendly structure.
- Structured sparsity: Structured pruning places nonzero parameters at defined locations, allowing CPUs and GPUs to exploit computational savings without sparse-location representation.The paper contrasts this with unstructured pruning, which requires extra representation and addressing overhead.
- Channel and kernel pruning: Channel pruning removes feature maps and their incoming and outgoing kernels, reducing a 2-3-3 architecture to 2-2-3 in the example.This is the largest pruning granularity described.
- Intra-kernel strided pruning: Intra-kernel strided sparsity assigns each kernel an offset and stride, forcing weights into well-defined locations rather than randomly scattered positions.The method is intended to bridge pruning with efficient convolution computation.
- Channel and kernel pruning: Kernel pruning deletes whole convolution kernels, while kernel-level pruning is a special case of intra-kernel pruning with 100% sparsity.The paper distinguishes channel, kernel, and finer intra-kernel pruning levels.
- Intra-kernel strided pruning: Constraining outgoing connections to similar strides and offsets reduces the feature matrix from 9×27 to 9×15 and the kernel matrix from 27×2 to 15×2.The reduction is demonstrated when convolution is lowered to matrix products.
- Particle-filter candidate selection: Particle filtering searches connection combinations because pruning n connections creates O(2^n) possibilities, while particle weights reflect the performance of simulated masks.The proposed evolutionary particle filter combines particle filtering with a genetic algorithm to address particle degeneracy and impoverishment.
4 Network Retraining and Fixed Point Optimization
The workflow trains a baseline network, prunes connections, retrains the pruned network to recover performance, and then applies fixed-point optimization. Quantizing weights and signals to 4 or 5 bits maintains the same performance level while reducing storage requirements.
- Network retraining: The method trains a baseline network, prunes parameters, and retrains the pruned network to compensate for performance degradation caused by pruning.Pruning limits are set per layer according to parameter count and learning capacity.
- Fixed-point optimization: Fixed-point optimization quantizes the pruned network using L2 error minimization while retaining both high-precision and low-precision weights during optimization.The procedure supplements pruning by reducing parameter word length.
5 Experimental Results
Experiments on CIFAR-10 and MNIST show that structured pruning can substantially reduce network connectivity and size while maintaining performance close to larger or unpruned networks. Intra-kernel strided pruning and fixed-point optimization further improve computational and storage efficiency.
- CIFAR-10: Pruning a trained NW1 produced an NW2 network with about half the convolution connections while retaining performance quite close to the baseline.NW2 contains 1566/3168 convolution connections relative to NW1.
- CIFAR-10: 35% and 62% of convolution connections were dropped in two CIFAR-10 layers with less than 1% increase in MCR.Channel-wise pruning reached 50% sparsity; increasing it further raised MCR by more than 1%.
- CIFAR-10: 75% network-size reduction was achieved with minimum accuracy loss using 46% macro pruning followed by intra-kernel strided sparsity.The resulting structured sparsity can be directly encoded when convolutions are represented as matrix-matrix multiplication.
- CIFAR-10: Fixed-point optimization represented the pruned network with 3 or 4 bit precision, further reducing network size for on-chip memory implementations.Table 1 reports the fixed-point optimization results for the pruned CIFAR-10 network.
- CIFAR-10: Feature-map/channel pruning followed by intra-kernel pruning provided the best result among the tested pruning combinations.
- MNIST: The MNIST network reached 0.93% MCR after pruning to 1-20-20-20-20-500-10, reducing second-layer convolution connections by 60%.The reduction also shrank the first fully connected layer from 50(4x4)-500 to 20(4x4)-500.
5 Conclusion
The paper concludes that structured channel, kernel, and intra-kernel sparsity reduce convolutional complexity, while quantization complements pruning to reduce storage requirements.
- Channel-level sparsity has a limit because excessive pruning affects the network’s representational capacity.
- Intra-kernel strided sparsity combined with convolution lowering can significantly reduce convolutional computational complexity.
- Quantization and pruning are orthogonal techniques that can augment savings.