Source-linked AI summary
Exploring the Regularity of Sparse Structure in Convolutional Neural Networks
Huizi Mao, Song Han, Jeff Pool, Wenshuo Li, Xingyu Liu, Yu Wang, William J. Dally
TL;DR
Large DNNs are costly to store and execute, while pruning creates a trade-off between accuracy and sparse-structure regularity. The paper systematically compares pruning granularities and finds that coarse-grained sparsity can preserve accuracy while improving compression and hardware efficiency. Its experiments report lower storage and memory-reference costs for structured sparsity, with scope limited to convolutional-layer comparisons and fine-grained fully connected pruning.
Problem
DNN parameter, computation, and energy costs motivate studying how pruning regularity trades off against prediction accuracy and hardware efficiency.
Method
The paper systematically compares scalar-weight, vector, kernel, and filter pruning under consistent sparsity and training settings.
Results
Coarse-grained pruning achieves similar or better compression than fine-grained pruning and saves about 2× output memory access compared with fine-grained sparsity.
Takeaways & Limitations
Index savings and structured computation make coarse-grained sparsity desirable for mobile implementation and more efficient hardware architecture design.
Takeaways & Limitations
Overall sparsity is counted for convolutional layers, while fully connected layers use only fine-grained pruning because they lack the same granularity hierarchy.
Abstract
from arXiv · showhide
Sparsity helps reduce the computational complexity of deep neural networks by skipping zeros. Taking advantage of sparsity is listed as a high priority in next generation DNN accelerators such as TPU. The structure of sparsity, i.e., the granularity of pruning, affects the efficiency of hardware accelerator design as well as the prediction accuracy. Coarse-grained pruning creates regular sparsity patterns, making it more amenable for hardware acceleration but more challenging to maintain the same accuracy. In this paper we quantitatively measure the trade-off between sparsity regularity and prediction accuracy, providing insights in how to maintain accuracy while having more a more structured sparsity pattern. Our experimental results show that coarse-grained pruning can achieve a sparsity ratio similar to unstructured pruning without loss of accuracy. Moreover, due to the index saving effect, coarse-grained pruning is able to obtain a better compression ratio than fine-grained sparsity at the same accuracy threshold. Based on the recent sparse convolutional neural network accelerator (SCNN), our experiments further demonstrate that coarse-grained sparsity saves about 2x the memory references compared to fine-grained sparsity. Since memory reference is more than two orders of magnitude more expensive than arithmetic operations, the regularity of sparse structure leads to more efficient hardware design.
1 Introduction
The paper examines how pruning granularity trades off sparse-structure regularity, prediction accuracy, compression, and hardware efficiency. It evaluates granularities consistently to identify structured sparsity that preserves accuracy while improving implementation efficiency.
- Motivation: DNNs with hundreds of millions of parameters and tens of billions of operations are difficult to deploy on resource-limited embedded systems.The resulting challenges include storage, computation, and energy costs.
- Motivation: Pruning preserves accuracy and improves compression, but its sparse computation patterns become irregular.Structured pruning is more hardware-friendly, yet pruning entire filters can cause larger accuracy loss than pruning individual weights.
- Approach: The paper evaluates a consistent range of pruning granularities to measure the relationship between regularity, accuracy, storage savings, and hardware efficiency.The range includes intermediate grain sizes such as 2D kernels and 1D sub-kernel vectors.
- Contributions: Coarse-grained pruning reaches similar or better compression rates than fine-grained pruning despite producing less sparsity.The paper attributes this advantage to structured computation and index savings.
- Contributions: Coarse-grained sparsity skips computations and reduces memory references in a structured manner, supporting more efficient accelerator implementation.This is presented as a hardware-design advantage of regular sparse structure.
2 Related Works
Prior work spans fine-grained weight pruning and very coarse channel or filter reduction, with a broad unexplored range of intermediate granularities. Hardware acceleration strategies also differ between coarse structured sparsity and fine-grained sparsity.
- Granularity of sparsity: Fine-grained sparsity deletes individual weights, while channel or filter reduction removes higher-dimensional structures.Fine-grained sparsity generally preserves accuracy better, whereas coarse-grained sparsity benefits acceleration through regularity but can reduce accuracy.
- Granularity of sparsity: A large range of pruning granularities lies between vanilla sparsity and channel reduction.Existing studies often explore only one or a few possibilities within this range.
- Granularity of sparsity: Intra-kernel strided pruning is one example of an intermediate granularity explored in prior work.It represents only one case among the available grain sizes.
- Accelerating sparse models: Very coarse filter- or channel-sparsity can accelerate general-purpose processors by yielding a smaller dense model.Fine-grained sparsity instead requires custom accelerators to exploit computation reduction.
3 Granularity of Sparsity
The paper organizes convolutional pruning by the dimensionality of the unit removed, spanning scalar weights through filters. It uses magnitude-based, iterative pruning to compare these granularities and their computation mappings.
- Notations: A convolutional layer’s weights form a C × K × R × S tensor, where C and K index output and input dimensions and R × S is the kernel shape.For a 3 × 3 kernel, R=3 and S=3.
- Range of Granularity: Convolutional weights can be grouped as filters, channels, kernels, sub-kernel vectors, or scalar weights.These correspond to 3D, 3D, 2D, 1D, and 0D structures respectively.
- Range of Granularity: The study compares four pruning grains: scalar weights (0D), sub-kernel vectors (1D), kernels (2D), and filters (3D).The grain size increases with the number of tensor dimensions.
- Range of Granularity: The four granularities map to computation units ranging from fine-grained multiply-accumulate operations to 1D and 2D convolutions.Fine-grained sparsity is implemented in systems such as EIE and SCNN, while kernel sparsity corresponds to 2D convolution.
- Coarse-grained Pruning Method: For a grain Gi, salience is computed as the sum of its weights’ absolute values, corresponding to the L1 norm.At a target sparsity, grains are ranked by this score and the lowest-scoring grains are deleted.
- Coarse-grained Pruning Method: Iterative pruning determines sparsity at each stage through sensitivity analysis and can reach higher sparsity than direct pruning.The analysis measures accuracy loss after individually pruning each layer.
4 Sparsity-Accuracy Relation with Different Grain Sizes
The paper compares pruning granularities under matched experimental settings to quantify their accuracy–sparsity trade-off. Larger grains can substantially hurt accuracy, while finer grains produce accuracy–sparsity behavior closer to fine-grained pruning.
- Coarse-grained sparsity constrains parameter positions as well as parameter count, making it a stronger regularizer than sparsity alone.The paper reports accuracy improvement at low sparsity rates in Figure 4 under this effect.
- Experiments use matched sparsity settings and training schedules, with AlexNet curves and additional comparisons on VGG-16, GoogLeNet, ResNet-50, and DenseNet-121.All experiments were performed on ImageNet with Caffe.
- Nearly 1% validation accuracy is lost by AlexNet at the first pruning stage when filters are pruned iteratively.The paper reports much smaller losses for finer-grained pruning, including small early accuracy increases.
- Smaller pruning grains produce accuracy–sparsity curves similar to fine-grained pruning, whereas filter pruning greatly hurts accuracy.Some curves rise smoothly during initial pruning stages, suggesting coarse-grained pruning can still reach similar compression rates.
5 Comparison of Storage
The storage comparison evaluates pruning granularities using quantized sparse models and highlights the storage benefit of sharing indices across larger grains. Larger grains generally require less storage, although the benefit depends on convolutional-kernel size.
- Coarse-grained sparsity saves indices through regularity, so models use less storage than fine-grained models at the same sparsity.Sparse models store both values and indices, making index sharing a direct source of storage reduction.
- The storage study uses 8-bit weights and 4-bit indices for distances between adjacent nonzeros.The comparison targets accuracy at matched storage rather than matched sparsity.
- Figure 7 shows closer accuracy–storage curves for fine-grained, vector, and kernel pruning than the corresponding accuracy–sparsity curves.The paper identifies this convergence as evidence of the index-saving effect.
- Storage ratios are estimated at target accuracies by linear interpolation because exact baseline-accuracy matches are generally unavailable.The ratio compares pruned 8-bit models with 4-bit indices against dense 8-bit models.
- Larger grains generally require less storage because kernel weights can share indices.This relationship is weaker for networks dominated by 1x1 convolutions, which provide little index-sharing benefit.
6 Regular Sparsity Helps Hardware Implementation
The paper examines how intermediate and coarse sparsity granularities affect customized hardware implementation. Structured sparsity can simplify computation and reduce memory references through locality and output-address reuse.
- Hardware implications: Intermediate grain sizes such as kernels and sub-kernel vectors offer hardware advantages despite limited acceleration on general-purpose processors.These granularities can enable simpler circuit design and higher energy efficiency on customized hardware.
- Hardware implications: Kernel and sub-kernel-vector pruning preserve low-level computation logic, simplifying customized hardware design.Kernel pruning suits 2-D convolution, while sub-kernel-vector pruning benefits designs using 1-D convolution.
- Sparse dataflow: Weights and input activations are stored sparsely, while output activations use dense storage and scatter-add accumulation based on computed output addresses.After each layer, output activations are compressed into sparse format for the next layer.
- Memory references: Repeated output-address references can bypass expensive reads and writes, and coarse-grained locality increases the probability of such collisions.The paper compares output memory references for VGG-16 convolutional layers in Table 3.
7 Summary
The authors compare their pruning results with prior work using AlexNet and a common evaluation framework. Their fine-grained model improves several efficiency measures, while vector pruning further reduces convolutional storage and memory references.
- Comparison setup: The comparison uses AlexNet because its layer-wise sparsity is reported in prior pruning studies.A smoother pruning process further improves the results of Han et al. without changing the algorithm.
- Comparison setup: Table 4 reports parameters, FLOPs, storage, convolutional-layer storage, and simulated memory references for AlexNet.Storage uses 8-bit weights and 4-bit indices, while the memory-reference baseline is a dense model with sparse activations.
- Results: The fine-grained pruned model outperforms prior fine-grained work in FLOPs, convolutional-layer storage, and memory references.The comparison is presented alongside previous pruning results in Table 4.
- Results: 23% lower convolutional-layer storage and 43% fewer memory references are achieved by vector pruning relative to the fine-grained baseline.These reductions are reported as additional improvements over the authors’ fine-grained model.
8 Conclusion
The paper evaluates sparsity granularity through accuracy-density experiments and hardware analysis. Coarse-grained sparsity improves compression and reduces output memory access, supporting simpler and more efficient accelerator designs.
- Conclusion: The experiments thoroughly characterize the relationship between sparsity granularity, accuracy, and density.The analysis covers a range of sparsity granularities rather than only extreme pruning cases.
- Conclusion: Index savings allow coarse-grained pruning to achieve a higher model compression ratio despite its structured sparsity.The paper identifies this property as desirable for mobile implementation.
- Conclusion: ∼2× fewer output memory accesses than fine-grained sparsity and ∼3× fewer than dense implementation are obtained with coarse-grained sparsity.The reported hardware benefit is measured for output memory access.
- Conclusion: Coarse-grained sparsity enables more efficient neural-network hardware architecture design through hardware simplicity and efficiency.This conclusion follows the paper’s hardware analysis of structured sparsity.