Faster CNNs with Direct Sparse Convolutions and Guided Pruning

Jongsoo Park; Sheng Li; Wei Wen; Ping Tak Peter Tang; Hai Li; Yiran Chen; Pradeep Dubey

Source-linked AI summary

Faster CNNs with Direct Sparse Convolutions and Guided Pruning

Jongsoo Park, Sheng Li, Wei Wen, Ping Tak Peter Tang, Hai Li, Yiran Chen, Pradeep Dubey

arXiv:1608.01409v5Aug 4, 2016cs.CV

TL;DR

The paper addresses the limited inference-speed gains of CNN pruning, which often reduces size without accelerating compute-heavy convolutions. It combines direct sparse convolution for arbitrary sparsity with a performance model and guided pruning, reporting 3.1×–7.3× AlexNet convolution speedups over dense methods without accuracy loss.

Problem
CNN pruning can reduce model size while preserving accuracy, but yields limited inference-speed improvement because convolutional layers dominate computation.
Method
The paper combines direct sparse convolution for arbitrary element-wise sparsity with a performance model and Guided Sparsity Learning for hardware- and layer-aware pruning.
Results
7.3×, 3.4×, and 3.1× convolution speedups over dense methods are achieved for AlexNet on Intel Atom, Xeon, and Knights Landing processors, respectively, with no accuracy drop.
Takeaways & Limitations
The combined approach realizes convolution speed improvements while supporting high sparsity and preserving accuracy in the reported AlexNet and GoogLeNet evaluations.
Takeaways & Limitations
Certain group-wise sparsity patterns require lowering, which can add considerable overhead and make ideal efficiency difficult to approach.

Abstract

from arXiv · show

Phenomenally successful in practical inference problems, convolutional neural networks (CNN) are widely deployed in mobile devices, data centers, and even supercomputers. The number of parameters needed in CNNs, however, are often large and undesirable. Consequently, various methods have been developed to prune a CNN once it is trained. Nevertheless, the resulting CNNs offer limited benefits. While pruning the fully connected layers reduces a CNN's size considerably, it does not improve inference speed noticeably as the compute heavy parts lie in convolutions. Pruning CNNs in a way that increase inference speed often imposes specific sparsity structures, thus limiting the achievable sparsity levels. We present a method to realize simultaneously size economy and speed improvement while pruning CNNs. Paramount to our success is an efficient general sparse-with-dense matrix multiplication implementation that is applicable to convolution of feature maps with kernels of arbitrary sparsity patterns. Complementing this, we developed a performance model that predicts sweet spots of sparsity levels for different layers and on different computer architectures. Together, these two allow us to demonstrate 3.1--7.3$\times$ convolution speedups over dense convolution in AlexNet, on Intel Atom, Xeon, and Xeon Phi processors, spanning the spectrum from mobile devices to supercomputers. We also open source our project at https://github.com/IntelLabs/SkimCaffe.

1 INTRODUCTION

The paper targets CNN pruning methods that reduce model size but deliver limited inference-speed gains, especially because convolutional layers dominate computation. It combines direct sparse convolution, a performance model, and Guided Sparsity Learning to pursue speed, size, and accuracy together.

Motivation: CNNs are widely deployed but often require many parameters, creating memory, energy, and inference-speed costs.AlexNet is cited as having 60 million parameters.
Motivation: Pruning substantially reduces CNN size while preserving accuracy, but inference-speed gains are much smaller.The paper identifies this mismatch as evidence that pruning benefits are not fully realized.
Motivation: Reducing fully connected layers cuts size, whereas convolutional layers dominate computation and therefore require fast sparse convolution for meaningful speedups.The paper distinguishes parameter-heavy fully connected layers from compute-heavy convolutional layers.
Method: Direct sparse convolution formulates convolution as sparse-matrix-dense-matrix multiplication, supports arbitrary element-wise sparsity, and generates dense-matrix columns on the fly.The design exploits data reuse across convolution channels and avoids imposing group-wise sparsity patterns.
Method: A roofline-based performance model predicts useful sparsity ranges across layers and architectures, including speed improvements around 70% sparsity.The model is used to guide sparsity targets and identify cases where layers should remain unpruned.
Method: Guided Sparsity Learning co-designs layer-wise pruning with hardware-aware speed potential to balance inference speed, accuracy, and model size.In GoogLeNet, the method prunes more than 80% of parameters in all 5×5/3×3 convolutional and fully connected layers with no accuracy drop.
Results: 7.3×, 3.4×, and 3.1× convolution speedups over dense methods are reported for AlexNet on Intel Atom, Xeon, and Knights Landing processors, respectively, with no accuracy drop.The implementation is open sourced as SkimCaffe.

2 GOING FASTER WITH DIRECT SPARSE CONVOLUTION, PERFORMANCE MODEL, AND GUIDED PRUNING

The paper combines direct sparse convolution, an architecture-aware performance model, and guided pruning to improve CNN inference speed while controlling model size and accuracy trade-offs.

Direct Sparse Convolution: Direct sparse convolution reformulates sparse CNN convolution as sparse-matrix-dense-matrix multiplication with input columns generated virtually on the fly.This supports kernels with arbitrary sparsity patterns and skips the input-lowering step used in standard frameworks.
Direct Sparse Convolution: CNN convolutions provide substantial data reuse, enabling optimized sparse computation through tiling, blocking, SIMDification, and register blocking.Sparse weight indices and values are reused across Hout·Wout output positions, while common tensor layouts can provide contiguous input access.
Performance Model: The performance model uses compute, activation, weight-storage, processor, bandwidth, and sparse-representation overheads to estimate dense and sparse convolution time.The model follows the roofline framework and captures architecture-dependent compute-to-bandwidth behavior.
Performance Model: 2–7× and 4–14× speedups are projected for AlexNet conv5 on Xeon and Atom, respectively, when filter density x is 0.05–0.15.This density range is described as typical without accuracy loss, and the projections are validated later in the paper.
Performance Model: Useful sparsity has both lower and upper bounds: insufficient sparsity leaves sparse convolution slower than dense convolution, while excessive sparsity becomes bandwidth bound.For AlexNet conv5, the Xeon upper-bound density is approximately x ∼0.02 and the Atom lower-bound density is approximately x ∼0.01; layer arithmetic intensity also changes the range.
Guided Pruning: Guided Sparsity Learning incorporates predicted speedup potential into pruning and can balance inference speed, accuracy, and model size through optional layer-specific constraints.Different regularization strengths for convolutional and fully connected layers can prioritize speed or size.

3 EXPERIMENTS

Experiments evaluate guided pruning and direct sparse convolution across ImageNet CNNs and three Intel platforms, showing layer-dependent sparsity benefits and substantial convolution speedups.

Experimental setup: The evaluation uses ImageNet models on Atom C2750, Xeon E5-2697 v4, and Xeon Phi 7250, with SGEMM as the dense-convolution proxy.Sparse matrices come from pruned AlexNet models with up to a 3% top-1 accuracy drop.
Guided training results: Guided ESL reduces AlexNet conv2-5 non-zero density below 0.2, targeting the model’s predicted speedup range.It also reduces regularization on fully connected layers and avoids pruning conv1.
Guided training results: GDNS reduces GoogLeNet non-zero density by 1.4× on average in layers with speedup potential compared with DNS.The method prioritizes those layers for pruning.
Guided training results: SSL’s constrained group-wise sparsity is insufficient to outperform guided element-wise sparse convolution under the authors’ performance model.This remains true even assuming ideal SSL efficiency and measured ESL overhead.
Guided training results: In GoogLeNet, guided pruning avoids 1×1 convolutions and concentrates sparsity in larger convolutions where the model predicts speedups.The authors report that larger convolutions contribute most FLOPs and often reach non-zero density below 0.2.
Layer-by-layer sparse convolution results: 3.4× speedup occurs on BDW at non-zero density 0.09 without accuracy drop in AlexNet conv2-5.Atom reaches 7.3× at the same density, while KNL reaches 3.1× over SGEMM.

4 RELATED WORK

Related work separates CNN compression into regularization-based pruning and factorization-based approaches, while differing in whether methods target convolution speed or model size.

Pruning methods: Regularization-based methods prune trained models using ridge, lasso, or group-lasso penalties with thresholding.Factorization-based methods instead approximate parameter tensors using low-rank decompositions.
Model size versus speed: Fully connected-layer pruning can reduce AlexNet model size by 10–50× but yields marginal inference speedup because those layers contribute less than 10% of modern CNN computation.Convolution-focused methods address the compute-heavy portion of inference.
Evaluation baselines: The paper recommends vendor-optimized GEMM FLOP/s as a baseline for comparing CNN speedup techniques.This avoids relying on a potentially suboptimal dense-convolution implementation.
Computing methods: Factorization-based methods naturally produce dense smaller models, whereas regularization-based convolution methods generally impose group-wise sparsity for dense computation.The paper’s approach instead computes arbitrary element-wise sparsity directly.
Complementary techniques: Direct sparse convolution can complement factorization because early convolution layers may not reach sufficient sparsity for sparse speedups.The paper also notes possible combinations with Winograd and FFT methods.

5 CONCLUSIONS

The paper combines direct sparse convolution with a performance model to guide architecture- and layer-specific pruning, jointly targeting model-size reduction and faster inference. On AlexNet, this approach achieves 3.1–7.3× convolution speedups versus extensively optimized dense operations, while sparse convolution does not accelerate every layer.

5 CONCLUSIONS: 3.1–7.3× convolution speedups were demonstrated in AlexNet across multiple platforms against extensively optimized dense linear algebra operations.The evaluated platforms span different computer architectures.
5 CONCLUSIONS: The method combines high-performance direct sparse convolution with a performance model that guides pruning across CNN layers and computer architectures.The approach co-designs pruning choices with the target architecture and layer.
5 CONCLUSIONS: The performance model reveals that sparse convolution cannot speed up all convolution layers, including 1×1 convolutions in GoogLeNet.The authors propose extending the model to other FLOP-reduction methods such as FFT, Winograd, and tensor factorization.