PaperlayerView exact revision on arXiv

Source-linked AI summary

Fast Sparse ConvNets

Erich Elsen, Marat Dukhan, Trevor Gale, Karen Simonyan

arXiv:1911.09723v1cs.CV

TL;DR

Efficient CNN inference remains difficult because theoretical sparsity reductions are not generally assumed to produce real hardware speedups. The paper introduces sparse kernels for ARM and WebAssembly and applies them to MobileNet and EfficientNet models. On Snapdragon 835, the resulting sparse networks outperform dense equivalents by 1.3–2.4× in wall-clock time at a given top-1 accuracy.

  • Problem

    Sparse models reduce theoretical FLOPs, but sparse operations are often disregarded as a practical way to accelerate inference.

  • Method

    The paper develops sparse matrix-dense matrix multiplication kernels and applies weight sparsity to MobileNet v1, MobileNet v2, and EfficientNet architectures.

  • Results

    1.3–2.4× wall-clock speedups at a given top-1 accuracy are achieved on Snapdragon 835, with sparse networks using approximately 66% as many parameters as dense equivalents.

  • Takeaways & Limitations

    Sparse networks provide approximately one entire generation of efficiency improvement across the evaluated MobileNet-family architectures.

  • Takeaways & Limitations

    The models are left unchanged and the same sparsity level is induced in all 1×1 convolutions.

Abstract

from arXiv · show

Historically, the pursuit of efficient inference has been one of the driving forces behind research into new deep learning architectures and building blocks. Some recent examples include: the squeeze-and-excitation module, depthwise separable convolutions in Xception, and the inverted bottleneck in MobileNet v2. Notably, in all of these cases, the resulting building blocks enabled not only higher efficiency, but also higher accuracy, and found wide adoption in the field. In this work, we further expand the arsenal of efficient building blocks for neural network architectures; but instead of combining standard primitives (such as convolution), we advocate for the replacement of these dense primitives with their sparse counterparts. While the idea of using sparsity to decrease the parameter count is not new, the conventional wisdom is that this reduction in theoretical FLOPs does not translate into real-world efficiency gains. We aim to correct this misconception by introducing a family of efficient sparse kernels for ARM and WebAssembly, which we open-source for the benefit of the community as part of the XNNPACK library. Equipped with our efficient implementation of sparse primitives, we show that sparse versions of MobileNet v1, MobileNet v2 and EfficientNet architectures substantially outperform strong dense baselines on the efficiency-accuracy curve. On Snapdragon 835 our sparse networks outperform their dense equivalents by $1.3-2.4\times$ -- equivalent to approximately one entire generation of MobileNet-family improvement. We hope that our findings will facilitate wider adoption of sparsity as a tool for creating efficient and accurate deep learning architectures.

1 Introduction

Efficient CNN research targets mobile inference, where 1×1 convolutions dominate cost. This work addresses skepticism about practical sparse acceleration with kernels for the neural-network sparsity range.

  • Mobile CNNs seek lower resource requirements while maintaining high ImageNet classification accuracy.
  • 1×1 convolutions dominate inference time, FLOPs, and parameter counts in these architectures.
  • 70–95% sparsity targets neural-network workloads, unlike prior sparse kernels designed mainly for problems exceeding 99% sparsity.
  • The proposed sparse matrix-dense matrix multiplication kernels outperform kernels generated by TACO and Intel MKL.
  • 1.3–2.4× faster inference and over 2× fewer parameters are reported for sparse models relative to previous generations.

2 Related Work

Prior work improves CNN efficiency through architectural changes, pruning, factorization, and neural architecture search, but theoretical and hardware efficiency do not necessarily align. The MobileNet family and sparse-weight approaches illustrate complementary design choices and deployment challenges.

  • Neural architecture search produced models exceeding 82% top-1 accuracy but requiring nearly 25 GFLOPs per inference.
  • Efficient CNNs are evaluated through inference speed, FLOPs reduction, and parameter or memory reduction, whose hardware effects can vary substantially.
  • MobileNet uses depthwise separable convolutions, while MobileNet v2 adds inverted residual blocks to improve parameter efficiency.
  • Direct weight pruning has struggled to accelerate 1×1 convolutions, encouraging channel-pruning methods that leave the final model dense.
  • ShuffleNet and butterfly factorizations reduce 1×1 convolution cost through structured matrix decompositions.
  • Prior TTS work linked greater sparsity and state size with improved quality at a fixed non-zero parameter count and developed fast block-sparse multiplication.

3 Methods

The paper trains sparse MobileNet and EfficientNet variants and implements ARM and WebAssembly kernels for sparse inference. Its design combines pruning, data-layout changes, vectorization, cache reuse, and structured sparsity.

  • Sparse MobileNet v1, MobileNet v2, and EfficientNet models are trained and evaluated alongside their sparse matrix-dense matrix multiplication kernels.
  • Gradual magnitude pruning creates sparse networks trained on ImageNet with standard augmentation and evaluated on its 50k-example validation set.
  • 3.1 Sparsifying Networks: The first full convolution remains dense because its parameter, FLOP, and runtime contributions are small.
  • 3.1 Sparsifying Networks: The sparse 1×1 convolution is implemented as SpMM, with unstructured sparsity or output-channel blocks representing alternative weight layouts.
  • 3.2 Kernel Implementation: Dense activations enable vector loads across spatial locations, while loop ordering and prefetching keep reused values in the L1 cache.
  • 3.2 Kernel Implementation: Output-channel blocking of sizes 2 and 4 increases register reuse without changing accuracy in the reported experiments.
  • 3.2 Kernel Implementation: ARM kernels use C with NEON intrinsics, and XNNPACK supplies the sparse kernels and supporting CHW-format operations.

4 Results

The experiments evaluate sparse kernels and sparse MobileNet/EfficientNet designs across accuracy, FLOPs, parameter count, and inference efficiency. Results show that sparsity level and block size shape efficiency-accuracy tradeoffs, while cache behavior constrains kernel performance.

  • 4.1 ARM Kernel Performance: 90% sparse MBv1, 85% sparse MBv2, and block-aware sparse models are compared with dense baselines using 224×224 inputs and millisecond timings.The table includes model sizes, FLOPs, inference speeds, sparsity settings, and block-size choices across ARM and WebAssembly measurements.
  • 4.1 ARM Kernel Performance: Sparse kernels achieve 2–7× effective FLOPs despite raw performance of 40–90% of dense kernels.The effective-FLOPs advantage accounts for the lower amount of work performed by sparse kernels.
  • 4.1 ARM Kernel Performance: MBv2 contract operations slow when expanded channels exceed L1-cache capacity, producing the observed sawtooth performance pattern.Expand kernels benefit from greater data reuse, whereas contract kernels lose performance beyond cache limits.
  • Training: Extending training and applying sparsification improves sparse-model performance without improving the dense baselines.MBv1 and MBv2 sparsification begins at iteration 28,000 and ends at 112,000, with pruning every 2,000 iterations.
  • 4.2 Model Performance: Block sparsity depends mainly on the number of elements per block, while its configuration has little effect on accuracy.For MBv1, 1 × 4, 4 × 1, and 2 × 2 block configurations produce overlapping curves.
  • 4.2 Model Performance: 70%, 80%, and 90% unstructured sparsity show that MBv1 and MBv2 become more efficient as sparsity increases.The models use constant sparsity throughout the network in this comparison.
  • 4.5 Wallclock Times: Sparse MBv1 is more FLOP- and parameter-efficient than MBv2, sparse MBv2 matches EfficientNet, and sparse EfficientNet exceeds the others in both categories.The comparison uses 90% sparsity for MobileNet v1 and 80% for the other models.
  • 4.3 Model Design for Block Size: A search over mixed unstructured and block-size-4 residual blocks selects models by the ratio of inference-time reduction to top-1 accuracy loss.The search leaves earlier blocks unstructured and applies block size 4 to later blocks under fixed sparsity assumptions.

5 Conclusion

The conclusion argues that sparse convolutional networks can be more accurate at a constant computational budget and that optimized sparse kernels make this practical. On Snapdragon 835, the sparse networks reduce wall-clock time by 1.3–2.4× at a given top-1 accuracy while using approximately 66% as many parameters.

  • Sparse convolutional networks are more accurate than dense ones at a constant computational budget.
  • Fast SpMM and supporting ARM kernels enable weight sparsity to accelerate state-of-the-art convolutional networks.
  • 1.3–2.4× lower wall-clock time and approximately 66% as many parameters are reported for sparse networks versus dense equivalents on Snapdragon 835 at a given top-1 accuracy.
  • The results challenge the misconception that sparsity is slow and motivate further research using sparsity in efficient architectures.

A Comparison with Intel MKL

The paper also evaluates an AVX512 implementation against Intel MKL. With minimal tuning, the sparse kernels are faster across both MobileNet v1 and v2 layers.

  • 1.2× geometric-mean speedup is achieved across all MBv1 and MBv2 layers versus Intel MKL.The comparison uses an AVX512-intrinsics implementation with minimal tuning.

B Non-Uniform Layerwise Sparsity with Variational Dropout

Variational Dropout reveals non-uniform layerwise sparsity patterns, but its parameter-constrained global pruning has practical and efficiency limitations. These patterns may still inform more efficient magnitude-pruned architectures.

  • Non-uniform sparsity: Variational Dropout induces sparsity globally, allowing non-uniform layerwise distributions without manual intervention.It is a Bayesian technique for inducing weight or activation sparsity.
  • Efficiency limitation: Global parameter-constrained pruning can produce more FLOPs than near-uniform pruning at the same parameter count.Spatially large layers contribute more FLOPs per removed parameter, while later layers often contain more parameters.
  • Practical limitation: VD cannot specify a desired final sparsity directly, requiring KL-penalty tuning after training and thresholding.Finding a model with the desired sparsity is therefore time consuming.
  • Observed patterns: Layerwise sparsity generally increases with depth as parameters per layer grow, while downsampling layers with channel increases remain less sparse than neighboring layers.This pattern is especially evident in MBv2 and EfficientNet.
  • Implication: The observed sparsity patterns did not yield more efficient architectures through hand tuning but may provide a useful prior for full architecture search.The authors examine these patterns despite VD's limitations to inform magnitude-pruning approaches.

C EfficientNet Plots

The EfficientNet plots examine how block size and sparsity affect efficiency. EfficientNet follows MobileNet-like behavior, but sparsity above 80% does not appear to produce more efficient models.

  • Observed behavior: EfficientNet generally follows the same efficiency trend as the MobileNet models in the main text.The comparison is presented through the EfficientNet plots.
  • Sparsity limit: Above 80% sparsity, EfficientNet does not seem to produce more efficient models.This is the reported boundary in the sparsity-level plots.
  • Measurement context: MBv1 and MBv2 achieved GFLOPs are plotted against increasing layer depth using measurements from an Intel Xeon W-2135.The figure provides the measurement context for the MobileNet comparison.
  • Layerwise pattern: MBv2 layerwise sparsity generally keeps expansion matrices less sparse than contraction matrices and early low-parameter layers less sparse than later layers.Layers that downsample spatial resolution while increasing channels are less sparse than otherwise expected.
  • Plot scope: EfficientNet behavior is evaluated across different block sizes and sparsity levels.Figure 11 contains the corresponding plots.
Loading 1911.09723v1…