PaperlayerView exact revision on arXiv

Source-linked AI summary

Learning Structured Sparsity in Deep Neural Networks

Wei Wen, Chunpeng Wu, Yandan Wang, Yiran Chen, Hai Li

arXiv:1608.03665v4cs.NEcs.LGstat.ML

TL;DR

Large DNNs are difficult to deploy because their computation and memory demands are high, while unstructured sparsity can limit practical acceleration. The paper proposes Structured Sparsity Learning, which uses group Lasso to learn compact structures across DNN components. SSL reports improved accuracy or no accuracy loss alongside substantial CPU and GPU speedups.

  • Problem

    Large DNNs are computation-intensive and memory-intensive, while non-structured sparsity can cause irregular memory access and limited practical acceleration.

  • Method

    SSL uses group Lasso regularization during training to learn compact filter, channel, filter-shape, and depth structures in DNNs.

  • Results

    SSL achieves average AlexNet layer-wise acceleration of 5.1× on CPU and 3.1× on GPU, while retaining or improving classification accuracy in reported experiments.

  • Takeaways & Limitations

    Structured compactness provides hardware-friendly DNN sparsity that accelerates evaluation with off-the-shelf CPU and GPU libraries.

Abstract

from arXiv · show

High demand for computation resources severely hinders deployment of large-scale Deep Neural Networks (DNN) in resource constrained devices. In this work, we propose a Structured Sparsity Learning (SSL) method to regularize the structures (i.e., filters, channels, filter shapes, and layer depth) of DNNs. SSL can: (1) learn a compact structure from a bigger DNN to reduce computation cost; (2) obtain a hardware-friendly structured sparsity of DNN to efficiently accelerate the DNNs evaluation. Experimental results show that SSL achieves on average 5.1x and 3.1x speedups of convolutional layer computation of AlexNet against CPU and GPU, respectively, with off-the-shelf libraries. These speedups are about twice speedups of non-structured sparsity; (3) regularize the DNN structure to improve classification accuracy. The results show that for CIFAR-10, regularization on layer depth can reduce 20 layers of a Deep Residual Network (ResNet) to 18 layers while improve the accuracy from 91.25% to 92.60%, which is still slightly higher than that of original ResNet with 32 layers. For AlexNet, structure regularization by SSL also reduces the error by around ~1%. Open source code is in https://github.com/wenwei202/caffe/tree/scnn

1 Introduction

Large DNNs are costly to deploy, while unstructured sparsity can produce poor hardware acceleration despite high sparsity. SSL learns compact, structured DNNs through group Lasso regularization and targets both accuracy and computation efficiency.

  • Motivation: High sparsity can still yield limited or negative practical speedups because scattered weights cause irregular memory access.The cited example controls accuracy loss within 2%, yet speedups remain poor even above 95% sparsity.
  • Motivation: Fixed layer structures can require repeated decomposition and fine-tuning to find accurate, efficient approximations.This motivates learning the compact structure directly rather than keeping layer structures fixed during fine-tuning.
  • Structured Sparsity Learning: SSL directly learns compressed filter, channel, shape, and depth structures during training using group Lasso regularization.The method combines structure regularization for classification with locality optimization for efficient memory access.

2 Related works

Prior compression methods reduce parameters or exploit low-rank structure, but often leave convolutional computation difficult to accelerate. SSL differs by applying group Lasso across multiple DNN structures to coordinate hardware-friendly sparsity.

  • Connection pruning: Connection pruning mainly accelerates fully connected layers, while convolutional layers remain difficult to speed up because of irregular sparsity.Convolution is identified as a computational bottleneck in newer DNNs with few fully connected parameters.
  • Hardware efficiency: Hardware and program customization can improve non-structured sparsity, but those optimizations are separate from SSL.SSL instead coordinates sparse weights in adjacent memory space for convolutional-layer acceleration.
  • Low-rank approximation: Low-rank approximation reports CPU and GPU speedups but leaves the compressed network structure fixed, requiring repeated optimization procedures.The cited methods use approximately 1% accuracy drop for their reported speedups.
  • Model structure learning: SSL applies group Lasso to filters, channels, filter shapes, and layer depth rather than a single structure.This multi-structure regularization distinguishes the method from the prior approaches discussed.

3 Structured Sparsity Learning Method for DNNs

SSL regularizes convolutional DNN structures during training by applying group Lasso to filters, channels, filter shapes, and layer depth. It can learn compact structures and variants designed for efficient convolution and GEMM computation.

  • Generic structured sparsity learning: SSL regularizes DNN structures by applying group Lasso during training to filters, channels, filter shapes, and layer depth.The method focuses mainly on convolutional layers and defines different groupings to induce each structured sparsity pattern.
  • Generic structured sparsity learning: Group Lasso can zero out entire weight groups, with the learned structure determined by how weights are partitioned into groups.Groups may overlap, allowing structured regularization to target different subsets of weights.
  • Filters and channels: Filter-wise and channel-wise regularization removes less important filters and channels while combining both patterns to account for dependencies between adjacent layers.A zeroed filter produces a zero output feature map, making the corresponding channel in the next layer useless.
  • Filter shapes: Shape-wise regularization learns arbitrary filter shapes by zeroing shape fibers corresponding to selected spatial positions.A shape fiber collects weights at one spatial position across filters for a channel, enabling homogeneous non-cubic shapes.
  • Layer depth: Depth-wise regularization removes unimportant layers to reduce computation and improve accuracy, using shortcuts to preserve feature-map propagation.The shortcut design allows feature maps to bypass an entire layer when SSL zeros all its filters.
  • Computationally efficient structures: Efficient-structure variants include 2D-filter-wise sparsity for convolution and combined filter-wise and shape-wise sparsity for GEMM.The combined GEMM formulation removes zero rows and columns from the lowered weight matrix, while 2D-filter removal saves convolution proportionally to the removed filters.

4 Experiments

Experiments across MNIST, CIFAR-10, and ImageNet evaluate SSL for learning structured sparsity in filters, channels, shapes, and depth. SSL reduces model computation while generally preserving or improving accuracy, with practical acceleration on AlexNet.

  • Experimental setup: SSL evaluations use published models on MNIST, CIFAR-10, and ImageNet, with speedups measured using Caffe matrix-matrix multiplication on a single-thread Intel Xeon E5-2630 CPU.The experiments initialize SSL from baseline network weights.
  • LeNet and MLP on MNIST: LeNet achieves similar error (±0.1%) with substantially fewer filters and channels, saving FLOP and computation time.The experiment penalizes unimportant filters and channels in convolutional layers.
  • LeNet and MLP on MNIST: SSL learns smoother LeNet filters, retaining five important stroke-pattern detectors while further removing a weak redundant detector lowers accuracy by only 0.2%.The regularized filters are described as more natural than the baseline’s random and blurry patterns.
  • ConvNet and ResNet on CIFAR-10: SSL reduces ConvNet 2 weight-matrix sizes by 50%, 70.7%, and 36.1% across convolutional layers while maintaining accuracy and achieving speedups.Half of ConvNet 2’s conv1 filters can be zeroed without accuracy drop.
  • ConvNet and ResNet on CIFAR-10: Depth regularization can improve classification accuracy, although test error generally declines as more layers are preserved and SSL-ResNet-20 shows a slight error rise over SSL-ResNet-18.The paper attributes the rise to suboptimal depth selection in the 32×32 layer group.
  • AlexNet on ImageNet: 30%–40% FLOP reduction is possible without accuracy loss, while retaining parameters reduces AlexNet error by approximately 1% to 41.69%.Shape-wise regularization also gives AlexNet 5 an average 1.4× layer-wise speedup on CPU and GPU without accuracy loss.
  • AlexNet on ImageNet: SSL averages 5.1× CPU and 3.1× GPU layer-wise acceleration, compared with 3.0× and 0.9× for ℓ1-norm sparsity.SSL maintains positive speedups and outperforms non-structured sparsity at comparable accuracy across reported platforms.

5 Conclusion

SSL regularizes multiple DNN structures to learn compact models without accuracy loss, while enabling evaluation speedups and potentially improving classification accuracy.

  • SSL regularizes filter, channel, filter-shape, and depth structures in DNNs.
  • SSL dynamically learns more compact DNN structures without accuracy loss.
  • Structured compactness produces significant DNN evaluation speedups on CPU and GPU with off-the-shelf libraries.
  • A variant of SSL can regularize DNN structures to improve classification accuracy.
Loading 1608.03665v4…