PaperlayerView exact revision on arXiv

Source-linked AI summary

Pruning Filters for Efficient ConvNets

Hao Li, Asim Kadav, Igor Durdanovic, Hanan Samet, Hans Peter Graf

arXiv:1608.08710v3cs.CVcs.LG

TL;DR

CNN inference costs matter where computational and power resources are limited, while sparse-operation libraries are often limited. The paper prunes low-magnitude filters and their feature maps, reporting about 30% FLOP reduction for VGGNet on CIFAR-10 and deep ResNets without significant loss in original accuracy.

  • Problem

    CNN inference costs matter where computational and power resources are limited, while sparse-operation libraries are often limited.

    Source: §1 · pp. 1–2

  • Method

    The method prunes low-magnitude filters and their corresponding feature maps, then retrains the network using a one-shot strategy.

    Source: §1 · p. 2, §3 · p. 3, §2 · p. 3

  • Results

    About 30% reduction in FLOP for VGGNet on CIFAR-10 and deep ResNets was achieved without significant loss in original accuracy.

    Source: §5 · p. 10

  • Takeaways & Limitations

    Pruning-sensitivity studies identify robust and sensitive layers, which may inform understanding and improvement of very deep CNN architectures.

    Source: §5 · p. 10

    Sparse-operation acceleration libraries are often limited, and sparse data structures can add significant storage overhead for low-precision weights.

    Source: §1 · p. 2

Abstract

from arXiv · show

The success of CNNs in various applications is accompanied by a significant increase in the computation and parameter storage costs. Recent efforts toward reducing these overheads involve pruning and compressing the weights of various layers without hurting original accuracy. However, magnitude-based pruning of weights reduces a significant number of parameters from the fully connected layers and may not adequately reduce the computation costs in the convolutional layers due to irregular sparsity in the pruned networks. We present an acceleration method for CNNs, where we prune filters from CNNs that are identified as having a small effect on the output accuracy. By removing whole filters in the network together with their connecting feature maps, the computation costs are reduced significantly. In contrast to pruning weights, this approach does not result in sparse connectivity patterns. Hence, it does not need the support of sparse convolution libraries and can work with existing efficient BLAS libraries for dense matrix multiplications. We show that even simple filter pruning techniques can reduce inference costs for VGG-16 by up to 34% and ResNet-110 by up to 38% on CIFAR10 while regaining close to the original accuracy by retraining the networks.

1 INTRODUCTION

This work prunes redundant CNN filters to reduce computation without introducing sparse connectivity, using one-shot pruning and retraining. The introduction reports about 30% FLOP reduction in ResNets without sacrificing too much accuracy.

1 INTRODUCTION

  • Magnitude-based weight pruning can remove many parameters yet yield little computation savings: VGG-16 fully connected layers hold 90% of parameters but contribute less than 1% of FLOP.Accelerating convolutional layers through sparse operations may require sparse BLAS libraries or specialized hardware; sparse libraries are limited, and sparse data structures add storage overhead, especially for low-precision weights.Source: §1 · pp. 1–2
  • As CNNs deepen, convolutional computation remains dominant despite architectures that reduce parameters with average pooling and computation by early image downsampling.The introduction motivates efficiency through limited compute and power on mobile or embedded devices and the inference-time demands of high-volume image services.Source: §1 · pp. 1–2
  • ResNets retain about 30% FLOP reduction without sacrificing too much accuracy, despite their lower parameter counts and inference costs than AlexNet or VGGNet.The authors also report sensitivity analysis of ResNet convolutional layers to improve understanding of these networks.Source: §1 · p. 2
  • Filter pruning removes whole filters and feature maps, avoiding sparse connectivity while directly reducing matrix multiplications and enabling speedup to be tuned.The authors use one-shot pruning and retraining across multiple layers instead of layer-wise iterative fine-tuning, saving retraining time for very deep networks.Source: §1 · p. 2

2 RELATED WORK

The related work spans weight and feature-map pruning, convolutional acceleration, and structured-sparsity training. This paper distinguishes its filter pruning by using simple ℓ1-based selection and conventional fine-tuning without additional regularization.

Related work

  • Le Cun et al. (1989) and Hassibi & Stork (1993) prune weights using saliency and second-order information, while Mariet & Sra (2016) select diverse fully connected neurons without retraining, yielding sparse connections.Mariet & Sra's method is limited to fully connected layers.Source: §2 · p. 2
  • Prior work reduces convolution cost via low-rank approximations, FFT- or Winograd-based convolution, quantization, and binarization; the proposed pruning can complement these techniques.Low-rank approaches include Denil et al. (2013), Jaderberg et al. (2014), Zhang et al. (2015b;a), Tai et al. (2016), and Ioannou et al. (2016); FFT and Winograd approaches include Mathieu et al. (2013) and Lavin & Gray (2016), while quantization and binarization include Han et al. (2016b), Rastegari et al. (2016), and Courbariaux & Bengio (2016).Source: §2 · p. 2
  • Anwar et al. (2015) use particle filtering to select weight-pruning masks, while Polyak & Wolf (2015) identify rarely activated feature maps for face detection; this work prunes filters and corresponding maps by magnitude.The proposed approach also introduces network-wide holistic pruning for simple and complex convolutional architectures.Source: §2 · p. 2
  • Concurrent compact-CNN methods use ℓ2,1-norm regularization for filter-level sparsity: Lebedev & Lempitsky (2016) apply group-sparsity to convolution kernels, while Zhou et al. (2016) and Wen et al. (2016) regularize neurons or layers.Wen et al. also target trivial filters and channels.Source: §2 · p. 2
  • Unlike regularized approaches, this method selects and physically removes filters with ℓ1-norm, fine-tunes conventionally, and uses pruning percentages—set stage-wise—as its only pruning hyperparameter.A single pruning rate can be set for every layer in a stage, and it corresponds directly to the desired speedup.Source: §2 · p. 3

3 PRUNING FILTERS AND FEATURE MAPS

The method removes entire convolutional filters and their feature maps, reducing computation in both the pruned layer and the next layer while retaining dense connectivity. It selects filters by weight magnitude, adapts pruning to layer sensitivity and network structure, and retrains pruned networks to recover accuracy.

3 PRUNING FILTERS AND FEATURE MAPS

  • Pruning a filter removes its output feature map and the next layer’s kernels connected to that map, reducing computation in both layers.Pruning m filters removes m/ni+1 of the computation cost for layers i and i + 1.Source: §3 · p. 3, Figure 1 · p. 3

3.1 DETERMINING WHICH FILTERS TO PRUNE WITHIN A SINGLE LAYER

  • The method ranks filters by their ℓ1-norm and prunes those with the smallest absolute-weight sums, using a data-free selection criterion.The authors report no noticeable difference between ℓ1- and ℓ2-norm selection; the procedure also removes corresponding feature maps and next-layer kernels.Source: §3.1 · pp. 3–4

3.2 DETERMINING SINGLE LAYER’S SENSITIVITY TO PRUNING

  • Layer sensitivity is assessed by pruning each layer independently; layers in the same stage tend to have similar sensitivity, so they share a pruning ratio, while sensitive layers are pruned less or skipped.Layers with relatively flat filter-weight distributions are more sensitive to pruning.Source: §3.2 · p. 4, Figure 2 · p. 4

3.3 PRUNING FILTERS ACROSS MULTIPLE LAYERS

  • Greedy multi-layer pruning accounts for filters removed in earlier layers and, though not globally optimal, yields higher accuracy than independent pruning, especially when pruning many filters.Independent pruning selects filters at each layer without accounting for earlier removals.Source: §3.3 · p. 5, Figure 3 · p. 5
  • In ResNets, pruning a residual block’s second convolution requires pruning the corresponding projected feature maps, while its first convolution can be pruned without that restriction.The second-layer filter indices are determined by pruning the shortcut projection, using the same selection criterion on its 1 × 1 filters.Source: §3.3 · p. 5, Figure 4 · p. 5

3.4 RETRAINING PRUNED NETWORKS TO REGAIN ACCURACY

  • Pruning and retraining once can recover accuracy after removing substantial portions of pruning-resilient layers, but sensitive-layer or large-scale pruning may resist recovery.Iterative pruning and retraining may improve recovery, but requires many more epochs, especially for very deep networks.Source: §3.4 · p. 6

4 EXPERIMENTS

Experiments evaluate filter pruning on VGG-16 and residual networks across CIFAR-10 and ImageNet, then compare filter-selection criteria. They report FLOP reductions and accuracy effects, identify layers sensitive to pruning, and assess alternatives to ℓ1-norm filter selection.

4 EXPERIMENTS

  • The experiments train baseline networks from scratch and retrain pruned models for 40 epochs on CIFAR-10 or 20 epochs on ImageNet at a constant 0.001 learning rate.The evaluated models include VGG-16 and ResNet-56/110 on CIFAR-10 and ResNet-34 on ImageNet; pruning creates a smaller model and removes corresponding batch-normalization weights.Source: §4 · p. 6

4.1 VGG-16 ON CIFAR-10

  • 34% FLOP reduction is achieved at the same accuracy by pruning 50% of filters in VGG-16 layer 1 and layers 8–13.With retraining, layers with 512 feature maps can lose almost 90% of their filters; the authors suggest their 4 × 4 or 2 × 2 feature maps may have no meaningful spatial connections. The first layer is comparatively robust on CIFAR-10, while pruning the second layer heavily can hurt accuracy.Source: §4.1 · p. 7

4.2 RESNET-56/110 ON CIFAR-10

  • ResNet-56/110 layers sensitive to pruning lie near feature-map-size transitions, and skipping them improves retraining performance.ResNet-56-pruned-B skips layers 16, 18, 20, 34, 38, and 54, pruning stages at 60%, 30%, and 10%; ResNet-110-pruned-B skips layers 36, 38, and 74, with stage rates of 50%, 40%, and 30%. The authors suggest ResNet-110 is easier to prune because its middle residual blocks may be redundant.Source: §4.2 · p. 8

4.3 RESNET-34 ON ILSVRC2012

  • ResNet-34 option B prunes first layers in stages one through three at 50%, 60%, and 40%, reducing FLOPs 24% with about 1% accuracy loss; with retraining, pruning stage three at 20% reduces FLOPs 7.5% with 0.75% accuracy loss.The first and last residual blocks of each stage are more sensitive to pruning than intermediate blocks. Pruning the first layer is more effective at reducing overall FLOPs than pruning the second layer.Source: §4.3 · pp. 8–9

4.4 COMPARISON WITH PRUNING RANDOM FILTERS AND LARGEST FILTERS

  • Pruning the smallest filters outperforms random-filter pruning for most layers and pruning ratios, while removing the largest ℓ1-norm filters causes accuracy to drop quickly as the ratio rises.At a 90% pruning ratio, smallest-filter pruning has better accuracy than random-filter pruning for all layers.Source: §4.4 · p. 9

4.5 COMPARISON WITH ACTIVATION-BASED FEATURE MAP PRUNING

  • Smallest-filter pruning outperforms four activation-based criteria, while σmean-std matches or exceeds ℓ1-norm performance only up to a 60% pruning ratio.σmean-std performance drops quickly beyond 60%, especially for conv 1, conv 2, and conv 3; the authors describe ℓ1-norm as a data-free heuristic. Activation statistics are computed from the whole CIFAR-10 training set (N = 50,000).Source: §4.5 · p. 10

5 CONCLUSIONS

The paper presents low-magnitude filter pruning to reduce CNN computation without introducing irregular sparsity, and uses lesion studies to identify layers that are robust or sensitive to pruning.

5 CONCLUSIONS

  • About 30% FLOP reduction was achieved for VGGNet on CIFAR-10 and deep ResNets without significant loss in original accuracy.Source: §5 · p. 10
  • Pruning relatively low-magnitude filters reduces computation costs without introducing irregular sparsity.Source: §5 · p. 10
  • The method uses one-shot pruning and retraining rather than layer-wise hyperparameters and time-consuming iterative retraining.Source: §5 · p. 10
  • Lesion studies on very deep CNNs identify layers that are robust or sensitive to pruning, informing further understanding and improvement of architectures.Source: §5 · p. 10

6 APPENDIX

Saved inference time is close to FLOP reduction, and ℓ1-norm pruning works slightly better than ℓ2-norm pruning for VGG-16 conv 2, with no significant difference on other layers.

6.1 COMPARISON WITH ℓ2-NORM BASED FILTER PRUNING

  • ℓ1-norm works slightly better than ℓ2-norm for VGG-16 conv 2, with no significant difference between norms for other layers.The comparison is for filter pruning on CIFAR-10.Source: §6.1 · p. 13, Figure 10 · p. 13

6.2 FLOP AND WALL-CLOCK TIME

  • Physical filter pruning creates a smaller model without masks or sparsity, so its FLOP and wall-clock time match a model with fewer filters built from scratch.FLOP is a statically computed measure of computation complexity that is independent of hardware and software implementations.Source: §6.2 · p. 13
  • Saved inference time is close to FLOP reduction on CIFAR-10 and ILSVRC 2012 inference tests.FLOP counts only Conv and FC operations, excluding Batch Normalization and other overheads; ILSVRC 2012 is used only for ResNet-34. Tests used Torch7, a Titan X (Pascal) GPU, cuDNN v5.1, and mini-batches of 128.Source: §6.2 · p. 13, Table 3 · p. 13

Related papers

Loading 1608.08710v3…