PaperlayerView exact revision on arXiv

Source-linked AI summary

Gate Decorator: Global Filter Pruning Method for Accelerating Deep Convolutional Neural Networks

Zhonghui You, Kun Yan, Jinmian Ye, Meng Ma, Ping Wang

arXiv:1909.08174v1cs.CVcs.LGeess.IV

TL;DR

Resource-intensive CNNs are difficult to deploy on constrained devices, motivating effective filter pruning. The paper introduces Gate Decorator with Taylor-based global importance ranking, together with Tick-Tock and Group Pruning, and reports substantial FLOPs reductions with competitive or improved accuracy.

  • Problem

    CNN computation, storage, and runtime-memory demands hinder deployment on resource-constrained devices, motivating network compression and acceleration.

  • Method

    The paper combines Gate Decorator for global filter importance ranking with Tick-Tock for pruning accuracy and Group Pruning for constrained networks with shortcuts.

  • Results

    The proposed method outperforms several state-of-the-art filter-pruning methods, including 70% FLOPs reduction on ResNet-56 without noticeable accuracy loss and a 0.31% top-1 accuracy gain with 40% FLOPs reduction on ImageNet ResNet-50.

  • Takeaways & Limitations

    Global filter pruning can be viewed as a task-driven network architecture search algorithm within the paper’s demonstrated scope.

Abstract

from arXiv · show

Filter pruning is one of the most effective ways to accelerate and compress convolutional neural networks (CNNs). In this work, we propose a global filter pruning algorithm called Gate Decorator, which transforms a vanilla CNN module by multiplying its output by the channel-wise scaling factors, i.e. gate. When the scaling factor is set to zero, it is equivalent to removing the corresponding filter. We use Taylor expansion to estimate the change in the loss function caused by setting the scaling factor to zero and use the estimation for the global filter importance ranking. Then we prune the network by removing those unimportant filters. After pruning, we merge all the scaling factors into its original module, so no special operations or structures are introduced. Moreover, we propose an iterative pruning framework called Tick-Tock to improve pruning accuracy. The extensive experiments demonstrate the effectiveness of our approaches. For example, we achieve the state-of-the-art pruning ratio on ResNet-56 by reducing 70% FLOPs without noticeable loss in accuracy. For ResNet-50 on ImageNet, our pruned model with 40% FLOPs reduction outperforms the baseline model by 0.31% in top-1 accuracy. Various datasets are used, including CIFAR-10, CIFAR-100, CUB-200, ImageNet ILSVRC-12 and PASCAL VOC 2011. Code is available at github.com/youzhonghui/gate-decorator-pruning

1 Introduction

CNNs deliver strong vision performance but their computation, storage, and memory demands hinder deployment on resource-constrained devices. The paper proposes a global filter-pruning pipeline combining Gate Decorator, Tick-Tock, and Group Pruning, with experiments showing strong pruning results.

  • Motivation: CNN computation, model size, and runtime-memory demands hinder deployment on mobile and IoT devices.VGG-16 is cited as having up to 138 million parameters and requiring more than 500MB of storage.
  • Filter pruning: Filter pruning removes filters and their feature maps while changing the next layer’s input channels.The paper distinguishes filter pruning from other compression approaches and illustrates the corresponding channel reduction.
  • Global pruning: Global pruning avoids predefined per-layer pruning ratios by removing unimportant filters across layers under an overall pruning objective.The paper contrasts this with time-consuming layer-by-layer pruning.
  • Results: 70% FLOPs reduction on ResNet-56 is achieved without noticeable accuracy loss, while ResNet-50 on ImageNet gains 0.31% top-1 accuracy with 40% FLOPs reduction.The paper reports these as representative results of the proposed approach.
  • Proposed approach: The proposed pipeline combines Gate Decorator for global filter importance ranking, Tick-Tock for pruning accuracy, and Group Pruning for shortcut constraints.Group Pruning addresses constrained pruning in networks with shortcuts such as ResNet.
  • Results: Experiments report that the approach outperforms state-of-the-art methods and study GBN, Tick-Tock, and global pruning as task-driven architecture search.These claims are presented as experimental and analytical contributions.

2 Related work

Related work covers filter pruning alongside quantization, fast convolution, and low-rank approximation. It reviews heuristic, reconstruction-based, Taylor-based, and restriction-based pruning strategies, while noting task-specific limitations of discrimination-aware losses.

  • Compression methods: Model compression methods are grouped into quantization, fast convolution, low-rank approximation, and filter pruning.The related work positions filter pruning as one major compression category.
  • Filter-pruning strategies: Filter-pruning methods rank filters using kernel magnitude, APoZ, reconstruction error, Taylor expansion, or learned restrictions.The reviewed approaches include layer-response reconstruction, final-response importance propagation, and discrimination-aware losses.
  • Filter-pruning strategies: Discrimination-aware losses can produce good results but are designed for classification tasks, limiting their scope of application.This limitation is explicitly attributed to the task design of those losses.
  • Other methods: Aggressive binary or ternary quantization usually incurs accuracy loss, whereas moderate quantization can sometimes outperform full-precision networks.The related work contrasts different quantization regimes.

3 Method

The method combines Gate Decorator for global filter importance estimation, Tick-Tock for iterative pruning accuracy, and Group Pruning for shortcut-connected networks. It estimates pruning impact with Taylor expansion, supports sparse gate-based ranking, and coordinates pruning patterns where residual shortcuts require alignment.

  • Gate Decorator: Gate Decorator multiplies each filter’s output by a trainable scalar gate, making a zero gate equivalent to pruning that filter.The method is applied to Batch Normalization as Gated Batch Normalization (GBN), or directly to convolution when Batch Normalization is absent.
  • Gate Decorator: The method estimates loss increase with a Taylor expansion and uses the resulting Θ(φ_i) value as each filter’s global importance score.The Lagrange remainder is omitted because computing it requires a massive amount of calculation, while the first-order estimate is available during back-propagation.
  • Tick-Tock Pruning Framework: Tick trains only gates and the final linear layer for one epoch on a small data subset, recalculating filter importance while avoiding overfitting and addressing pruning-related internal covariate shift.The convolution kernels are non-updatable during Tick.
  • Tick-Tock Pruning Framework: Tock fine-tunes the network with full training data and a sparse constraint on φ, reducing accumulated removal errors while improving importance estimation.After pruning, a separate fine-tuning stage trains longer and removes the sparse constraint.
  • Group Pruning: Group Pruning assigns GBNs linked by pure shortcuts to one group and gives corresponding filters a shared pruning pattern to prevent feature-map misalignment.The group importance score is formed by summing the scores of its members.

4 Experiments

Experiments evaluate the proposed pruning pipeline across classification, segmentation, global-ranking, and architecture-search settings. Results report substantial FLOPs and parameter reductions while retaining or improving accuracy, and support iterative pruning and task-driven channel allocation.

  • Experimental Setup: Experiments span CIFAR-10, CIFAR-100, CUB-200, ImageNet ILSVRC-12, and PASCAL VOC 2011 using VGGNet, ResNet, and FCN architectures.The implementation uses SGD-based training and Tick-Tock pruning settings that vary the fraction of filters removed per stage by architecture.
  • Overall Comparisons: 60% FLOPs reduction on ResNet-56 improves test accuracy by 0.33%, while 70% reduction lowers it by only 0.03% versus baseline.The CIFAR-10 baseline accuracy is 93.1%.
  • Overall Comparisons: 40% FLOPs reduction on ImageNet ResNet-50 increases inference throughput to 1127 images per second for GBN-60 and 1237 for GBN-50.These correspond to reported speedups of 30% and 43%, respectively, relative to the 864-images-per-second baseline.
  • Overall Comparisons: Pruning an FCN-32s reduces FLOPs by 27% and parameters by 73% while maintaining mIoU at 62.84%→62.86%.The experiment evaluates semantic segmentation on the extended PASCAL VOC 2011 dataset.
  • Global Filter Importance Ranking: On CUB-200, GBN produces a smoother accuracy curve and outperforms PCNN, whereas Slim’s magnitude-based ranking changes accuracy dramatically.The comparison uses VGG-16-M and reports test accuracy before fine-tuning.
  • Iterative Pruning and Architecture Search: At equal FLOPs, Tick-Tock and Tick-Only outperform One-Shot pruning, while the pruned VGG-16-M architecture retains more accuracy than uniformly shrinking all channels.The architecture-search experiment reports a 1.30% drop for the pruned network versus 1.98% for the uniformly shrunk network, and 71.02% accuracy after reinitialization and scratch training.

5 Conclusion

The paper proposes three components for global filter pruning and frames the method as task-driven network architecture search. Experiments report that it outperforms several state-of-the-art filter pruning methods.

  • The Gate Decorator algorithm addresses global filter importance ranking.
  • The Tick-Tock framework is designed to improve pruning accuracy.
  • The Group Pruning method addresses constrained pruning problems.
  • Global filter pruning can be viewed as a task-driven network architecture search algorithm.
  • Extensive experiments show that the proposed method outperforms several state-of-the-art filter pruning methods.

A Details of the transformation between GBN and BN

The transformation converts batch normalization into gated batch normalization for pruning and later converts it back by merging the gate into BN parameters.

  • The magnitude of γ provides information for filter ranking.
  • Batch normalization is converted to gated batch normalization using a transformation formula.
  • During pruning, γ is fixed as non-updatable.
  • After pruning, gated batch normalization is converted back to batch normalization by merging φ⃗ into γ and β.

B The Design of Gated Convolution

The gated convolution design introduces a gate into the convolutional operation and merges it back into the filter after pruning.

  • A convolution filter W operates on input tensor X to produce output tensor Y.
  • The filter is converted into a gated version.
  • The convolution operation is modified to incorporate the gate.
  • After pruning, φ is merged into W.

C Segmentation Instances

The proposed method is evaluated on semantic segmentation by pruning an FCN-32s network on the extended PASCAL VOC 2011 dataset. The pruned network reduces FLOPs relative to the baseline.

  • The semantic segmentation experiment prunes an FCN-32s network on the extended PASCAL VOC 2011 dataset.
  • Because FCN-32s has no batch-normalization layer, its convolution layer is replaced with the gated convolution from Appendix B.
  • Figure 6 compares segmentation results from the baseline and pruned networks.
  • 27% FLOPs reduction is achieved by the pruned network.

D Structure of Pruned ResNet-50 (GBN-60).

Figure 7 reports the channel pruning percentage for each convolutional layer, excluding shortcuts, in the GBN-60 network from Table 2.

  • Figure 7 reports channel pruning percentages for each convolutional layer in GBN-60.Shortcut layers are excluded.
  • The figure identifies each layer by its residual-block location.The index format “a-b” indicates the residual block containing the convolutional layer.
  • An index such as “3-6” denotes the convolutional layer conv3_6.

E Structures of the Pruned Networks in Table 4.3

Figure 8 presents the structures of the pruned networks in Table 4.3 at 80% FLOPs reduction, with a minimum channel count of 9. The observations indicate that One-Shot pruning is less accurate, while Tick-Only and Tick-Tock produce similar structures.

  • At 80% FLOPs reduction, Figure 8 shows the structures of the pruned networks in Table 4.3.
  • The minimum number of channels in the pruned networks is set to 9.
  • One-Shot mode cannot accurately perform network pruning.
  • Tick-Only and Tick-Tock modes produce similar network structures.
Loading 1909.08174v1…