PaperlayerView exact revision on arXiv

Source-linked AI summary

Learning Efficient Convolutional Networks through Network Slimming

Zhuang Liu, Jianguo Li, Zhiqiang Shen, Gao Huang, Shoumeng Yan, Changshui Zhang

arXiv:1708.06519v1cs.CVcs.AIcs.LG

TL;DR

Large CNNs are difficult to deploy because their size, memory footprint, and computation burden exceed many resource-constrained platforms. Network slimming imposes sparsity on batch-normalization scaling factors to identify and prune channels, then fine-tunes compact networks. Across architectures and datasets, it achieves substantial compression and operation reductions without accuracy loss, using conventional execution support.

  • Problem

    Large CNNs burden deployment through model size, run-time memory, and computational demands, especially on resource-constrained platforms.

  • Method

    Network slimming jointly trains CNN weights with L1-regularized batch-normalization scaling factors, prunes channels with small factors, and fine-tunes the resulting network.

  • Results

    20x model-size compression and 5x fewer computing operations were achieved while maintaining the same or higher accuracy across benchmark datasets and architectures.

  • Takeaways & Limitations

    The resulting narrower models reduce model size, run-time memory, and computing operations while requiring no special libraries or hardware for efficient inference.

Abstract

from arXiv · show

The deployment of deep convolutional neural networks (CNNs) in many real world applications is largely hindered by their high computational cost. In this paper, we propose a novel learning scheme for CNNs to simultaneously 1) reduce the model size; 2) decrease the run-time memory footprint; and 3) lower the number of computing operations, without compromising accuracy. This is achieved by enforcing channel-level sparsity in the network in a simple but effective way. Different from many existing approaches, the proposed method directly applies to modern CNN architectures, introduces minimum overhead to the training process, and requires no special software/hardware accelerators for the resulting models. We call our approach network slimming, which takes wide and large networks as input models, but during training insignificant channels are automatically identified and pruned afterwards, yielding thin and compact models with comparable accuracy. We empirically demonstrate the effectiveness of our approach with several state-of-the-art CNN models, including VGGNet, ResNet and DenseNet, on various image classification datasets. For VGGNet, a multi-pass version of network slimming gives a 20x reduction in model size and a 5x reduction in computing operations.

1. Introduction

Large CNNs improve representation power but create deployment burdens in model size, run-time memory, and computing operations. Network slimming addresses these constraints by identifying and pruning insignificant channels while preserving comparable accuracy.

  • A 152-layer ResNet exceeds 60 million parameters and 20 Giga FLOPs for one 224×224 image, challenging resource-constrained platforms.
  • CNN deployment is constrained by stored model size, intermediate activation memory, and computationally intensive convolution operations.
  • Existing compression methods often address only one or two deployment challenges, while some require specialized software or hardware accelerators.
  • Network slimming applies L1 regularization to batch-normalization scaling factors so insignificant channels can be identified and pruned without changing CNN architectures.
  • 20x model-size compression and 5x fewer computing operations were achieved while maintaining the same or higher accuracy across benchmark datasets and architectures.

2. Related Work

Prior CNN-efficiency methods compress weights, reduce precision, or sparsify structures, but commonly leave some resource constraints unresolved or require specialized execution support. Network slimming uses structured channel sparsity with conventional software and hardware.

  • Low-rank decomposition provides ∼3x model-size compression but does not notably accelerate inference because CNN computation mainly comes from convolutional layers.
  • Weight quantization saves storage, but shared weights must be restored during inference, preventing run-time memory and inference-time savings.
  • Binary or ternary weight quantization can provide speedup with bitwise libraries but usually incurs moderate accuracy loss.
  • Weight pruning reduces storage through sparse formats, yet speedup requires dedicated sparse-operation libraries or hardware and activation maps remain dense.
  • Structured-pruning methods learn or impose structure-level sparsity, whereas network slimming applies simple L1 sparsity to channel-wise scaling factors.
  • Network slimming belongs to structured sparsity methods and requires no special libraries to obtain inference speedup and run-time memory savings.

3. Network slimming

Network slimming learns channel importance through sparse batch-normalization scaling factors, then prunes low-valued channels and fine-tunes the narrower network. The procedure can be repeated for further compactness and adapted to architectures with cross-layer connections.

  • Motivation: Channel-level sparsity requires removing every incoming and outgoing connection associated with a channel, making direct pretrained-weight pruning ineffective.
  • Scaling Factors and Sparsity-induced Penalty: The method jointly trains network weights and per-channel scaling factors, imposing sparsity regularization so small factors identify channels for pruning.
  • Leveraging the Scaling Factors in BN Layers: Batch normalization supplies trainable channel-wise scale and shift parameters, allowing existing BN scaling factors to serve as channel-importance indicators without network overhead.
  • Channel Pruning and Fine-tuning: A global percentile threshold removes channels with near-zero scaling factors and their connections, producing lower-parameter, lower-memory, lower-computation networks.
  • Channel Pruning and Fine-tuning: Fine-tuning largely compensates temporary pruning-related accuracy loss, and can yield higher accuracy than the original network in many cases.
  • Multi-pass Scheme: The full training, pruning, and fine-tuning procedure can be repeated to obtain an even more compact model.
  • Handling Cross Layer Connections and Pre-activation Structure: The method applies directly to plain CNNs, while ResNet and DenseNet require adaptations for cross-layer connections and pre-activation structures.

4. Experiments

Experiments evaluate network slimming across multiple datasets and CNN architectures, using sparsity training, channel pruning, and fine-tuning. The method achieves substantial parameter and FLOP reductions while generally preserving or improving accuracy.

  • Experimental Setup: Experiments cover CIFAR, SVHN, ImageNet, and MNIST using VGGNet, ResNet, DenseNet, and fully connected networks.CIFAR and SVHN evaluations use VGGNet, ResNet-164, and DenseNet-40; ImageNet uses VGG-A, while MNIST uses a three-layer fully connected network.
  • Training, Pruning and Fine-tuning: The training pipeline selects sparsity regularization strengths, prunes channels using a global scaling-factor threshold, then fine-tunes a narrower model.The sparsity coefficient λ is selected by validation search or fixed per architecture, and the pruning threshold is set by a percentile among all scaling factors.
  • CIFAR and SVHN Results: Up to 10× parameter savings and typically around 50% FLOP reductions are achieved while maintaining similar accuracy with at least 60% of channels pruned.VGGNet shows substantial redundancy, whereas ResNet-164 yields relatively smaller savings, possibly because its bottleneck structure already selects channels.
  • CIFAR and SVHN Results: 40% channel pruning can lower test error after fine-tuning, including DenseNet-40 reaching 5.19% on CIFAR-10, nearly 1% below its original model.The paper hypothesizes that L1 sparsity regularization provides feature selection in intermediate layers.
  • ImageNet Results: On ImageNet, pruning 50% of channels saves more than 5× parameters and 30.4% FLOPs with no accuracy loss.Most pruned channels come from parameter-intensive fully connected layers rather than computation-intensive convolutional layers.
  • Multi-pass Scheme: The multi-pass VGGNet scheme reaches 20× parameter and 5× FLOP reductions on CIFAR-10 while achieving lower test error.On CIFAR-100, pruning near 90% of parameters and near 70% of FLOPs remains possible without notable accuracy loss, although more aggressive pruning eventually increases error.

5. Analysis

The analysis examines how sparsity regularization and pruning percentage affect channel selection and classification accuracy. Increasing regularization concentrates scaling factors near zero, while fine-tuning generally offsets pruning-related accuracy loss until pruning becomes excessive.

  • Effect of Pruned Percentage: A DenseNet-40 trained on CIFAR-10 with λ=10−5 was used to evaluate the effects of pruning varying channel percentages.The results are summarized in Figure 5.
  • Effect of Pruned Percentage: Pruning ratios below a threshold preserve classification performance, whereas performance degrades after the threshold is surpassed.This pattern is observed for both pruned and fine-tuned models.
  • Effect of Pruned Percentage: Beyond 80% pruning, the fine-tuned model’s test error falls behind the baseline model.Fine-tuning typically compensates for accuracy loss caused by pruning, but not at this extreme ratio.
  • Channel Sparsity Regularization: Increasing λ concentrates channel scaling factors increasingly near zero.At λ=0 the distribution is relatively flat, whereas at λ=10−4 almost all factors lie near zero.
  • Channel Sparsity Regularization: Scaling-factor sparsity acts as feature selection in intermediate layers by retaining channels with non-negligible factors.The heatmap shows selected channels becoming brighter while prunable channels become darker during training.

6. Conclusion

The paper concludes that network slimming learns compact CNNs by inducing sparsity in batch-normalization scaling factors and pruning unimportant channels. Across multiple datasets, it reduces computational cost and deployment resources without accuracy loss or specialized inference support.

  • 6. Conclusion: Network slimming learns compact CNNs by regularizing batch-normalization scaling factors so unimportant channels can be identified and pruned.The method introduces minimum training overhead.
  • 6. Conclusion: Up to 20× computational-cost reduction is achieved on state-of-the-art networks with no accuracy loss.The conclusion reports results across multiple datasets.
  • 6. Conclusion: The resulting models reduce model size, run-time memory, and computing operations without requiring special libraries or hardware for efficient inference.

A. Detailed Structure of a Compact Network

The compact VGGNet structure shows how pruning changes channel widths across layers. Deeper layers tend to have more channels removed, and the compact model has lower reported test error than the original.

  • A. Detailed Structure of a Compact Network: Deeper VGGNet layers tend to have more channels pruned in the compact CIFAR-10 model.The compact model comes from the multi-pass experiment labeled “Iter 5 Trained” in Table 4(a).
  • A. Detailed Structure of a Compact Network: The compact VGGNet reduces each layer’s channel width relative to the original network.Table 5 reports the original and compact widths together with parameter and FLOP pruning ratios.
  • A. Detailed Structure of a Compact Network: The compact VGGNet has a 5.96% test error compared with 6.34% for the original VGGNet.These values are reported in Table 5.

B. Wall-clock Time and Run-time Memory Savings

Inference measurements of a 70%-pruned VGGNet compare wall-clock time and run-time memory savings. Time savings roughly track FLOP savings, while memory savings are smaller because pruning is concentrated in deeper layers with smaller activation maps.

  • B. Wall-clock Time and Run-time Memory Savings: A 70%-pruned VGGNet was evaluated during CIFAR-10 inference using Torch on an NVIDIA GeForce 1080 GPU with batch size 64.The experiment measured wall-clock speed and memory footprint.
  • B. Wall-clock Time and Run-time Memory Savings: Wall-clock time savings roughly match the model’s FLOP savings.The comparison is reported for the compact VGGNet in Table 1(a).
  • B. Wall-clock Time and Run-time Memory Savings: Memory savings are less significant because deeper layers have smaller activation maps and therefore occupy less memory.Those deeper layers also tend to have more channels pruned.
  • B. Wall-clock Time and Run-time Memory Savings: The measured savings require no special libraries or hardware.The result follows from producing a narrower model rather than using sparse computation formats.

C. Comparison with [23]

The paper compares network slimming with a previous channel-pruning technique on CIFAR-10 and CIFAR-100 using VGGNet. The comparison follows the earlier method’s pruning criterion and per-layer strategy.

  • The comparison evaluates network slimming against a previous channel-pruning technique on CIFAR-10 and CIFAR-100.
  • Network slimming uses a global pruning threshold, whereas [23] removes predefined proportions of channels at different layers.
  • For a controlled comparison, the authors adopt [23]’s pruning criterion and closely follow its per-layer pruning strategy on VGGNet.
  • The comparison results are presented in Table 7.
Loading 1708.06519v1…