PaperlayerView exact revision on arXiv

Source-linked AI summary

Asymptotic Soft Filter Pruning for Deep Convolutional Neural Networks

Yang He, Xuanyi Dong, Guoliang Kang, Yanwei Fu, Chenggang Yan, Yi Yang

arXiv:1808.07471v4cs.CV

TL;DR

Deep CNNs offer strong performance but have expensive computation, and existing filter pruning can reduce model capacity and lose filter information. ASFP updates pruned filters and increases pruning gradually; on ResNet-50, it reports 41.8% speedup with a 0.14% top-5 accuracy drop. The authors identify extra hyperparameters and a hand-crafted pruning schedule as limitations.

  • Problem

    Deep CNNs require costly computation, while hard pruning reduces optimization space and pruning many filters initially can cause unrecoverable information loss.

    Source: §I · pp. 1–2

  • Method

    ASFP updates pruned filters during training and changes the pruning rate across epochs, with constant-rate SFP described as a special case.

    Source: §I · p. 2, D. Asymptotic Soft Filter Pruning (ASFP) · p. 4

  • Results

    For pre-trained ResNet-50, ASFP achieves 41.8% speedup with a 0.14% top-5 accuracy drop, compared with SFP’s 8.27% top-5 accuracy drop at the same speedup.

    Source: D. ResNet on ILSVRC-2012 · p. 7

  • Takeaways & Limitations

    The authors report competitive performance without pretraining and better results when using a pre-trained model.

    Source: §V · p. 10

    ASFP requires additional hyperparameters, uses a hand-crafted pruning-rate schedule that may not be best, and lacks a theoretical demonstration of information loss after pruning.

    Source: §V · p. 10

Abstract

from arXiv · show

Deeper and wider Convolutional Neural Networks (CNNs) achieve superior performance but bring expensive computation cost. Accelerating such over-parameterized neural network has received increased attention. A typical pruning algorithm is a three-stage pipeline, i.e., training, pruning, and retraining. Prevailing approaches fix the pruned filters to zero during retraining, and thus significantly reduce the optimization space. Besides, they directly prune a large number of filters at first, which would cause unrecoverable information loss. To solve these problems, we propose an Asymptotic Soft Filter Pruning (ASFP) method to accelerate the inference procedure of the deep neural networks. First, we update the pruned filters during the retraining stage. As a result, the optimization space of the pruned model would not be reduced but be the same as that of the original model. In this way, the model has enough capacity to learn from the training data. Second, we prune the network asymptotically. We prune few filters at first and asymptotically prune more filters during the training procedure. With asymptotic pruning, the information of the training set would be gradually concentrated in the remaining filters, so the subsequent training and pruning process would be stable. Experiments show the effectiveness of our ASFP on image classification benchmarks. Notably, on ILSVRC-2012, our ASFP reduces more than 40% FLOPs on ResNet-50 with only 0.14% top-5 accuracy degradation, which is higher than the soft filter pruning (SFP) by 8%.

I. INTRODUCTION

Deep CNNs offer strong performance but impose costly computation, motivating filter pruning that can exploit structured sparsity. ASFP addresses capacity and information-loss problems in existing pruning through filter updates and gradually increasing pruning.

I. INTRODUCTION

  • Deep CNNs incur expensive computation, while filter pruning offers structured sparsity that can use BLAS libraries for more efficient memory use and realistic acceleration.The introduction also notes that storage, memory, and computation demands exceed the limits of mobile devices and drones.Source: §I · p. 1
  • ASFP updates pruned filters during retraining and increases pruning gradually, aiming to preserve model capacity and make training and pruning more stable.The authors report experiments on CIFAR-10 and ImageNet and say the method can be trained and pruned from scratch.Source: §I · p. 2

1) Matrix Decomposition:

  • Matrix decomposition represents convolutional weights as products of smaller matrices, but its computational cost can hinder deep-CNN training and its use with 1×1 kernels can be difficult.The passage gives ResNet bottleneck blocks as an example of 1×1 convolution use.Source: 1) Matrix Decomposition · p. 2

2) Low Precision:

The related-work discussion contrasts compression approaches and their trade-offs, while the preliminary section describes CNN filters and how pruning reduces output-feature-map dimensions.

2) Low Precision:

  • Low-precision methods reduce weight precision; the cited examples include ternary quantization and converting pretrained full-precision CNNs to low precision for inference.The passage states that storing and using low-precision weights reduces storage and computation costs.Source: 2) Low Precision · p. 2

3) Weight Pruning:

  • Weight pruning can produce unstructured models that cannot leverage efficient BLAS libraries, making realistic speedup difficult; the cited Bayesian methods were evaluated on small datasets.The passage also describes approaches that prune small weights or exploit sparsity in feature maps or parameters.Source: 3) Weight Pruning · p. 2

4) Filter Pruning:

  • Filter pruning removes corresponding feature maps and can reduce storage and memory; the discussion distinguishes data-dependent from data-independent selection and notes capacity and information-loss concerns in prior methods.Data-independent methods are described as more efficient when training data is unavailable during pruning.Source: 4) Filter Pruning · p. 2

A. Preliminary

  • A convolutional layer’s weight matrix comprises filters, and pruning filters reduces the number and dimensions of useful output feature maps.The passage defines each filter as a set of weights and describes the output feature-map size after applying a layer’s pruning rate.Source: A. Preliminary · p. 3

B. Pruning with Hard Manner

Hard filter pruning freezes removed filters, whereas soft filter pruning keeps them in the model during retraining so they can update before final removal.

B. Pruning with Hard Manner

  • Hard filter pruning typically prunes one layer of a pretrained model at a time, fine-tunes, and leaves each pruned filter frozen thereafter.The remaining filters may update during retraining to compensate for performance degradation.Source: B. Pruning with Hard Manner · p. 3

C. Pruning with Soft Manner

  • Soft filter pruning sets selected filters to zero during training but continues updating them, retaining the original filter count until unimportant filters are removed after convergence.The described selection criterion is filter norm, and back-propagation can make previously pruned filters nonzero again.Source: C. Pruning with Soft Manner · p. 3

D. Asymptotic Soft Filter Pruning (ASFP)

ASFP gradually increases pruning across training, starting with a small rate to limit early information loss. Unlike SFP’s fixed sparsity level, ASFP allows sparsity to vary by epoch and ranks filters by norm for pruning.

D. Asymptotic Soft Filter Pruning (ASFP)

  • A small early pruning rate limits information loss; iterative training and pruning then concentrate training-set information in important filters and stabilize the process.The pruning rate increases at later epochs until it reaches the goal rate at the last retraining epoch.Source: D. Asymptotic Soft Filter Pruning (ASFP) · p. 4
  • SFP uses a fixed sparsity level, whereas ASFP makes sparsity epoch-dependent; SFP is the special case with the same pruning rate throughout training.The ASFP constraint sets the number of filters to retain for the next epoch.Source: D. Asymptotic Soft Filter Pruning (ASFP) · p. 4

1) Asymptotic Filter Selection:

  • Filters with smaller ℓ_p-norm receive higher pruning priority because their convolutional outputs are described as having less impact on prediction.The implementation uses the ℓ2-norm based on empirical analysis.Source: 1) Asymptotic Filter Selection · p. 4
  • ASFP changes the pruning rate exponentially, with the goal rate and minimum rate set as parameters for the schedule.The passage describes the exponential schedule as having parameters that are solved from three points.Source: 1) Asymptotic Filter Selection · p. 4

9 Obtain the compact model with parameters W∗from W

The pruning-rate schedule begins at a minimum rate, passes through a specified intermediate point, and reaches the goal rate at the final retraining epoch.

9 Obtain the compact model with parameters W∗from W

  • At epoch 0, the pruning rate is set to P_min_i.This is the first point used to determine the exponential schedule.Source: §9 · p. 5
  • At epochmax, the pruning rate reaches the goal rate P_goal_i.The final-epoch point is used to solve the schedule parameters.Source: §9 · p. 5
  • At epochmax × D, the schedule sets the pruning rate to 3/4 of P_goal_i.This intermediate point supplies the third point for solving the exponential schedule.Source: §9 · p. 5

2) Filter Pruning:

ASFP applies a shared goal pruning rate, reconstructs zeroed filters through training, and removes filters that remain zero after convergence to form a compact model.

2) Filter Pruning:

  • Using one shared pruning-rate parameter P balances acceleration and accuracy while avoiding inconvenient hyperparameter search or complicated sensitivity analysis.The method uses the same goal pruning rate for all weighted layers.Source: 2) Filter Pruning · p. 5

3) Reconstruction:

  • Back-propagation updates pruned filters to nonzero values during reconstruction, preserving the original model capacity while training and pruning proceed together.The network is trained for one epoch after pruning before the next reconstruction step.Source: 3) Reconstruction · p. 5

4) Obtaining Compact Model:

  • After convergence, filters that remain zero and their corresponding feature maps can be removed without affecting inference outputs.Figure 4 illustrates a residual block at 50% pruning; red and green labels mark remaining output and input channel counts, respectively.Source: 4) Obtaining Compact Model · p. 5, Figure 4 · p. 5

E. Pruning Strategy for Convolutional Network

The section explains how filter pruning changes channel dimensions and convolutional computation, then describes practical speedup caveats and the benchmarks and settings used to evaluate the method.

E. Pruning Strategy for Convolutional Network

  • Pruning residual blocks changes convolution output channels and the following layers’ input channels, while residual additions must still align their channel dimensions.The passage describes a 50% pruning example and specifies how the residual output and convolution output are combined.Source: E. Pruning Strategy for Convolutional… · p. 5

1) Theoretical Speedup Analysis:

  • Pruning adjacent layers reduces the next layer’s computation by the product of their remaining-channel fractions, yielding a reduced proportion of 1 − (1 − P_i+1) × (1 − P_i).The calculation accounts for pruning in both the current layer and its successor.Source: 1) Theoretical Speedup Analysis · p. 5

2) Realistic Speedup Analysis:

  • Reduced FLOPs do not translate directly into the same realistic speedup because non-tensor operations, IO delays, buffer switching, and BLAS efficiency also affect GPU inference time.The theoretical analysis treats batch normalization and pooling as negligible relative to convolution.Source: 2) Realistic Speedup Analysis · p. 6

A. Benchmark Datasets and Experimental Setting

  • The evaluation uses CIFAR-10 and ILSVRC-2012; the datasets contain 50,000/10,000 and 1.28 million/50,000 training/test or validation images, respectively.CIFAR-10 has ten classes and ILSVRC-2012 has 1,000 classes.Source: A. Benchmark Datasets and Experimental… · p. 6
  • The experiments target ResNet models and also prune VGGNet on CIFAR-10; ResNet pruning uses a shared rate across convolutional layers and leaves projection shortcuts unpruned.The stated reason for omitting projection shortcuts is their negligible time and lack of effect on overall cost.Source: A. Benchmark Datasets and Experimental… · p. 6
  • Baseline accuracy comes from the original papers, and the authors use accuracy drop rather than accelerated accuracy to account for differing settings and implementations.For ImageNet, the tables list one-view accuracy; CIFAR-10 results are averaged over three runs with standard deviations.Source: A. Benchmark Datasets and Experimental… · p. 6
  • ASFP’s extra pruning operations add little training time: ResNet-110 takes 171.45 seconds per epoch with ASFP versus 171.01 seconds with hard pruning on GTX 1080.The comparison is presented as evidence that the time difference is negligible.Source: A. Benchmark Datasets and Experimental… · p. 6

B. VGGNet on CIFAR-10

ASFP’s VGGNet results favor it over across scratch and pre-trained settings. Additional fine-tuning increases its advantage over.

B. VGGNet on CIFAR-10

  • 94.02% vs. 93.28% accuracy after 160 fine-tuning epochs shows ASFP’s advantage over grows with longer fine-tuning.The accuracy of was nearly unchanged at 93.28% vs. 93.22%.Source: B. VGGNet on CIFAR-10 · p. 7
  • 81.66% vs. 77.45% accuracy: the pruned model without fine-tuning performs better with ASFP than with.After 40 fine-tuning epochs, ASFP achieves similar accuracy to.Source: B. VGGNet on CIFAR-10 · p. 7
  • 93.37% vs. 93.31% accuracy: ASFP slightly outperforms when pruning random-initialized VGGNet.The table covers scratch and pre-trained VGGNet on CIFAR-10; FT denotes fine-tuning.Source: Table II · p. 7, B. VGGNet on CIFAR-10 · p. 7

C. ResNet on CIFAR-10

On CIFAR-10 and ImageNet, ASFP compares favorably with other pruning methods, including SFP at higher pruning levels. Its performance gap during training is also reported as more stable than SFP’s.

C. ResNet on CIFAR-10

  • 52.3% speed-up with 0.48% accuracy drop: ASFP improves on PFEC for scratch ResNet-110, which achieves 38.6% speedup with 0.61% drop.For scratch ResNet-56, ASFP achieves 52.6% acceleration with 1.15% accuracy drop, versus CP at 50.0% and 1.90%.Source: C. ResNet on CIFAR-10 · p. 7

D. ResNet on ILSVRC-2012

C. ResNet on CIFAR-10

  • At large pruning rates, ASFP outperforms SFP; their performance is competitive when the pruning rate is small.The comparison is shown against the unpruned model and across different pruned-FLOP ratios.Source: C. ResNet on CIFAR-10 · p. 7, Figure 6 · p. 8

D. ResNet on ILSVRC-2012

  • ASFP outperforms SFP on almost all tested pruned-FLOP ratios for pre-trained models and also outperforms it with random initialization.The comparison tests ResNet-56 and ResNet-110.Source: E. Comparing SFP and ASFP · p. 8
  • At a 30% pruning rate, across ResNet-18 and ResNet-50, ASFP’s performance gap is much more stable than SFP’s during almost all 100 retraining epochs in the reported comparison.Source: E. Comparing SFP and ASFP · p. 8

1) Filter Selection Criteria:

The ablations examine filter selection criteria and performance across pruning rates. They report a slight advantage for ℓ2-norm selection and a larger ASFP advantage over SFP beyond 40% pruned FLOPs.

1) Filter Selection Criteria:

  • ℓ2-norm selection performs slightly better than ℓ1-norm selection; the authors attribute this to preserving filters with large weights and their discriminative features.The comparison is for ResNet-110 on CIFAR-10 under different pruning rates.Source: 1) Filter Selection Criteria · p. 8, Table V · p. 9

2) Varying Pruned FLOPs:

  • Above 40% pruned FLOPs, ASFP performs much better than SFP; below 40%, their accuracy is similar.The authors relate ASFP’s advantage at higher pruning rates to its effectiveness when pruning removes more information.Source: 2) Varying Pruned FLOPs · p. 9
  • From 0% to about 23% pruned FLOPs, both ASFP and SFP exceed baseline accuracy, which the authors describe as a regularization effect.Source: 2) Varying Pruned FLOPs · p. 9

3) Selection of the Pruned Layers:

The section compares pruned layer choices, tests sensitivity to the ASFP interval and parameter D, and closes with performance claims and limitations for ASFP.

3) Selection of the Pruned Layers:

  • At 30% pruning in ResNet-110, pruning first block layers yielded 93.96 ± 0.13% accuracy versus 93.38 ± 0.44% for second layers; sensitivity-based tuning was out of scope.The authors note that careful layer selection based on sensitivity may improve performance, but tuning those hyperparameters was not the manuscript’s focus.Source: 3) Selection of the Pruned Layers · p. 9

4) Sensitivity of the ASFP Interval:

5) Sensitivity of Parameter D of ASFP:

  • For scratch and pretrained ResNet-56 on CIFAR-10 at 40% pruning, changing D from 7 to 16 caused less than 0.3% accuracy fluctuation.The authors conclude that the final pruning result is not sensitive to D in this tested range.Source: 5) Sensitivity of Parameter D of ASFP · p. 9

V. CONCLUSION & FUTURE WORK

  • ASFP updates pruned filters and adjusts pruning rates asymptotically, achieving competitive performance from scratch and better results with pretrained models.The conclusion reports these outcomes relative to state-of-the-art approaches.Source: §V · pp. 9–10
  • The authors identify additional pruning-rate hyperparameters, a hand-crafted schedule, and the lack of theoretical analysis of post-pruning information loss as limitations.They list combining ASFP with matrix decomposition or low-precision weights as future work.Source: §V · p. 10

Related papers

Loading 1808.07471v4…