PaperlayerView exact revision on arXiv

Source-linked AI summary

Pruning Convolutional Neural Networks for Resource Efficient Inference

Pavlo Molchanov, Stephen Tyree, Tero Karras, Timo Aila, Jan Kautz

arXiv:1611.06440v2cs.LGstat.ML

TL;DR

The paper addresses the computational cost of fine-tuned CNN inference and the difficulty of selecting parameters to prune efficiently. It proposes iterative greedy pruning with fine-tuning and a Taylor expansion-based saliency criterion, which performs strongly across CNN tasks and achieves a 12.6× GFLOPs reduction with a 2.5% accuracy loss in recurrent gesture recognition.

  • Problem

    Fine-tuned heavyweight CNNs provide strong accuracy for specialized tasks but make inference expensive, while exact parameter-subset selection is computationally infeasible for current networks.

  • Method

    The paper interleaves greedy feature-map pruning with backpropagation fine-tuning and estimates parameter importance using a Taylor expansion-based criterion, with layer-wise normalization for global scaling.

  • Results

    The Taylor criterion generally outperforms alternative pruning criteria, and recurrent gesture recognition reaches a 12.6× GFLOPs reduction with only a 2.5% accuracy loss.

  • Takeaways & Limitations

    Iterative greedy pruning can reduce computation while preserving much of the pruned network’s accuracy across fine-tuned CNNs and recurrent 3D-CNNs.

  • Takeaways & Limitations

    Accuracy comparisons are rough because additional fine-tuning can improve both the initial and pruned networks.

Abstract

from arXiv · show

We propose a new formulation for pruning convolutional kernels in neural networks to enable efficient inference. We interleave greedy criteria-based pruning with fine-tuning by backpropagation - a computationally efficient procedure that maintains good generalization in the pruned network. We propose a new criterion based on Taylor expansion that approximates the change in the cost function induced by pruning network parameters. We focus on transfer learning, where large pretrained networks are adapted to specialized tasks. The proposed criterion demonstrates superior performance compared to other criteria, e.g. the norm of kernel weights or feature map activation, for pruning large CNNs after adaptation to fine-grained classification tasks (Birds-200 and Flowers-102) relaying only on the first order gradient information. We also show that pruning can lead to more than 10x theoretical (5x practical) reduction in adapted 3D-convolutional filters with a small drop in accuracy in a recurrent gesture classifier. Finally, we show results for the large-scale ImageNet dataset to emphasize the flexibility of our approach.

1 INTRODUCTION

Transfer learning improves accuracy on specialized vision tasks but leaves expensive heavyweight CNNs for inference. The paper motivates structured feature-map pruning and contrasts it with existing pruning approaches and their computational trade-offs.

  • Transfer learning adapts large pretrained networks to specialized tasks with limited labeled data, but inference remains costly in time, power, and memory.
  • Convolutional layers dominate prediction runtime, motivating pruning entire feature maps for efficient execution on embedded devices.
  • The proposed procedure interleaves greedy criteria-based pruning with backpropagation fine-tuning to preserve generalization.
  • Second-order pruning methods use Hessian information, which adds memory and computation costs to standard fine-tuning.
  • Structured feature-map and kernel pruning has shown good results on small CNNs, while larger CNNs were not addressed in the cited work.
  • Unstructured pruning can compress networks effectively, but hardware regularities may prevent compression from translating directly into faster inference.
  • Other approaches, including parameter sharing, reduced precision, and tensor decomposition, often require separate training or significant fine-tuning.

2 METHOD

The method alternates pruning and fine-tuning while selecting feature maps with efficient saliency criteria. Its Taylor criterion approximates pruning-induced cost changes using first-order information and avoids Hessian computation.

  • Pruning procedure: The procedure fine-tunes to convergence, alternates pruning with further fine-tuning, and stops at a target accuracy–FLOPs or memory trade-off.
  • Optimization problem: The pruning objective seeks a parameter subset that preserves the adapted network’s cost while satisfying an ℓ0 bound.
  • Optimization problem: Exact subset selection is combinatorial and infeasible for large networks such as VGG-16, which has 4224 convolutional feature maps.
  • Saliency criteria: Pruning removes the least-important parameters iteratively, using saliency to approximate each parameter’s effect on accuracy.
  • Saliency criteria: The oracle ranks parameters by directly measuring cost changes, but requires one training-set evaluation per remaining nonzero parameter.
  • Saliency criteria: Candidate heuristics include kernel-weight norms, activation statistics, mutual information, and Taylor-based approximations.
  • Taylor criterion: The Taylor criterion uses activation–gradient products to approximate absolute pruning-induced cost changes from first-order information.
  • Taylor criterion: Unlike OBD, the method avoids computing the second-order Taylor term or a Hessian diagonal.

3 RESULTS

Experiments evaluate iterative feature-map pruning across transfer-learning, recurrent gesture, and ImageNet settings, emphasizing criterion quality, accuracy, computational cost, and inference speed. The Taylor criterion generally performs strongly, while fine-tuning helps recover accuracy after pruning.

  • 3.1 CHARACTERIZING THE ORACLE RANKING: The oracle ranks feature maps by the loss change caused by individually removing each map, revealing importance differences across layers.Median global importance tends to decrease with depth, while every layer contains both highly and weakly important maps.
  • 3.2 EVALUATING PROPOSED CRITERIA VERSUS THE ORACLE: Taylor achieves the highest Spearman correlation with oracle rankings across layers when layer-wise ℓ2-normalization is used.OBD performs best across layers without normalization and on ImageNet correlation.
  • 3.3 PRUNING FINE-TUNED NETWORKS: Taylor maintains the highest accuracy across nearly the full pruning range for VGG-16 on Birds-200 and performs best relative to operations with FLOPs regularization.OBD is slightly worse for parameter pruning but significantly worse in terms of FLOPs.
  • 3.3 PRUNING FINE-TUNED NETWORKS: Taylor and OBD show superior performance for AlexNet adapted to Flowers-102 in both parameter count and GFLOPs.The unpruned fine-tuned network reaches 80.1% test accuracy, and pruning uses 10 minibatch updates between iterations.
  • 3.5 PRUNING NETWORKS FOR IMAGENET: Fine-tuning after pruning improves VGG-16 ImageNet top-5 validation accuracy from 83% to 87% at 11.5 GFLOPs and from 77.8% to 84.5% at 8.0 GFLOPs.Inference-time gains from FLOPs reductions depend on convolution implementation, parallelization, hardware, scheduling, memory transfer, and related factors.

4 CONCLUSIONS

The paper proposes iterative pruning of deep CNNs and identifies criteria and normalization choices that support effective pruning.

  • Iteratively removing the least important feature maps can successfully prune CNNs.
  • A Taylor expansion-based criterion significantly outperforms other pruning criteria.
  • Per-layer normalization is important for obtaining globally comparable criterion values.

A.1 FLOPS COMPUTATION

The FLOPs computation estimates convolutional and fully connected layer costs from their dimensions, then prioritizes high-cost convolutional neurons for pruning.

  • FLOPs for convolutional kernels are computed under a sliding-window implementation with free nonlinearities.
  • Convolutional FLOPs depend on input height, input width, input channels, kernel width, and output channels.
  • Fully connected FLOPs are computed from input and output dimensionality.
  • FLOPs regularization prunes neurons with higher FLOPs first.

A.2 NORMALIZATION ACROSS LAYERS

Criterion scaling across layers is necessary for global pruning, and ℓ2 normalization makes rankings more similar to the oracle.

  • Without normalization, weight magnitude, activation, and Taylor criteria favor different layer-depth patterns.
  • After ℓ2 normalization, all criteria produce layer-wise rankings more similar to the oracle.
  • The normalization analysis is presented through feature-map ranking statistics and Spearman correlations with oracle-abs.

A.3 ORACLE COMPUTATION FOR VGG-16 ON BIRDS-200

The oracle removes each VGG-16 feature map after fine-tuning on Birds-200 and measures the resulting training-loss change to characterize feature-map importance.

  • The oracle estimates each feature map’s importance by removing it and evaluating network predictions on the training set.
  • Some feature maps reduce training cost when removed, while many have little effect on network output.
  • A small number of feature maps in both examined layers cause substantial loss increases when removed.
  • Figure 11 plots training-loss changes for individual feature-map removals in two convolutional layers.
  • Table 3 compares criteria with oracle-abs using layer-by-layer Spearman rank correlations.

A.4 COMPARISON WITH WEIGHT REGULARIZATION

The comparison examines regularization-based feature-map pruning against the paper’s greedy pruning scheme. The greedy procedure achieves higher test accuracy at severe pruning levels, while regularization can suppress important transfer-learning weights.

  • Method: Regularization-based pruning uses layer-specific sensitivity considerations because different layers tolerate pruning differently.Evaluating that sensitivity requires pruning layers independently during evaluation.
  • Method: The regularization approach prunes entire feature maps when their kernel ℓ2 norms fall below a threshold.The comparison targets the last convolutional layer of VGG-16 fine-tuned on Birds-200.
  • Results: The greedy scheme achieves higher test accuracy than regularization when pruning 85% or more of feature maps.The comparison is made at the same number of remaining unpruned feature maps.
  • Interpretation: High regularization drives all weights toward zero, potentially affecting important connections in transfer learning.The authors contrast this with iterative pruning, which removes unimportant parameters while leaving others untouched.

A.5 COMBINATION OF CRITERIA

The paper tests whether combining Taylor and activation criteria improves saliency estimation. A grid search over the mixing parameter finds only negligible correlation gains.

  • Combination of criteria: The joint criterion linearly combines the Taylor criterion with mean neuron activation using parameter λ.The combination is intended to improve saliency estimation by using both criteria.
  • Evaluation: A grid search over λ identifies the highest correlation value for each dataset and reports its gain.The results are illustrated in Figure 13.
  • Results: The gain from linearly combining criteria is negligibly small.Figure 13 marks each experiment’s correlation gain with a Δ.

A.6 OPTIMAL BRAIN DAMAGE IMPLEMENTATION

The paper compares its Taylor-based saliency approach with an Optimal Brain Damage implementation and characterizes Taylor saliency through its activation and gradient components. The comparison emphasizes the computational cost of Hessian-based estimation and the constituent correlations of Taylor criteria.

  • Optimal Brain Damage: Optimal Brain Damage scores parameters using squared parameter magnitude multiplied by the corresponding diagonal Hessian element.The diagonal Hessian is estimated through random-vector evaluations of Hessian-vector products.
  • Implementation: Estimating the Hessian diagonal for OBD requires 10 iterations over a single minibatch for 1000 minibatches.The authors report that this computational procedure is expensive for iterative pruning.
  • Taylor criterion: The Taylor criterion contains both activation and gradient terms.For the gradient component, the authors use expected absolute gradient because the mean tends to zero.
  • Evaluation: Figure 14 compares Taylor criterion values with gradient and activation components for all neurons and the bottom 10% after layer-wise normalization.The plots use an unpruned VGG network fine-tuned on Birds-200.
Loading 1611.06440v2…