PaperlayerView exact revision on arXiv

Source-linked AI summary

Structural Compression of Convolutional Neural Networks

Reza Abbasi-Asl, Bin Yu

arXiv:1705.07356v4cs.CV

TL;DR

CNNs’ many weights and filters make them difficult to interpret, while reducing filter counts while preserving accuracy is presented as a way to support human interpretation. The paper introduces CAR, a greedy filter-pruning scheme based on accuracy reduction, and reports substantial compression alongside redundant-filter removal and close-to-original accuracy.

  • Problem

    The large numbers of weights and filters in deep CNNs make them difficult to interpret, motivating compression that reduces filters while keeping predictions accurate.

    Source: §I · p. 1

  • Method

    CAR greedily prunes filters according to the classification-accuracy reduction caused by removing each filter.

    Source: §I · p. 2

  • Results

    CAR combined with Deep Compression achieves 8 to 21 folds of compression without serious loss of classification accuracy, and pruned AlexNet filters include redundant color functionality.

    Source: §V · pp. 8, 10

  • Takeaways & Limitations

    The class-based CAR index yields meaningful filter interpretations from the most and least important class labels, consistent with visualized pattern selectivity.

    Source: §I · p. 2

    Speed-oriented tweaks that prune multiple filters per iteration or use smaller evaluation batches make performance slightly lower than with the original CAR compression.

    Source: B. The proposed algorithm based on CAR… · p. 3

Abstract

from arXiv · show

Deep convolutional neural networks (CNNs) have been successful in many tasks in machine vision, however, millions of weights in the form of thousands of convolutional filters in CNNs makes them difficult for human intepretation or understanding in science. In this article, we introduce CAR, a greedy structural compression scheme to obtain smaller and more interpretable CNNs, while achieving close to original accuracy. The compression is based on pruning filters with the least contribution to the classification accuracy. We demonstrate the interpretability of CAR-compressed CNNs by showing that our algorithm prunes filters with visually redundant functionalities such as color filters. These compressed networks are easier to interpret because they retain the filter diversity of uncompressed networks with order of magnitude less filters. Finally, a variant of CAR is introduced to quantify the importance of each image category to each CNN filter. Specifically, the most and the least important class labels are shown to be meaningful interpretations of each filter.

I. INTRODUCTION

The paper proposes structural compression to reduce CNN filter counts while preserving predictive accuracy, addressing both interpretability and deployment constraints. It introduces CAR pruning and a class-based variant, and links filter removal to visual redundancy and human interpretation.

I. INTRODUCTION

  • CNNs’ many weights and filters make them difficult to interpret, while reducing filter counts without sacrificing accuracy can support interpretation and use on memory- and compute-limited devices.Source: §I · p. 1
  • Unlike weight-compression approaches, structural compression removes filters; prior filter-pruning studies emphasized compression and memory use, with limited study of interpretability.Source: §I · p. 1
  • CAR-compressed AlexNet without retraining reaches 54% top-1 classification accuracy versus the original 57%, with compression ratios of 1.17 (layer 1) to 1.5 (layer 5).The paper also reports that CAR plus Deep Compression reduces convolutional-layer size by factors of 8 to 21 while maintaining 54% versus 57% classification accuracy through retraining.Source: §I · p. 2
  • CAR greedily prunes filters according to the classification-accuracy reduction caused by removing each filter, and the paper reports higher hold-out accuracy than baseline structural-compression methods.The paper also describes retraining and combining CAR with Deep Compression.Source: §I · p. 2
  • CAR removes functionally redundant filters, including color filters, making compressed CNNs more accessible to human interpreters without much classification-accuracy loss.Source: §I · p. 2
  • A CAR-index variant quantifies each image class’s importance to each filter, with the most and least important labels providing interpretations consistent with visualized filter selectivity.Source: §I · p. 2

II. CAR-BASED STRUCTURAL COMPRESSION

The notation section defines convolutional filters and their activations, then specifies accuracy and computation measures used in the compression discussion.

A. Notation

  • A filter in layer L is a three-dimensional tensor with dimensions n_(L-1) × f_L × f_L, where f_L × f_L is its spatial receptive-field size.Source: A. Notation · p. 2
  • A filter’s activation applies the network’s nonlinearity to an input block of activations from the previous layer; first-layer inputs may be image patches.Source: A. Notation · p. 2
  • Top-1 classification accuracy is the number of correct predictions divided by the total correct and incorrect predictions.Source: A. Notation · p. 2
  • The paper measures each convolutional layer’s computational cost by the floating-point operations needed to classify one image.Source: A. Notation · p. 2

B. The proposed algorithm based on CAR importance index

CAR ranks filters by the classification accuracy reduction caused by removing them, then greedily prunes the least important filter. The method can be fine-tuned after each removal, and its speed-ups trade some performance for faster compression.

B. The proposed algorithm based on CAR importance index

  • The algorithm recomputes filter importance after each removal and stops when it reaches the target compression ratio.Fine-tuning after pruning is optional; the number of remaining filters and current compression ratio are tracked during the loop.Source: B. The proposed algorithm based on CAR… · p. 3
  • CAR importance is the classification accuracy reduction when a filter and its next-layer connections are removed.The filter with the least effect on accuracy is pruned at each iteration.Source: B. The proposed algorithm based on CAR… · p. 3
  • Pruning multiple filters per iteration or evaluating accuracy on fewer images speeds compression, but slightly lowers performance than the one-filter-at-a-time algorithm.Updating importance after each single removal lets the method adapt gradually to the new network structure.Source: B. The proposed algorithm based on CAR… · p. 3
  • CAR compresses each layer separately, which the authors consider sufficient for most transfer-learning and interpretability applications because layers are interpreted individually.Multiple layers can also be compressed together.Source: B. The proposed algorithm based on CAR… · p. 3

A. Compression rate and classification accuracy of the CAR

The evaluation compresses LeNet, AlexNet, and ResNet-50, with implementation and fine-tuning settings specified for the experiments. The authors also report the computation required for pruning AlexNet's first layer.

A. Compression rate and classification accuracy of the CAR

  • The evaluation compressed LeNet, AlexNet, and ResNet-50, spanning networks with 2, 5, and 50 convolutional layers, respectively.LeNet was trained on MNIST; AlexNet and ResNet-50 were trained on the ILSVRC 2012 ImageNet subset.Source: A. Compression rate and classification… · p. 3
  • Each AlexNet first-layer pruning iteration took 45 minutes on average; pruning its 96 filters took 72 hours, reducible to 14 hours by pruning up to five filters per iteration without accuracy loss.The CAR index used half of the ImageNet test set, and final compressed-network performance was evaluated on the other half.Source: A. Compression rate and classification… · p. 3
  • Fine-tuning used 500 iterations, with learning rates of 0 for the compressed layer, 0.001 for the subsequent layer, and 0.0001 for other layers.The subsequent layer received the higher learning rate because it is directly affected by compression.Source: A. Compression rate and classification… · p. 4

1) LeNet on MNIST dataset:

On LeNet, CAR's classification accuracy was similar to the benchmark except under very aggressive pruning. On AlexNet, CAR outperformed the benchmark filter-pruning scheme, with additional compression results reported at a specified accuracy level.

1) LeNet on MNIST dataset:

  • The LeNet comparison included CAR, fine-tuned CAR, a weight-based benchmark, and random filter pruning.The random-pruning error bars show standard deviation over 10 repeats.Source: 1) LeNet on MNIST dataset · p. 4
  • On LeNet, CAR achieved classification accuracy similar to, except when fewer than 8 of 20 first-layer filters or fewer than 10 of 50 second-layer filters were kept.Fine-tuning improved accuracy, and the retained filters partly overlapped with those selected by the outgoing-weight method.Source: 1) LeNet on MNIST dataset · p. 4

2) AlexNet on ImageNet dataset:

  • 16% to 25% higher classification accuracies than the best benchmark filter-pruning scheme were achieved by CAR when pruning half of an AlexNet convolutional layer's filters.The reported range spans layer 5 to layer 2; retraining produced 50% to 52% classification accuracy, compared with 57% for uncompressed AlexNet.Source: 2) AlexNet on ImageNet dataset · p. 4
  • At 54% classification accuracy, CAR without retraining achieved compression ratios of 1.17 for layer 1 to 1.50 for layer 5, 21% to 43% above benchmark methods.Fine-tuning or retraining reached a compression ratio as high as 1.79 for layer 3 while maintaining the same 54% classification accuracy.Source: 2) AlexNet on ImageNet dataset · p. 4

3) Combination with Deep Compression:

CAR can be combined with Deep Compression for additional compression, and on ResNet-50 it outperforms weight-based pruning at high compression of the first layer. The residual-module results also show that shortcuts help the network tolerate extensive filter pruning.

3) Combination with Deep Compression:

  • Combining CAR with Deep Compression achieves 5-fold to 12-fold higher compression ratios while accuracy remains 54% compared with 57% uncompressed.The additional weight compression sparsifies weights in the filters retained by CAR.Source: 3) Combination with Deep Compression · p. 4, Table II · p. 6

4) ResNet-50 on ImageNet dataset:

  • 72% classification accuracy versus 75% for uncompressed ResNet-50 when half the first-layer filters are pruned; this is 15% higher than pruning by average incoming or outgoing weights.At low compression, performance is similar across methods; CAR’s advantage grows at higher compression.Source: 4) ResNet-50 on ImageNet dataset · pp. 4–6
  • Pruning all filters from Conv a - Branch 2 drops accuracy by less than 15%, and fine-tuning reduces the drop to less than 5%.The authors attribute this tolerance to shortcuts that let information pass through the parallel branch.Source: 4) ResNet-50 on ImageNet dataset · p. 6

B. CAR-compression algorithm prunes visually redundant

CAR tends to prune filters with visually redundant functionality while retaining diverse patterns. Its compressed networks also preserve accuracy close to the uncompressed network for many ImageNet categories.

B. CAR-compression algorithm prunes visually redundant

  • Higher-CAR-importance first-layer AlexNet filters span diverse orientations and spatial frequencies, while most color-selective filters have lower importance.Among the top 20 pruned filters, 15 in the first layer and 14 in the second layer are color filters.Source: B. CAR-compression algorithm prunes… · pp. 6–7, Figure 5 · p. 8
  • With accuracy at 54% or within a relative 5% of the uncompressed network, CAR prunes 103 of 256 second-layer filters and tends to keep at least one filter from each visually similar group.The authors interpret the pruned filters as having redundant functionality.Source: B. CAR-compression algorithm prunes… · p. 8
  • Although compressed networks are about a relative 5% lower in total accuracy, 37% to 49% of ImageNet categories are no more than 3% below uncompressed accuracy.The range spans compressed layer 5 to layer 2, respectively.Source: B. CAR-compression algorithm prunes… · p. 8

IV. CLASS-BASED INTERPRETATION OF FILTERS

A class-specific variant of CAR quantifies how much each filter matters to each image class. The classes with the highest and lowest scores provide verbal interpretations that align with visualized filter patterns in higher layers.

IV. CLASS-BASED INTERPRETATION OF FILTERS

  • CARc(i, L) measures the accuracy reduction for class c when filter i in layer L is pruned, identifying classes that depend more or less on that filter.The highest- and lowest-scoring class labels are used as verbal interpretations of the filter.Source: §IV · p. 8
  • For three AlexNet layer-5 filters, classes with the highest CARc share patterns with their top activating image patches, including curvature, insect or bird-head patterns, and single long tools.The authors note that class-label interpretation fits higher, more semantic layers and is consistent there with pattern-selectivity visualization.Source: §IV · p. 8

V. DISCUSSION AND FUTURE WORK

CAR applies greedy filter pruning to compress AlexNet while examining what removed filters reveal about learned functionality. The discussion also presents a class-based interpretation variant and notes limits of greedy search and questions for ResNet compression.

V. DISCUSSION AND FUTURE WORK

  • CAR+Deep Compression achieved an 8 to 21 folds gain in compression ratio without a serious loss of classification accuracy.The paper frames these as AlexNet results.Source: §V · p. 8
  • 103 of 256 second-layer AlexNet filters were pruned within a 5% relative accuracy range; many redundant filters for ImageNet categories were color-based rather than shape-based.The paper says this suggests the first-order importance of shape for those categories.Source: Figure 6 · p. 9, §V · p. 10
  • CARc interpretation was consistent with visualized pattern selectivity, contrasting the top and bottom five classes for each layer-5 filter.Figure 8 also shows sample images and class-level classification accuracy.Source: Figure 8 · p. 10
  • Greedy pruning is likely sub-optimal for selecting filters, while searching all subsets may be computationally expensive and risk over-pruning.The authors identify subset-selection procedures as a possible alternative direction.Source: §V · p. 10
  • ResNet compression merits further investigation because residual blocks contain identity branches.The discussion notes CAR pruning of the identity branch and redundant connections, while emphasizing that this architecture needs more study.Source: §V · p. 10
  • A CAR variant compares original and pruned CNNs by class-level accuracy and can compare filters in networks trained on similar datasets through their important class labels.The paper also identifies adaptation to fully connected networks as a possible direction.Source: §V · pp. 10–11

I. BOOSTING THE COMPRESSION SPEED

The supplemental experiments test faster CAR settings, joint compression across layers, and filter pruning in AlexNet layers 3 and 4. Multiple-filter pruning and smaller evaluation batches speed LeNet compression, while joint compression reduces accuracy as more layers are included.

I. BOOSTING THE COMPRESSION SPEED

  • A 60-fold compression-speed increase for LeNet kept accuracy close to original CAR compression by combining five-filter pruning with batch size 128 instead of 5000.Five-filter pruning increased speed by a factor of 5, and the smaller batch increased it by a factor of 12.Source: §I · p. 13, Figure S.1 · p. 14
  • For LeNet layer 2, pruning up to five filters per iteration kept accuracy close at batch size 512, but batch size 64 degraded it.The supplementary figure compares accuracy curves across pruning batch sizes.Source: §I · p. 13, Figure S.2 · p. 15

II. COMPRESSING MULTIPLE LAYERS

  • Classification accuracy decreases as the number of layers involved in joint compression increases.The curves plot accuracy against the portion of remaining filters in the network.Source: §II · p. 13, Figure S.3 · p. 16

III. CAR PRUNED FILTERS FOR LAYERS 3 AND 4 OF ALEXNET

  • Within a 5% relative accuracy range, CAR pruned 170 of 384 filters in AlexNet layer 3 and 159 of 384 in layer 4, while retaining diverse filter functionality.The removed filters are marked in the visualizations; filters were manually grouped by pattern selectivity.Source: §III · p. 13, Figure S.4 · p. 17, Figure S.5 · p. 18

Related papers

Loading 1705.07356v4…