PaperlayerView exact revision on arXiv

Source-linked AI summary

Exploiting Kernel Sparsity and Entropy for Interpretable CNN Compression

Yuchao Li, Shaohui Lin, Baochang Zhang, Jianzhuang Liu, David Doermann, Yongjian Wu, Feiyue Huang, Rongrong Ji

arXiv:1812.04368v2cs.CV

TL;DR

CNN compression methods often do not interpret network structure to distinguish redundancy, despite the high computation and memory costs of deep CNNs. The paper uses KSE, based on kernel sparsity and entropy, to guide kernel clustering for compression. On ResNet-50, it reports 4.7× FLOPs reduction and 2.9× compression with a 0.35% Top-5 accuracy drop on ImageNet.

  • Problem

    Many CNN compression methods do not interpret network structure to distinguish implicit redundancy, while deep CNNs require substantial computation and memory.

    Source: §1 · pp. 1–2

  • Method

    KSE estimates feature-map importance from corresponding kernels’ sparsity and entropy, then guides kernel clustering to compress the network.

    Source: §4 · p. 4, §4.3 · p. 5, §4.4 · p. 5

  • Results

    4.7× FLOPs reduction and 2.9× compression on ResNet-50 came with a 0.35% Top-5 accuracy drop on ImageNet, outperforming state-of-the-art methods.

    Source: §1 · p. 2

  • Takeaways & Limitations

    The authors conclude that feature-map sparsity and information richness are key elements for evaluating importance, and that KSE-compressed networks achieve better results than state-of-the-art methods.

    Source: §6 · p. 8

    The authors identify a need for a more rigorous theoretical proof, with bounds or conditions, for the relationship between feature maps and kernels.

    Source: §6 · p. 8

Abstract

from arXiv · show

Compressing convolutional neural networks (CNNs) has received ever-increasing research focus. However, most existing CNN compression methods do not interpret their inherent structures to distinguish the implicit redundancy. In this paper, we investigate the problem of CNN compression from a novel interpretable perspective. The relationship between the input feature maps and 2D kernels is revealed in a theoretical framework, based on which a kernel sparsity and entropy (KSE) indicator is proposed to quantitate the feature map importance in a feature-agnostic manner to guide model compression. Kernel clustering is further conducted based on the KSE indicator to accomplish high-precision CNN compression. KSE is capable of simultaneously compressing each layer in an efficient way, which is significantly faster compared to previous data-driven feature map pruning methods. We comprehensively evaluate the compression and speedup of the proposed method on CIFAR-10, SVHN and ImageNet 2012. Our method demonstrates superior performance gains over previous ones. In particular, it achieves 4.7 \times FLOPs reduction and 2.9 \times compression on ResNet-50 with only a Top-5 accuracy drop of 0.35% on ImageNet 2012, which significantly outperforms state-of-the-art methods.

1. Introduction

The paper frames CNN compression as finding redundancy through an interpretable account of feature-map importance. It proposes KSE to guide kernel clustering and reports strong compression results across benchmark datasets.

1. Introduction

  • CNNs’ computation and memory costs constrain direct use on mobile or embedded devices, while pruning methods target different network elements.The introduction contrasts unstructured weight pruning with filter pruning, which can reduce computation and memory overhead but may create dimensional mismatch in multi-branch networks.Source: §1 · pp. 1–2
  • The paper identifies sparsity and information richness as key elements of feature-map importance, motivating an interpretable way to locate network redundancy.It argues that feature maps vary in importance both across and within layers.Source: §1 · p. 2
  • KSE measures input feature-map importance from corresponding 2D kernels’ sparsity and entropy, then guides kernel clustering for compression.The approach is feature-agnostic and can process layers in parallel, unlike data-driven methods that compute feature maps across the training dataset.Source: §1 · pp. 2–3
  • 4.7× FLOPs reduction and 2.9× compression on ResNet-50 came with only a 0.35% Top-5 accuracy drop on ImageNet.The authors report superior performance to state-of-the-art methods.Source: §1 · p. 2

2. Related Work

Related work spans unstructured weight pruning, structured pruning, channel pruning, and other compression approaches. The paper positions its kernel-level reduction as avoiding a dimensional mismatch associated with filter pruning while estimating feature-map importance without extra training.

2. Related Work

  • Unstructured pruning removes individual weights but requires special storage formats and sparse-matrix software or hardware for speedup.Han et al. [3] prune small-magnitude weights, while Yang et al. [42] use layer-wise error reconstruction.Source: §2 · p. 3
  • Structured filter pruning can reduce memory footprint but may cause dimensional mismatch in multi-branch networks; this method instead reduces kernels tied to input maps without changing convolution outputs.The paper contrasts filter-pruning approaches [6] [7] [24] with its kernel-level reduction, which avoids the mismatch problem in ResNets [5].Source: §2 · p. 3
  • Channel-pruning methods suit ResNets and DenseNets but use complex training procedures with significant offline cost; this approach estimates map importance from kernels without extra training.Examples include batch-normalization scaling-factor regularization [22] and weight pruning combined with group convolution [8].Source: §2 · p. 3
  • Other compression methods have constraints: some use uniform centroid counts, while a spatial-kernel method is mainly suited to 3 × 3 kernels and is difficult to apply to 1 × 1 kernels.The paper says KSE can be applied to all layers.Source: §2 · p. 3

3. Interpretation of Feature Maps

The section interprets feature maps through receptive fields on the input image and weights those regions by an image heat map. It concludes that feature-map importance depends on receptive-field sparsity and information richness, while noting the cost of computing this measure across the training dataset.

3. Interpretation of Feature Maps

  • Receptive fields are computed on the original input image for feature maps from the same convolutional layer, with visual differences indicating different information content.The cited approach scales feature maps to input resolution and thresholds them to obtain binary masks [1].Source: §3 · p. 3
  • A heat map is formed by summing the last convolutional layer’s output feature maps and scaling the result to input resolution; masked heat-map values quantify a feature map’s information.The red region within a receptive field represents the corresponding feature map’s interpretation.Source: §3 · p. 4
  • Feature-map importance is defined by receptive-field area times mean heat-map intensity, capturing sparsity and information richness, but computing it over the full training dataset is costly.I{M = 1} denotes the number of mask elements equal to 1, and H is the mean heat-map value over those elements.Source: §3 · p. 4

4. Proposed Method

The method estimates input feature-map importance from corresponding 2D kernels, combining kernel sparsity and entropy in a feature-agnostic KSE indicator. It uses that score to cluster kernels for fine-grained compression.

4. Proposed Method

  • KSE represents input feature-map sparsity and information richness using corresponding 2D kernels, and can also treat fully connected layers as 1 × 1 convolutions.During inference, the method computes 2D activation maps from input feature maps and sums them to produce output feature maps.Source: §4 · p. 4, §4.4 · p. 5

4.1. Kernel Sparsity

  • Kernel sparsity uses the summed ℓ1-norms of an input feature map’s corresponding 2D kernels as a proxy for feature-map sparsity.The paper argues that sparse input feature maps receive relatively small kernel gradients during training, linking their sparsity to kernel sparsity.Source: §4.1 · p. 4

4.2. Kernel Entropy

  • Kernel entropy measures the complexity of corresponding-kernel distributions: more diverse kernels indicate more information in the input feature map.The paper links more isolated kernels to more different convolutional results and greater information in the corresponding feature map.Source: §4.2 · pp. 4–5

4.3. Definition of the KSE Indicator

  • KSE combines min-max-normalized kernel sparsity and entropy into an importance indicator, with α set to 1 in this work.The indicator is rescaled across input feature maps within each convolutional layer.Source: §4.3 · p. 5

4.4. Kernel Clustering

  • Kernel clustering retains an intermediate number of kernels per input feature map, rather than keeping or deleting all corresponding kernels.The retained count is selected from the feature-map importance score, and k-means produces that number of kernel centroids.Source: §4.4 · p. 5

5. Experiments

The experiments evaluate KSE on CIFAR-10, SVHN, and ImageNet 2012, examining compression, feature-map/kernel relationships, benchmark results, and ablations. KSE reports strong compression results and analyzes how granularity and indicator choice affect performance.

5.1. Compression and Acceleration Ratios

  • Compression accounting includes centroid storage and per-feature-map kernel indices; shared intermediate convolution results underpin theoretical acceleration, and fully connected layers are treated as 1×1 convolutions.Source: §5.1 · p. 6

5.2. Relationship between Input Feature Maps and their Corresponding Kernels

  • Input-map sparsity rises with corresponding kernel sparsity, while information richness falls as kernel entropy rises; the reported correlations are 0.833 and −0.826.Source: §5.2 · p. 6

5.3. Comparison with State-of-the-Art Methods

5.4.1 Effect of the Compression Granularity G

  • With G = 2, each feature map’s kernels are either pruned or all kept; finer-grained G permits varied kernel counts and, at G = 4, yields 0.52% higher Top-1 accuracy at the same compression ratio.Source: §5.4.1 · p. 8

5.4.2 Indicator Analysis

  • Across compression granularities, KSE performs better than kernel sparsity or kernel entropy alone, which represent receptive-field area or position information, respectively.Source: §5.4.2 · p. 8

5.5. Visualization Analysis

  • In a ResNet-50 layer visualization, lower KSE values correspond to feature maps with less bird information, while values near 1 correspond to maps containing bird and background information.Source: Figure 8 · p. 8, §5.5 · p. 8

6. Conclusion

The paper frames feature-map sparsity and information richness as signals of importance, combines kernel sparsity and entropy into KSE, and uses kernel clustering to compress networks. It reports better results than state-of-the-art methods and identifies more rigorous theoretical bounds or conditions as future work.

  • The paper identifies feature-map sparsity and information richness as key elements for evaluating feature-map importance.Source: §6 · p. 8
  • KSE combines kernel sparsity and entropy to measure feature-map importance in a feature-agnostic manner, guiding kernel clustering to reduce kernels before a few-epoch fine-tuning.Source: §6 · p. 8
  • The paper reports better results than state-of-the-art methods and proposes more rigorous theoretical proof with bounds or conditions as future work.Source: §6 · p. 8
Loading 1812.04368v2…