PaperlayerView exact revision on arXiv

Source-linked AI summary

Pruning by Explaining: A Novel Criterion for Deep Neural Network Pruning

Seul-Ki Yeom, Philipp Seegerer, Sebastian Lapuschkin, Alexander Binder, Simon Wiedemann, Klaus-Robert Müller, Wojciech Samek

arXiv:1912.08881v3cs.LGcs.NEstat.ML

TL;DR

CNNs require substantial computation and storage, while pruning needs criteria that identify removable units without sacrificing predictive performance. The paper uses LRP relevance scores to prune CNNs iteratively and finds especially strong performance in scarce-data transfer learning without fine-tuning. The method also performs favorably with retraining across datasets and architectures, subject to implementation- and variant-dependent limitations.

  • Problem

    CNN predictive performance comes with high storage and computation costs, and pruning requires a reliable way to identify units for deletion while preserving performance.

  • Method

    The paper uses Layer-wise Relevance Propagation relevance scores as a pruning criterion, removing low-relevance units iteratively with optional fine-tuning.

  • Results

    Across datasets and architectures, LRP performed favorably with and without retraining, and clearly outperformed competing criteria when fine-tuning was prohibited.

  • Takeaways & Limitations

    LRP-based pruning is especially recommended for transfer-learning settings with small target datasets and no retraining after pruning.

  • Takeaways & Limitations

    The pruning result may depend on the chosen LRP variant, and LRP is not implementation invariant.

Abstract

from arXiv · show

The success of convolutional neural networks (CNNs) in various applications is accompanied by a significant increase in computation and parameter storage costs. Recent efforts to reduce these overheads involve pruning and compressing the weights of various layers while at the same time aiming to not sacrifice performance. In this paper, we propose a novel criterion for CNN pruning inspired by neural network interpretability: The most relevant units, i.e. weights or filters, are automatically found using their relevance scores obtained from concepts of explainable AI (XAI). By exploring this idea, we connect the lines of interpretability and model compression research. We show that our proposed method can efficiently prune CNN models in transfer-learning setups in which networks pre-trained on large corpora are adapted to specialized tasks. The method is evaluated on a broad range of computer vision datasets. Notably, our novel criterion is not only competitive or better compared to state-of-the-art pruning criteria when successive retraining is performed, but clearly outperforms these previous criteria in the resource-constrained application scenario in which the data of the task to be transferred to is very scarce and one chooses to refrain from fine-tuning. Our method is able to compress the model iteratively while maintaining or even improving accuracy. At the same time, it has a computational cost in the order of gradient computation and is comparatively simple to apply without the need for tuning hyperparameters for pruning.

1. Introduction

CNNs deliver strong predictive performance but impose substantial storage and computation costs, motivating pruning criteria that preserve task-relevant performance. This paper proposes LRP-based pruning and evaluates it in fine-tuned and resource-constrained transfer-learning scenarios.

  • VGG-16 has approximately 138 million parameters, exceeds 500MB of storage, and requires 15.5 billion FLOPs per image.
  • Pruning reduces storage and computation by removing network units deemed least important for the intended task.
  • Overparameterized models may contain dormant components, unused outputs, or capacity that cannot be fine-tuned under scarce-data and resource constraints.
  • The proposed criterion uses Layer-wise Relevance Propagation to score unit contributions and identify pruning candidates.
  • Across benchmark datasets and four CNN architectures, LRP was more scalable and efficient than existing criteria with retraining, and clearly outperformed them without retraining.

2. Related Work

Related work reduces neural-network cost through quantization, decomposition, architecture design, and pruning. Pruning methods differ mainly in how they identify units whose removal should minimally affect predictive performance.

  • Quantization compresses storage by reducing the number of possible and unique parameter values.
  • Tensor decomposition estimates informative parameters through low-rank approximation or factorization of network matrices.
  • Architecture-efficiency methods target memory or runtime by replacing layers or optimizing convolution operations and architectures directly.
  • Pruning removes redundant nodes, filters, or layers while supporting training from scratch and transfer learning.
  • Existing pruning criteria include second- or first-order Taylor approximations, gradients, weight magnitudes, propagated response importance, and data-driven channel statistics.

3. LRP-Based Network Pruning

The method prunes CNN units iteratively by using LRP relevance as a global importance criterion. Relevance is propagated from model outputs through layers, allowing units to be removed with optional fine-tuning across different network structures.

  • 3. LRP-Based Network Pruning: The pruning algorithm computes unit importance, removes the least important units in groups, and repeats until the pruning threshold is reached.
  • 3.1. Layer-wise Relevance Propagation: LRP assigns each network unit a relevance value representing its contribution to the network output.
  • 3.1. Layer-wise Relevance Propagation: LRP has runtime similar to gradient backpropagation and can quantify unit importance across pooling, fully connected, and convolutional layers.
  • 3. LRP-Based Network Pruning: The pipeline performs a forward pass, propagates the output score backward with LRP, then prunes irrelevant units and optionally fine-tunes the model.
  • 3.2. LRP-based Pruning: The method uses the LRP-α1β0 rule, whose α = 1 setting isolates inference signals supporting the selected output class.
  • 3.1. Layer-wise Relevance Propagation: LRP’s conservation principle redistributes relevance layer by layer without loss or injection, supporting importance estimates across changing hidden-layer sizes.
  • 3.2. LRP-based Pruning: Pruning priorities can target convolutional layers for FLOP reduction or fully connected layers for memory reduction.

4. Experiments

Across toy and image-classification experiments, LRP generally preserves or improves post-pruning accuracy, especially when little data is available and fine-tuning is omitted. Its selections are consistent, globally adaptive, and competitive in compression, although it does not always minimize parameters or FLOPs.

  • 4.1. Pruning Toy Models: LRP consistently outperforms reference-sample criteria across toy datasets and sample sizes, except that Weight performs best with one sample per class.With five reference points per class, LRP also surpasses Weight while preserving the predictor’s functional core.
  • 4.1. Pruning Toy Models: LRP preserves functionally important components with little data and is less sensitive to reference-sample choice than Gradient and Taylor criteria.Gradient and Taylor do not reach LRP’s performance even with 200 reference samples per class.
  • 4.1. Pruning Toy Models: LRP’s neuron selections are largely distinct from other criteria while remaining highly consistent across reference-sample seeds.Gradient and Taylor show lower consistency, particularly for units preserved longer during pruning, helping explain their greater result variation.
  • 4.2. Pruning Deep Image Classifiers for Large-scale Benchmark Data: With fine-tuning, LRP achieves higher test accuracy than competing criteria in a large majority of VGG-16, AlexNet, ResNet-18, and ResNet-50 cases across datasets.The results indicate stable performance that is largely independent of the chosen dataset.
  • 4.2. Pruning Deep Image Classifiers for Large-scale Benchmark Data: LRP does not always yield the smallest parameter count or FLOPs, but consistently identifies removals and preservations associated with the best post-pruning model performance.FLOPs depend on pruning location: LRP and Weight focus more on upper layers, whereas Taylor and Gradient prune more lower-layer filters.
  • 4.2.1. Scenario 1: Pruning with Fine-tuning: LRP can produce initial accuracy gains up to approximately 30% pruning on Event 8, Oxford Flower 102, and CIFAR-10, rather than only gradual degradation.The paper relates this behavior to removing filters associated with source-dataset classes absent from the target dataset.
  • 4.2.2. Scenario 2: Pruning without Fine-tuning: Without fine-tuning, LRP-pruned models vastly outperform competing criteria in the resource-constrained transfer-learning scenario.For three-class ILSVRC tasks, the reported LRP-versus-competitor differences for convolutional-layer pruning reach 9.6–63.6 percentage points, depending on model and criterion.
  • 4.2. Pruning Deep Image Classifiers for Large-scale Benchmark Data: LRP applies to fully connected, convolutional, and pooling layers, and can measure weight or filter importance globally without additional normalization.Its relevance-conservation principle naturally normalizes the criterion and helps preserve important network substructures and bottlenecks.

5. Discussion

Across datasets, architectures, and settings, LRP generally matched or outperformed competing pruning criteria, especially when scarce resources prevented fine-tuning. Its advantages include preserving predictive behavior with few reference samples, though outcomes can depend on implementation and LRP variant.

  • LRP consistently performed well across datasets, model architectures, and experimental settings, often outperforming competing criteria.
  • Without fine-tuning, LRP-pruned models vastly outperformed competitors on image and toy data, while preserving the original prediction function in toy settings.
  • At fixed filter-removal targets, LRP may not produce the cheapest sub-network by parameter count or FLOPs, but it identifies components yielding the best post-pruning performance.
  • LRP’s continuous importance measure supports stable pruning with relatively few reference samples, unlike the volatility observed for gradient and Taylor criteria.
  • LRP has computational cost comparable to Taylor and gradient criteria but requires fewer reference samples, lowering practical cost when data are scarce.
  • The pruning outcome may depend on the chosen LRP variant, and LRP is not implementation invariant because network structure can affect criterion computation.

6. Conclusion

The paper introduces LRP-based iterative CNN pruning, linking model interpretability with compression. Across datasets, LRP performs favorably with and without retraining, especially for small target datasets without retraining.

  • LRP relevance scores quantify each weight or filter’s contribution to the network output, providing a pruning criterion grounded in model explanations.Low-relevance units can be removed without further postprocessing such as per-layer normalization.
  • LRP-based pruning showed favorable compression performance across varied datasets both with and without retraining.
  • The same relevance-based method can produce intuitive heatmaps for visually interpreting individual model decisions.
  • Future work proposes using heatmaps to identify image features affected by pruning and help avoid undesired Clever Hans phenomena.

Supplementary Methods 1: Data Preprocessing

The supplementary experiments use transfer-learning datasets with stated image preprocessing and train/test configurations. ImageNet classes are also sampled for a small-sample pruning scenario.

  • During fine-tuning, images are resized to 256×256, randomly cropped to 224×224, and horizontally flipped with 50% probability.Testing uses images resized to 224×224 pixels.
  • Scene 15 contains about 4,485 images across 15 natural-scene categories, with 20% of each class used for fine-tuning.Initial Top-1 accuracies are reported for four transferred CNN architectures.
  • Event-8 contains eight sports-event categories, using 40% of images for fine-tuning and 60% for testing.
  • Cats and Dogs includes 4,000 dog-training images, 4,005 cat-training images, and 2,023 test images.Oxford Flowers 102 contains 102 flower categories, with over 2,000 training and 6,100 test images.
  • CIFAR-10 has 50,000 training and 10,000 test images across 10 object categories, with 32×32 images resized to 224×224.
  • For the ImageNet small-sample scenario, models are pruned and tested on randomly selected k = 3 classes from 1,000 classes.

Supplementary Results 1: Additional results on toy data

Toy-data experiments examine how reference-sample size affects LRP neuron-selection consistency. The reported comparisons use fixed sample-size grids and many random-seed combinations.

  • Supplementary Table 1 compares training accuracy after one-shot pruning one third of filters using Weight, Gradient, Taylor, and LRP criteria.Reference-sample counts are n ∈[1, 5, 20, 100].
  • For the moon and circle datasets, neuron-set consistency generally increases with larger reference sets before reaching a plateau.
  • For the mult toy dataset, set similarity decreases minimally for m ≥10, indicating some deviation from the neurons selected with n = 10.
  • LRP neuron-selection consistency is compared using n = 10 reference samples against m ∈[1, 2,5, 10, 20, 50, 100, 200] samples per class.The comparison reports k = 250 and averages over 1225 unique random seed combinations.

Supplementary Results 2: Additional results for image processing neural networks

Additional image experiments extend pruning comparisons across architectures, datasets, and retraining settings. Without fine-tuning, LRP consistently removes the largest filter fraction before accuracy declines on composed binary tasks.

  • Additional AlexNet and ResNet-18 results indicate that favorable LRP pruning performance is not limited to one network architecture.CIFAR-10 shows greater robustness at higher pruning rates.
  • Supplementary Figure 1 compares LRP with other criteria across pruning rates on AlexNet and ResNet-18 over five datasets.Scatter-plot points represent paired criterion performance at specific pruning rates.
  • Supplementary Figure 2 compares test accuracy as pruning increases, with fine-tuning, and marks premature termination when vital filters disconnect model input from output.
  • Without fine-tuning after pruning, all tested criteria remove some filters without notable discrimination loss on three composed binary problems.
  • Supplementary Table 3 compares Weight, Taylor, Gradient with ℓ2-norm, LRP, and an Unpruned model using accuracy, loss, parameters, and FLOPs.
  • Supplementary Figure 3 averages results over 20 repetitions using randomly drawn samples for ResNet-50 pruning without subsequent fine-tuning.
  • LRP pruning consistently removes the largest fraction of filters before prediction accuracy begins to decline, while weight-based pruning performs second best.
  • The composed datasets retain less redundant capacity than Cats versus Dogs, consistent with their higher variance.
Loading 1912.08881v3…