PaperlayerView exact revision on arXiv

Source-linked AI summary

Importance Estimation for Neural Network Pruning

Pavlo Molchanov, Arun Mallya, Stephen Tyree, Iuri Frosio, Jan Kautz

arXiv:1906.10771v1cs.LGcs.CVstat.ML

TL;DR

Computing a filter’s exact contribution to loss is expensive in large networks. The paper approximates that contribution with Taylor expansions to prune less important filters, reporting significant agreement with true importance and better performance than prior work across deep networks.

  • Problem

    Computing the exact loss contribution of individual filters is prohibitively expensive for large networks, motivating a practical importance estimate.

    Source: §1 · p. 1

  • Method

    The method approximates the squared loss change from removing a filter with first- or second-order Taylor expansions, then iteratively removes filters with lower estimated importance.

    Source: §1 · pp. 1–2

  • Results

    The first-order approximation shows significant agreement with true importance and outperforms prior work across a range of deep networks.

    Source: §1 · p. 2, §5 · p. 8

  • Takeaways & Limitations

    Applying the first-order criterion after batch normalization yields the best results under practical computational and memory constraints.

    Source: §5 · p. 8

    A Hessian-based approach assumes that removing any neuron increases loss, but minibatch stochasticity and saddle points can violate that assumption.

    Source: §2 · p. 2

Abstract

from arXiv · show

Structural pruning of neural network parameters reduces computation, energy, and memory transfer costs during inference. We propose a novel method that estimates the contribution of a neuron (filter) to the final loss and iteratively removes those with smaller scores. We describe two variations of our method using the first and second-order Taylor expansions to approximate a filter's contribution. Both methods scale consistently across any network layer without requiring per-layer sensitivity analysis and can be applied to any kind of layer, including skip connections. For modern networks trained on ImageNet, we measured experimentally a high (>93%) correlation between the contribution computed by our methods and a reliable estimate of the true importance. Pruning with the proposed methods leads to an improvement over state-of-the-art in terms of accuracy, FLOPs, and parameter reduction. On ResNet-101, we achieve a 40% FLOPS reduction by removing 30% of the parameters, with a loss of 0.02% in the top-1 accuracy on ImageNet. Code is available at https://github.com/NVlabs/Taylor_pruning.

1. Introduction

The introduction motivates compact networks for resource-constrained deployment and presents a Taylor-based criterion for estimating filter contributions to loss. It reports strong agreement with greedy-search estimates and improved pruning results across benchmark networks.

Introduction

  • Compact networks can benefit mobile and embedded deployments with limited compute, as well as cloud services where prediction latency and energy consumption matter.Pruning derives compact networks by removing structural parameters and their associated computations.Source: §1 · p. 1
  • The method estimates a filter’s importance as the squared loss change from removing it, approximates that change with a Taylor expansion, and iteratively removes the least important filters.The criterion uses parameter gradients readily available during standard training and adds minimal computational overhead.Source: §1 · p. 1
  • The criterion requires no hyperparameters beyond the desired pruning count, has a globally consistent scale across layers, supports parallel scoring, and applies to any layer, including skip connections.These properties avoid per-layer sensitivity analysis and greedy layer-by-layer computation.Source: §1 · pp. 1–2
  • On ImageNet ResNets and DenseNets, both Taylor variants show >93% estimated Spearman correlation with the greedy-search oracle; pruning across CIFAR-10 and ImageNet networks improves over state of the art.The authors also report that the first-order criterion is significantly faster to compute, with slightly worse accuracy than the second-order variant.Source: §1 · p. 2
  • The authors report that squared loss contribution yields better oracle correlation and pruning accuracy than signed difference.They compare first- and second-order Taylor variants against a greedy-search oracle.Source: §1 · p. 2

2. Related work

Prior pruning work often relies on magnitude proxies or layer-specific sensitivity, while this paper motivates a contribution criterion that allows negative contributions and supports globally comparable pruning. Earlier approaches also include distillation, regularization, thresholding, and empirical importance estimation.

Related work

  • Network distillation trains a smaller model to mimic a larger model’s output [14], but requires defining the smaller architecture beforehand.Source: §2 · p. 2
  • Pruning relaxes ℓ0 minimization with Bayesian methods [25] [29] or regularization, while ℓ0 optimization itself is non-convex, NP-hard, and combinatorial.Structural pruning can use predefined per-layer ratios or prune across layers simultaneously to find an architecture automatically [24].Source: §2 · p. 2
  • Prior work used weight decay and energy minimization or positive monotonic energy penalties to encourage sparse network solutions.Source: §2 · p. 2
  • Magnitude-based approaches include ISTA soft-thresholding [3], weight pruning, filter pruning by ℓ1 norm [22], and batch-norm scaling thresholds [23] [33] [16].Batch-norm approaches can add FLOPS-based penalties [5]; a general ISTA-like scaling-factor method [16] applies to any layer.Source: §2 · p. 2
  • LeCun et al. measure importance with the Hessian’s diagonal times squared weight, but this assumes a positive-definite Hessian; our criterion permits negative contributions and uses first- or second-order approximations.Because neuron removal can decrease loss, the method approximates the squared loss difference without requiring every neuron’s contribution to be positive.Source: §2 · p. 2
  • Luo et al. [26] and Yu et al. [34] estimate importance empirically, but incur computational or implementation costs; our method outperforms them with little additional computation and engineering.Luo et al. use greedy per-layer reconstruction-loss minimization; Yu et al. propagate linear-classifier feature importance under Lipschitz continuity.Source: §2 · pp. 2–3
  • Unlike methods requiring sensitivity analysis or global-scaling heuristics that fail with skip connections and use more activation memory, our criterion scales globally and applies to any layer, including skip connections.Batch-normalization-based methods also avoid sensitivity analysis but are limited to batch-norm layers.Source: §2 · p. 3
  • Pruning has also served as a training regularizer: Han et al. [6] reinitialize and fine-tune pruned weights, while He et al. [11] continue training zeroed filters.The paper focuses on removing filters but notes pruning might be extended as a regularizer.Source: §2 · p. 3

3. Method

The method greedily estimates parameter or filter importance with Taylor approximations, then prunes low-scoring neurons or filters during iterative fine-tuning. It uses gates to estimate structural contributions and specifies implementation choices for score aggregation and pruning.

3. Method

  • Greedy first-order search removes parameters incrementally by importance, while Taylor approximations avoid evaluating a separate network for every removal.The first-order score uses gradients available from backpropagation; a second-order approximation is also described.Source: §3 · p. 3
  • For structural parameter sets such as filters, gates simplify importance computation and make gate importance equivalent to the preceding layer’s group contribution.Gates have constant value, are not optimized, and implicitly combine filter-weight and bias contributions.Source: §3 · pp. 3–4
  • Under the stated assumptions, the first-order metric is a variance estimate and the diagonal of the Fisher information matrix, consistent with a conclusion in [32].The interpretation is derived for a loglikelihood error function.Source: §3 · p. 4

3.1. Pruning algorithm

  • Each epoch updates weights and computes neuron or filter importance from minibatch gradients, then removes the N lowest-scoring units after a predefined number of minibatches.Pruning and fine-tuning continue until the target is reached or the maximum tolerable loss can no longer be achieved.Source: §3.1 · p. 4

3.2. Implementation details

  • Implementation uses summed individual contributions unless gates compute group contributions; second-order scores use gates, and ImageNet experiments omit the Hessian for memory constraints.Gates are placed after batch normalization unless otherwise stated, and importance scores are averaged across pruning iterations with an exponential moving average coefficient of 0.9.Source: §3.2 · p. 4
  • Pruning in batches followed by fine-tuning performed better than continuous pruning; iterations typically remove around 2% of initial filters.The method uses 10 minibatches for CIFAR datasets and 30 minibatches with batch size 256 for ImageNet pruning.Source: §3.2 · p. 4

4. Experiments

Experiments on CIFAR-10 and ImageNet evaluate Taylor-based pruning across network architectures, comparing importance estimates with oracle rankings and assessing pruning outcomes. Taylor FO is used for larger ImageNet experiments because it is faster and uses less memory than Taylor SO.

4.1.1 LeNet3

  • On LeNet3, Taylor SO and Taylor FO nearly match the oracle’s pruning performance, while weight-magnitude pruning performs about as poorly as random pruning.The greedy oracle closely tracks the computationally exponential combinatorial oracle for small pruning changes; Taylor FO is comparable to Taylor SO in this setting.Source: §4.1.1 · p. 5, Figure 3 · p. 5, Figure 2 · p. 5

4.1.2 ResNet-18

  • On ResNet-18, Taylor SO has the highest oracle correlation, followed by Taylor FO; placing gates after BN2 substantially improves correlation over placing them before BN2.For Taylor FO on conv1, correlation drops from 0.95 to 0.28 when the gate is placed before BN2. Re-estimating criteria after each pruning iteration outperforms fixed criteria; OBD can be unstable without fine-tuning.Source: Table 1 · p. 6, §4.1.2 · p. 6

4.2.1 Neuron importance correlation study

  • Taylor FO achieves >93% Spearman correlation across ResNet-101, VGG11-BN, and DenseNet201, and >92% Pearson correlation on ResNet and DenseNet.The method also retains high correlation when skip connections are included, motivating simultaneous pruning of skip-connection channels and bottleneck layers.Source: Table 2 · p. 7, §4.2.1 · p. 7

4.2.2 Pruning and fine-tuning

  • Removing 40% of ResNet-101 FLOPs and 30% of its parameters costs just 0.02% accuracy; the method also uses at least 1.22 fewer GFLOPs than BN-ISTA [33].Pruning results are reported for ImageNet validation; pruning only skip connections yields larger errors but a faster network.Source: §4.2.2 · p. 7, Table 3 · p. 8
  • On ResNet-50 and ResNet-34, pruning shows significant improvements over other methods; on VGG11-BN, removing 76% of parameters costs 0.19% accuracy.A single-step ResNet-101 prune yields 25.3% top-1 error, 0.68% higher than Taylor-FO-BN-50%; DenseNet-201 has 1.79% lower error than DenseNet-121 at the same FLOPs.Source: §4.2.2 · p. 8

5. Conclusions

The paper proposes Taylor-based estimation of a neuron's contribution by approximating the squared loss change caused by its removal. It finds that the first-order approximation agrees significantly with true importance and that applying the criterion after batch normalization gives the best results under practical computational and memory constraints.

Conclusions

  • Applying the first-order criterion after batch normalization yields the best results under practical computational and memory constraints.The conclusion identifies this placement as the outcome of extensive analysis.Source: §5 · p. 8
  • The proposed Taylor-based method estimates a neuron's contribution by approximating the squared change in loss induced by removing it.Source: §5 · p. 8
  • The first-order approximation shows significant agreement with true importance and outperforms prior work across a range of deep networks.Source: §5 · p. 8

6. Supplementary material

The supplementary material reports ResNet20/CIFAR10 pruning results, compares ResNet101/ImageNet pruning schedules and inference speed, and checks the robustness of the oracle analysis.

6.1. ResNet20 on CIFAR10

  • The ResNet20/CIFAR10 comparison used a 92%-accurate initial model and removed 10 neurons every 30 mini-batches during pruning.The initial model was trained for 200 epochs; pruning and fine-tuning ran for 70 epochs, with the learning rate decayed by 10 every 20 epochs.Source: §6.1 · p. 10
  • First- and second-order Taylor pruning outperform BT-ISTA [33] on ResNet20/CIFAR10, while first-order Taylor after BN is a reasonable choice unaffected by the pruning paradox [24].Optimal Brain Damage and the proposed methods differ by less than one standard deviation; the table reports averages over 10 seeds, pruning only the first layer in each residual block.Source: §6.1 · p. 10, Table 4 · p. 11

6.2. Additional details on ResNets pruning

  • Iterative pruning clearly outperforms single-step and continuous schedules across epochs for ResNet-101 on ImageNet.Each schedule could prune up to 10,000 of 20,096 neurons; iterative pruning corresponds to TaylorFO-BN-50% in the main paper. Single-step removes all neurons at once, whereas continuous pruning removes 100 every 30 mini-batches only when loss exceeds 1.04.Source: §6.2 · p. 10, Figure 6 · p. 10
  • 1.59× speedup followed removal of 33% of FLOPs with Taylor-FO-BN-22%, while 1.51× followed removal of 68% with Taylor-FO-BN-50%.Pruning skip connections reduced inference time more than pruning all layers; batch inference time was measured on an NVIDIA Tesla V100.Source: §6.2 · p. 10, Table 5 · p. 11

6.3. Oracle computation details

  • 95.67% correlation between training-set and test-set oracles accompanied little change in Table 2 entries and no reordering of methods.Recomputing Table 2 with the test-set oracle produced an average deviation of 0.04 between raw table entries.Source: §6.3 · p. 10

Related papers

Loading 1906.10771v1…