Source-linked AI summary
Importance Estimation for Neural Network Pruning
Pavlo Molchanov, Arun Mallya, Stephen Tyree, Iuri Frosio, Jan Kautz
TL;DR
Large CNNs can be difficult to deploy under computational, latency, and energy constraints. This paper estimates filter contributions with Taylor-based criteria, finding that first-order approximation performs strongly and works best after batch normalization.
Problem
Growing CNN sizes create deployment challenges for mobile, embedded, and cloud settings where computation, latency, and energy matter.
Method
The method estimates each filter’s squared loss change using Taylor approximations and gradients available during backpropagation, then iteratively removes the least important filters.
Results
First-order approximation shows significant agreement with true importance and outperforms prior work across a range of deep networks.
Takeaways & Limitations
The first-order criterion offers a practical pruning method with minimal overhead, globally consistent scaling across layers, and applicability to skip connections.
Takeaways & Limitations
The Hessian-based approach assumes a positive definite Hessian after convergence, although minibatch stochasticity and saddle points can violate this assumption.
Abstract
from arXiv · showhide
Structural pruning of neural network parameters reduces computation, energy, and memory transfer costs during inference. We propose a novel method that estimates the contribution of a neuron (filter) to the final loss and iteratively removes those with smaller scores. We describe two variations of our method using the first and second-order Taylor expansions to approximate a filter's contribution. Both methods scale consistently across any network layer without requiring per-layer sensitivity analysis and can be applied to any kind of layer, including skip connections. For modern networks trained on ImageNet, we measured experimentally a high (>93%) correlation between the contribution computed by our methods and a reliable estimate of the true importance. Pruning with the proposed methods leads to an improvement over state-of-the-art in terms of accuracy, FLOPs, and parameter reduction. On ResNet-101, we achieve a 40% FLOPS reduction by removing 30% of the parameters, with a loss of 0.02% in the top-1 accuracy on ImageNet. Code is available at https://github.com/NVlabs/Taylor_pruning.
NVIDIA
The passage identifies the authors by their NVIDIA email addresses.
- The listed contact addresses use the nvidia.com domain.
1. Introduction
Large CNNs create deployment costs, motivating structural pruning. The proposed Taylor-based criterion estimates neuron contributions efficiently and supports globally consistent pruning across layers, including skip connections.
- Growing CNN size can conflict with mobile and embedded resource limits, while cloud inference also faces latency and energy concerns.
- Structural pruning removes filters or neurons and their associated computations to produce more compact networks.
- Weight magnitude can poorly reflect importance, motivating a greedy criterion aimed at the gap between weight-based and empirically optimal pruning decisions.
- The method defines importance as the squared loss change from removing a filter and approximates it with Taylor expansions using gradients available during training.
- The criterion requires little overhead, has globally consistent scale without per-layer sensitivity analysis, computes scores in parallel, and applies to skip connections.
- >93% Spearman correlation with oracle importance was measured on ImageNet-trained ResNets and DenseNets, alongside improved pruning results across CIFAR-10 and ImageNet networks.
2. Related work
Prior pruning approaches include distillation, greedy search, regularization, batch-normalization parameters, and magnitude-based criteria, each with computational, architectural, or assumption-related limitations.
- Network distillation trains a smaller model to mimic a larger model but requires defining the smaller architecture beforehand.
- Structural pruning can use predefined per-layer ratios or simultaneous global pruning, with the latter able to find a better architecture.
- Regularization-based methods relax ℓ0 minimization with ℓ1 or ℓ2 penalties and threshold parameters or filter norms.
- Batch-normalization pruning regularizes scaling factors and thresholds them, optionally adding FLOPS-based penalties to target computational costs.
- Magnitude-based pruning assumes parameter magnitude correlates with importance, whereas Hessian-informed criteria were proposed to improve on magnitude-only decisions.
- The positive-definite Hessian assumption can fail under minibatch stochasticity, limited observations, and saddle points, allowing some neuron removals to decrease loss.
- The proposed method instead approximates squared loss differences without requiring every neuron contribution to be positive, using first- or second-order expansions.
3. Method
The method formulates pruning as iteratively removing parameters with the smallest loss-based importance, approximated efficiently using Taylor expansions and gradient information. It extends this procedure to structured parameters and supports layer-wide scoring, grouped contributions, and iterative fine-tuning.
- Importance estimation: The method starts from a trained network and incrementally removes parameters using a greedy first-order search that approximates full combinatorial pruning.Parameters are removed a few at a time after convergence, assuming parameter independence for the simplified search.
- Importance estimation: Exact parameter importance is defined by the squared error change caused by removing a parameter, but computing it requires evaluating a separate network for each parameter.This makes direct computation expensive for large networks.
- Importance estimation: Taylor expansions approximate removal importance near the trained parameters, using gradients for the first-order criterion and Hessian terms for the second-order criterion.The first-order approximation is more compact, while the second-order form incorporates Hessian information.
- Structured pruning: Structured importance can be computed either as a group contribution or by summing individual parameter contributions, with gates providing an equivalent filter-level formulation.Gates are fixed at one, excluded from optimization, and combine filter-weight and bias contributions.
- Interpretation: The first-order importance score can also be interpreted as a variance estimate and as the diagonal of the Fisher information matrix.This interpretation follows when the error is related to log-likelihood and the expected Fisher information is considered.
- Iterative pruning: The pruning loop averages neuron or filter importance over minibatches, removes the least important units in batches, and continues fine-tuning until the target pruning level is reached.Importance is computed from minibatch-averaged gradients, with second-order estimates used when Hessian information is available.
4. Experiments
Experiments on CIFAR-10 and ImageNet show that Taylor-based criteria closely track oracle importance and support effective iterative pruning across architectures, including skip connections. The first-order criterion is substantially faster than the second-order variant while achieving comparable results, and ImageNet pruning reduces computation and parameters with small accuracy losses.
- LeNet3: Taylor FO and Taylor SO prune nearly the same number of LeNet3 neurons as the Oracle before the loss reaches 1.0.Weight-based pruning performs about as poorly as random pruning in this setting.
- LeNet3: The Greedy oracle closely matches the Combinatorial oracle for small network changes while being exponentially faster to compute.Subsequent experiments therefore use the Greedy oracle as the best-outcome reference.
- ResNet-18: Placing gates after batch normalization substantially improves correlation, while Taylor SO ranks highest and Taylor FO follows closely.Using the full gradient reduces the batch-normalization placement effect but produces lower overall correlation.
- ResNet-18: Reestimating criteria after each pruning iteration improves adaptation to the changing network, whereas fixed criteria perform significantly worse.The ResNet-18 comparison is conducted without fine-tuning.
- ResNet-18: Taylor criteria have very high correlation with oracle neuron rankings on CIFAR-10, and first- and second-order variants are comparable.Taylor FO is selected for larger ImageNet experiments because it is faster and uses less memory.
- ImageNet: >93% Spearman correlation is obtained across all three ImageNet networks, while skip connections also show high correlation.The adopted Taylor-FO-BN variant prunes channels in skip connections and bottleneck layers simultaneously.
- ImageNet: 40% FLOPs and 30% of parameters are removed from ResNet-101 with only a 0.02% accuracy loss on ImageNet.The proposed method also reports smaller errors and fewer GFLOPs than BN-ISTA for ResNet-101.
- ImageNet: Removing 76% of VGG11-BN parameters causes only a 0.19% accuracy loss and improves previously reported results.The reported improvements are 0.65% over one prior result and more than 2% over another.
5. Conclusions
The paper proposes Taylor-expansion criteria for estimating neuron contributions and finds that the first-order criterion performs strongly across deep networks, especially when applied after batch normalization.
- Taylor expansion estimates a neuron's contribution from the squared loss change induced by removing it.The method uses first- and second-order approximations to estimate importance rather than computing the exact loss change.
- First-order Taylor importance agrees significantly with true importance and outperforms prior work across diverse deep networks.
- Applying the first-order criterion after batch normalization gives the best results under practical computational and memory constraints.
6. Supplementary material
The supplementary experiments examine pruning schedules, architectures, datasets, oracle reliability, and inference speed. They show advantages for iterative pruning, Taylor-based criteria, and selected skip-connection pruning settings.
- ResNet20 on CIFAR10: Taylor first- and second-order criteria outperform BN-ISTA on ResNet20 pruning experiments, while first-order Taylor after batch normalization is a reasonable residual-network choice.Random and magnitude criteria perform worst in the reported comparison, and the proposed method is reported as unaffected by the pruning paradox.
- Pruning schedules: Iterative pruning removes 100 neurons every 30 mini-batch updates until a predefined neuron count is reached.The supplementary material also defines single-step and continuous schedules, with continuous pruning triggered by a loss threshold of 1.04.
- Pruning schedules: Iterative pruning clearly outperforms other schedules across all epochs when pruning 10000 of 20096 ResNet-101 neurons on ImageNet.The iterative setting corresponds to TaylorFO-BN-50% in the main paper.
- Inference speed: 1.59× speedup results from removing 33% of FLOPs in Taylor-FO-BN-22%, whereas removing 68% of FLOPs yields 1.51× speedup in Taylor-FO-BN-50%.The reported inference measurements use pruned ResNet-101 models and show larger speed reductions for larger batch sizes.
- Oracle validation: 95.67% correlation between training-set and test-set oracles supports the representativeness of the correlation study.Recomputing the table with the test-set oracle caused an average raw-entry deviation of 0.04 and no method reordering.