PaperlayerView exact revision on arXiv

Source-linked AI summary

Provable Filter Pruning for Efficient Neural Networks

Lucas Liebenwein, Cenk Baykal, Harry Lang, Dan Feldman, Daniela Rus

arXiv:1911.07412v2cs.LGstat.ML

TL;DR

Large neural networks are costly to deploy, while existing pruning methods often lack guarantees and require heuristic tuning. The paper introduces data-informed sensitivity sampling for filter pruning, with theoretical size and performance guarantees. Across architectures and datasets, it produces sparser, more efficient models with minimal accuracy loss relative to competing filter-pruning methods.

  • Problem

    Existing pruning approaches can lack guarantees on pruned-network size and performance and may require cumbersome ablation studies or manual hyper-parameter tuning.

  • Method

    The method scores filters using sensitivity measured on a small input batch, samples filters proportionally to those scores, and uses the resulting bounds to allocate sampling budgets across layers.

  • Results

    The method consistently generates sparser and more efficient models with minimal accuracy loss compared with state-of-the-art filter-pruning approaches.

  • Takeaways & Limitations

    The approach provides a data-informed, broadly applicable filter-pruning procedure with theoretical guarantees and minimal hyper-parameter tuning.

  • Takeaways & Limitations

    The theoretical sensitivity guarantees rely on a mild assumption about the activation distribution that is satisfied by a wide class of distributions.

Abstract

from arXiv · show

We present a provable, sampling-based approach for generating compact Convolutional Neural Networks (CNNs) by identifying and removing redundant filters from an over-parameterized network. Our algorithm uses a small batch of input data points to assign a saliency score to each filter and constructs an importance sampling distribution where filters that highly affect the output are sampled with correspondingly high probability. In contrast to existing filter pruning approaches, our method is simultaneously data-informed, exhibits provable guarantees on the size and performance of the pruned network, and is widely applicable to varying network architectures and data sets. Our analytical bounds bridge the notions of compressibility and importance of network structures, which gives rise to a fully-automated procedure for identifying and preserving filters in layers that are essential to the network's performance. Our experimental evaluations on popular architectures and data sets show that our algorithm consistently generates sparser and more efficient models than those constructed by existing filter pruning approaches.

1 INTRODUCTION

The paper targets efficient neural-network deployment by pruning redundant CNN filters while preserving predictive accuracy. It introduces a data-informed, sampling-based method with theoretical guarantees and automated allocation of filters across layers.

  • Modern neural networks impose substantial memory and computational demands, making deployment on resource-limited platforms difficult.
  • Existing pruning methods often rely on heuristics, lack guarantees on pruned-model size and performance, or require ablation studies and manual tuning.
  • The method estimates filter sensitivity from a small batch of input points and samples filters with probabilities proportional to their sensitivities.
  • The resulting importance-sampling scheme provides theoretical guarantees while automatically allocating samples to preserve critical network structures.
  • Compared with state-of-the-art filter-pruning approaches, the method produces sparser and more efficient models with minimal accuracy loss.

2 SAMPLING-BASED FILTER PRUNING

The pruning procedure estimates feature-map importance from sampled inputs, uses sensitivity-weighted sampling to remove channels and corresponding filters, and controls approximation error probabilistically. Its bounds quantify layer-wise sampling requirements and extend to guarantees for the compressed network.

  • The algorithm uses empirical sensitivity from a small input set to construct a low-variance importance-sampling distribution over feature maps.
  • Pruning sampled channels in the next layer simultaneously removes their corresponding filters in the current layer, reducing network size.
  • Reweighted channel samples form an unbiased estimator of the original layer output, so the expected sampled pre-activation equals the true pre-activation.
  • Uniform sampling can require linear sampling complexity in the number of filters, whereas sensitivity-based sampling is designed to reduce the variance and concentration requirements.
  • Under the stated activation-distribution assumption, the method preserves each layer’s output entry-wise within a multiplicative factor of 1 ± ε with probability at least 1 − δ.
  • The layer-wise guarantees propagate through the network to bound the size and performance of the final compressed model.

3 RELATIVE LAYER IMPORTANCE

The method allocates the sampling budget across layers using bounds that connect layer compressibility with output importance. A binary search selects layer sizes so the maximum layer-wise pruning error remains at most ε.

  • Budget allocation: Uniformly allocating the sampling budget can be suboptimal because layers differ in their importance and compressibility.The section frames the practical problem as distributing a fixed budget across layers to minimize pruned-model error.
  • Budget allocation: Sampling complexity bounds quantify each layer’s compressibility and thereby capture its relative importance to the network output.Less important layers are expected to be more compressible, whereas influential layers are harder to compress.
  • Allocation procedure: A binary search adjusts ε until the layer-wise sample counts sum to the user-specified budget N.The procedure repeatedly narrows the search interval according to whether the current allocation exceeds or falls below N.
  • Allocation guarantee: The resulting allocation ensures that the maximum layer-wise error incurred by pruning is at most ε.In the VGG example, early layers receive relatively more samples, later layers fewer, and the final layer receives a large share because it directly influences the output.

4 RESULTS

Across LeNet, CIFAR-10, ImageNet, and a lightweight regression network, the method produces compact models while retaining predictive performance. It also automates layer-wise allocation and requires only the pruning ratio and failure probability as practical inputs.

  • Experimental scope: The experiments compare the proposed method with Filter Thresholding, SoftNet, and ThiNet across multiple architectures and data sets.The evaluation includes LeNet, CIFAR-10 CNNs, ImageNet CNNs, and a lightweight real-time regression network.
  • LeNet architectures on MNIST: The proposed algorithm generates significantly sparser LeNet models than competing filter-pruning approaches while preserving commensurate accuracy.On LeNet-5, it reaches an approximately 90% prune ratio, with favorable performance becoming more pronounced near a 95% prune ratio.
  • CNNs on CIFAR-10: On CIFAR-10, the method achieves the most sparse models with minimal predictive-power loss and also reduces inference-time complexity.The reported Flops ratios indicate efficiency gains beyond sparsity measured by total parameter count.
  • CNNs on ImageNet: The method follows similar trends on ImageNet and scales to larger data sets without manual hyperparameter tuning.The ImageNet evaluation considers ResNet18, ResNet50, and ResNet101 under iterative pruning settings.
  • Discussion: The method consistently outperforms competing approaches on the lightweight DeepKnight regression network, even without fine-tuning.This isolates the performance of the core pruning procedure in the reported comparison.
  • Automation: In practice, the algorithm requires only the pruning ratio and failure probability because it automatically assigns each layer’s sample count.The experiments use iterative prune-retrain procedures and report pruning and FLOPS reductions at commensurate accuracy.

5 CONCLUSION

The paper presents a data-informed filter-pruning algorithm with theoretical guarantees on the size and performance of the resulting network. Its structured subnetworks are broadly applicable and empirically effective across architectures and data sets.

  • Conclusion: The method is presented as the first filter-pruning algorithm with theoretical guarantees on the generated network’s size and performance.The conclusion contrasts these guarantees with prior approaches that are generally data-oblivious, heuristic, or tuning-intensive.
  • Conclusion: Empirical evaluations demonstrate practical effectiveness in obtaining sparse, efficient networks across varying architectures and data sets.The conclusion also identifies potential use as a sub-procedure for lottery-ticket identification and efficient architecture search.
  • Conclusion: Unlike weight pruning, filter pruning produces slimmer subnetworks that reduce storage requirements and improve inference-time performance on standard hardware.Weight pruning can create irregular sparsity requiring specialized libraries or hardware, whereas filter removal is structured.
  • Conclusion: Prior neuron- and filter-pruning methods generally lack guarantees and principled layer-wise budget allocation, while this method uses data-informed saliency and theoretical error bounds.The bounds automatically identify important layers and allocate the user-specified pruning budget across them.
  • Conclusion: The approach applies to both fully connected and convolutional networks and generates sparse subnetworks that accelerate inference.This extends provable sampling-based pruning beyond methods limited to fully connected networks and non-structured sparsity.

B ALGORITHMIC AND ANALYTICAL DETAILS

The algorithm assigns empirical sensitivity to features using sampled inputs, builds an importance distribution, and uses reweighted sampling to preserve layer outputs with probabilistic guarantees. Its analysis bounds estimator bias, magnitude, variance, and network approximation error under an input-distribution assumption.

  • Sampling procedure: Algorithm 2 prunes features by sampling neurons or convolutional channels from an importance distribution and constructing reduced weight tensors layer by layer.The procedure accepts a trained network, layer, input sample, relative error ε, and failure probability δ, then samples indices and reweights them for unbiasedness.
  • Sensitivity construction: Empirical edge sensitivity separates positive and negative contributions, and neuron sensitivity aggregates these contributions to quantify each feature’s importance.The analysis treats positive and negative parts separately, while the neuron-sensitivity definition extends empirical sensitivity from edges to neurons.
  • Concentration analysis: Under Assumption 1, a high-probability event bounds each sampled contribution, enabling concentration control through bounds on magnitude and variance.The event holds with high probability over inputs; conditioned on it, the sampled variables satisfy explicit magnitude and variance bounds.
  • Guarantee derivation: Bernstein’s inequality and a union bound convert the per-sample bounds into simultaneous layer-output approximation guarantees.The proof applies Bernstein’s inequality to the centered estimator and then unions over neurons in the layer.

B.1 BOOSTING SAMPLING VIA DETERMINISTIC CHOICES

The boosting variant keeps selected high-sensitivity features deterministically and samples the remainder from a renormalized distribution. The resulting estimator remains unbiased while potentially retaining fewer unique features for the same approximation guarantee.

  • Hybrid sampling: The hybrid scheme deterministically retains k feature maps and samples m′ remaining indices from a distribution that removes the deterministic indices’ probability mass.The deterministic set is chosen from the feature indices, while the residual distribution is normalized over indices outside that set.
  • Estimator properties: The hybrid estimator remains unbiased because the deterministic contribution plus the expected residual sample equals the original output.The residual estimator has expectation z − ẑdet, so adding ẑdet recovers z.
  • Error analysis: Bernstein-style analysis applies to the residual sampling stage, with its bounds adjusted for the sensitivity mass left after deterministic selection.The variance and magnitude calculations differ from the original scheme through the remaining sensitivity term S − S_k.
  • Compression comparison: The number of unique retained features is analyzed through indicator variables for whether each sampled index appears at least once.The expected number of unique samples under the original and hybrid distributions is compared using concentration for their difference.
  • Guarantee: With probability at least 1 − δ, the deterministic-plus-sampling approach preserves the approximation guarantee while using fewer unique samples when the stated comparison condition holds.The paper concludes that this yields better compression under the condition established by the concentration argument.

B.1.1 EXPERIMENTAL EVALUATION OF DERANDOMIZATION

The experiments compare random, partial, and complete derandomization on LeNet300-100 trained on MNIST across retained-parameter levels, before and after retraining. Partial derandomization preserves performance, while complete derandomization’s added error becomes negligible after retraining.

  • Experimental variants: The evaluation compares no derandomization, partial derandomization, and complete derandomization using sensitivity-based channel selection.Partial derandomization keeps top channels before sampling the remainder; complete derandomization keeps top channels until the sampling budget is exhausted.
  • Before retraining: Partial derandomization does not impact performance, whereas complete derandomization has a slightly detrimental effect before retraining.This pattern agrees with the theorem’s prediction that partial rather than complete derandomization is preferable.
  • After retraining: After retraining, the additional error from complete derandomization is negligible in practical settings.The caption identifies this reduced error in the post-retraining comparison.

C MAIN COMPRESSION THEOREM

The main compression result extends layer-wise sampling guarantees across a network by propagating approximation errors through layers. The CNN adaptation accounts for convolutional patches while preserving the feature-importance analysis and supports broad empirical evaluation.

  • Network-level guarantee: The main compression theorem combines a generalized layer-wise bound with error propagation across the network.The theorem is obtained by extending Theorem 2 to layers and applying prior error-propagation bounds.
  • Error assumptions: The layer constants controlling propagated error depend on the input distribution and are required to dominate layer-wise activation deviations with high probability.The paper also gives a sub-Exponential example for bounding these constants as a function of expected maximum deviations and failure probability.
  • Theorem 3: Theorem 3 gives a high-probability guarantee for the pruned parameter set generated by applying Algorithm 2 across all layers.Its statement uses i.i.d. input samples from D and bounds both approximation probability and the number of retained filters.
  • CNN extension: For CNNs, feature-map sensitivity is modified by taking a maximum over convolutional patches, reflecting weight sharing across spatial locations.The resulting CNN definition associates activations with sliding-window patches, while the remaining analysis is analogous.
  • Evaluation scope: Experiments cover MNIST, CIFAR-10, and ImageNet with LeNet, VGG, ResNet, WideResNet, and DenseNet, comparing against state-of-the-art filter-pruning methods.Reported experimental quantities are averaged over three separately trained and pruned networks.

E.1 COMPARISON METHODS

The paper compares its filter-pruning method with norm-based and greedy feature-map pruning approaches, using different saliency criteria and pruning procedures.

  • The competing methods are re-implemented and evaluated with the same iterative pruning and fine-tuning strategy, while fixed per-layer pruning ratios are used because they lack automatic relative-ratio selection.
  • Filter Thresholding retains filters with the largest entry-wise ℓ2-norm until a specified sparsity level is reached.
  • SoftNet uses the entry-wise ℓ1-norm as its filter saliency score and can restore pruned filters during fine-tuning.
  • ThiNet iteratively and greedily removes the feature map whose deletion causes the least absolute pre-activation error in the next layer.
  • ThiNet’s greedy procedure is quadratic in both layer size and the number of input data points, making it slow in practice.The authors use around 100 data points, while Luo et al. report using 100 data points per output class.

E.2 LENET ARCHITECTURES ON MNIST

The experiments evaluate pruning across LeNet and other CNN architectures on MNIST, CIFAR-10, and ImageNet, including prune-only and limited prune-retrain settings. In the prune-only ImageNet scenario, the proposed method produces more compact and accurate networks than competing approaches.

  • The MNIST experiments use LeNet300-100 and LeNet-5, with training and fine-tuning hyperparameters specified for the LeNet architectures.
  • Additional evaluations cover VGG16, ResNet20, DenseNet22, and Wide ResNet-16-8 trained on CIFAR-10.
  • ImageNet experiments use ResNet18, ResNet50, and ResNet101 under iterative pruning without retraining and limited-iteration prune-retrain scenarios.
  • In the prune-only scenario, the proposed algorithm outperforms competing approaches in generating compact, more accurate networks without fine-tuning.The method reevaluates data-informed filter importance after each pruning iteration, whereas competing weight-based criteria capture inter-layer dependencies after retraining.
  • The prune-retrain evaluation reports pruned error, error difference, percentage pruned, and FLOP reduction after only 2-3 iterations per experiment.

E.5 APPLICATION TO REAL-TIME REGRESSION TASKS

The paper evaluates filter pruning for real-time autonomous-driving regression, where inference must meet a hard time limit while preserving accuracy. Without retraining, the proposed method outperforms competing methods in performance versus prune ratio and can support inference-budget-driven compression.

  • Real-time autonomous-driving inference must produce accurate predictions within a hard upper bound on computation time.
  • The evaluated Deepknight network predicts a human driver’s steering angle from driving data in real time.
  • Without retraining, the proposed method outperforms other state-of-the-art methods in performance versus prune ratio.Performance is measured using test loss, defined here as mean-squared error on the test data set.
  • Table 7 reports the driving-network training and pruning hyperparameters, with no fine-tuning conducted for this architecture.
  • Given an inference-time budget T, the method can compress a larger trained architecture until the budget is satisfied.

E.6 COMPARISONS TO ADDITIONAL METHODS ON CIFAR-10

On CIFAR-10, the proposed method is compared with state-of-the-art pruning approaches across modern CNN architectures and multiple accuracy-efficiency criteria. It generally produces models that are both sparser and more accurate, including a ResNet20 model with over 43% parameter reduction and 0.04% error difference.

  • The evaluation covers ResNet20, ResNet56, ResNet110, and VGG16 using iterative pruning and fine-tuning.
  • Across the considered models and metrics, the proposed algorithm outperforms competing approaches in virtually all cases.The metrics are pruned error, error difference, percentage of parameters pruned, and FLOP reduction.
  • >43% PR is achieved for ResNet20 with 8.64% Err. and 0.04% Err. Diff, making the model both the sparsest and most accurate reported result in that comparison.
  • For ResNet110, and similarly for VGG16 and ResNet56, the proposed method tends to produce models that are among the sparsest, most efficient, and most accurate.
Loading 1911.07412v2…