Source-linked AI summary
Filter Pruning via Geometric Median for Deep Convolutional Neural Networks Acceleration
Yang He, Ping Liu, Ziwei Wang, Zhilan Hu, Yi Yang
TL;DR
Norm-based filter pruning depends on large norm variation and very small minimum norms, conditions that are not always met. The paper proposes FPGM, which prunes redundant filters using geometric-median relationships, and reports state-of-the-art performance across several benchmarks. Its practical acceleration is constrained by non-tensor layers and hardware-related overheads.
Problem
Norm-based filter pruning relies on large norm deviation and arbitrarily small prunable-filter norms, but these requirements are not always satisfied.
Method
FPGM computes the geometric median of filters within a layer and prunes filters with redundant, most replaceable information rather than relying on norms.
Results
FPGM achieves state-of-the-art performance in several benchmarks and demonstrates effectiveness and efficiency across two benchmarks.
Takeaways & Limitations
FPGM explicitly considers mutual relations between filters and can achieve good performance when the norm-based criterion fails.
Takeaways & Limitations
Practical acceleration is constrained by non-tensor-layer inference time, IO delay, buffer switching, and BLAS-library efficiency.
Abstract
from arXiv · showhide
Previous works utilized ''smaller-norm-less-important'' criterion to prune filters with smaller norm values in a convolutional neural network. In this paper, we analyze this norm-based criterion and point out that its effectiveness depends on two requirements that are not always met: (1) the norm deviation of the filters should be large; (2) the minimum norm of the filters should be small. To solve this problem, we propose a novel filter pruning method, namely Filter Pruning via Geometric Median (FPGM), to compress the model regardless of those two requirements. Unlike previous methods, FPGM compresses CNN models by pruning filters with redundancy, rather than those with ''relatively less'' importance. When applied to two image classification benchmarks, our method validates its usefulness and strengths. Notably, on CIFAR-10, FPGM reduces more than 52% FLOPs on ResNet-110 with even 2.69% relative accuracy improvement. Moreover, on ILSVRC-2012, FPGM reduces more than 42% FLOPs on ResNet-101 without top-5 accuracy drop, which has advanced the state-of-the-art. Code is publicly available on GitHub: https://github.com/he-y/filter-pruning-geometric-median
1. Introduction
The paper identifies conditions that limit norm-based filter pruning and proposes FPGM, which removes redundant filters instead. Experiments demonstrate FPGM’s effectiveness and efficiency on two benchmarks.
- Motivation: Deep CNNs deliver strong vision performance but require costly computation, complicating deployment on mobile devices.Even efficient architectures can contain millions of parameters and require billions of FLOPs.
- Prior pruning approaches: Filter pruning preserves regular model structures, making it more suitable for acceleration than weight pruning’s unstructured sparsity.Regular structures can better leverage efficient BLAS libraries while reducing storage usage.
- Limitations of norm-based pruning: Norm-based pruning assumes that smaller-norm filters are less important and selects filters below a preset threshold for removal.This criterion retains filters with larger norms.
- Limitations of norm-based pruning: Norm-based pruning requires large norm deviation and arbitrarily small prunable-filter norms, but these conditions are not always satisfied.Small norms must indicate absolutely small rather than merely relatively smaller contributions.
- Proposed method: FPGM prunes filters with redundant, most replaceable information by computing the geometric median within each layer.Filters near the geometric median can be represented by the remaining filters, avoiding reliance on norm-based selection.
- Evaluation: Extensive experiments on two benchmarks demonstrate FPGM’s effectiveness and efficiency.The introduction identifies this evaluation as a central contribution.
2. Related Works
Related work spans weight pruning, data-dependent and data-independent filter pruning, and methods that revisit norm-based filter selection. The paper contrasts FPGM with a batch-normalization-based approach on structural applicability, fine-tuning, and gradient-computation requirements.
- Pruning approaches: Weight pruning removes individual weights but produces unstructured models that are difficult to accelerate with efficient BLAS libraries.Filter pruning instead removes entire filters and maintains regular structure.
- Data Dependent Filter Pruning: Data-dependent filter-pruning methods use training data, next-layer statistics, activation reconstruction, PCA, or subspace clustering to select filters.These approaches determine pruning choices from data or activation-related information.
- Data Independent Filter Pruning: Data-independent methods include ℓ1- and ℓ2-norm criteria, batch-normalization scaling sparsity, and spectral clustering.These strategies select filters without using training data for the pruning criterion.
- Comparison with prior work: Compared with the batch-normalization-based method, FPGM is not limited to structures containing batch normalization.The comparison also states that FPGM does not require extra fine-tuning or scaling-factor gradients.
3. Methodology
FPGM addresses limitations of norm-based filter pruning by identifying redundant filters through geometric-median relationships within each layer. The method selects filters near the layer’s geometric median, prunes them during training, and produces a compact model, while realistic acceleration can differ from theoretical FLOPs reduction.
- Limitations of Norm-Based Criterion: Norm-based pruning requires large filter-norm deviation and very small minimum norms, conditions that may not hold in real networks.Small deviations narrow the threshold search space, while large minimum norms mean supposedly least-important filters may still contribute significantly.
- Norm Statistics in Real Scenarios: ResNet norm statistics show densely concentrated filters and non-negligible minimum norms, making thresholds difficult to choose and minimum-norm pruning potentially harmful.Examples include norms around 10^-6, intervals of roughly 0.3 versus 1.7, and minimum norms around 0.8.
- Filter Pruning via Geometric Median: FPGM computes the geometric median of filters within each layer and selects filters nearest to it because their information is shared with the remaining filters.The geometric median is used as a robust centrality estimator; nearest filters are treated as replaceable and therefore removable with little negative impact.
- Filter Pruning via Geometric Median: For computational efficiency, FPGM selects known candidate filters minimizing summed distances to other filters instead of solving the general geometric-median problem directly.The general approximate geometric-median algorithms are described as time-consuming, whereas the candidate-filter formulation searches among filters already present in the layer.
- Algorithm: During training, FPGM identifies Ni+1Pi filters per layer, zeroizes them, and obtains a compact model after iterating through epochs.The pruning rate Pi controls how many filters are removed, and the algorithm updates model parameters before selecting and zeroizing filters.
- Pruning and Acceleration: Pruning rate Pi reduces adjacent-layer computation to (1 −Pi+1) × (1 −Pi) of the original computation, while practical acceleration is constrained by non-convolutional operations and hardware effects.The cited practical constraints include BN and pooling inference time, IO delay, buffer switching, and BLAS-library efficiency.
4. Experiments
FPGM is evaluated on VGGNet and ResNet across CIFAR-10 and ILSVRC-2012, using uniform layer pruning rates and comparisons with prior methods. Experiments report strong accuracy–acceleration trade-offs, realistic-speed considerations, and ablations of pruning intervals, pruning amounts, distance functions, and criterion mixing.
- Experimental Setup: FPGM is evaluated on VGGNet and ResNet using CIFAR-10 and ILSVRC-2012, including pretrained and scratch-model settings.The experiments cover single-branch and multiple-branch networks across two image-classification benchmarks.
- Experimental Setup: FPGM prunes all weighted layers simultaneously at a shared rate P, applying pruning at each training epoch without requiring sensitivity analysis.The pruning procedure uses one hyper-parameter to balance acceleration and accuracy, and sensitivity analysis is reported as nonessential for good performance.
- CIFAR-10 Results: 52.3% versus 38.6% acceleration on pretrained ResNet-110 accompanies a 0.16% performance increase for FPGM, while PFEC uses a lower acceleration ratio and harms performance.On ResNet-56, pruning 52.6% FLOPs produces a 0.66% accuracy drop with FPGM versus 1.33% with SFP.
- ILSVRC-2012 Results: More than 40% FLOPs reduction on pretrained ResNet-101 causes no top-5 accuracy loss and only 0.05% top-1 accuracy loss with FPGM.The reported performance degradation for Rethinking is 2.10%, while FPGM is attributed superior performance to explicitly using relationships between filters.
- Practical Acceleration: Theoretical and realistic acceleration can differ because of non-tensor layers, IO delay, buffer switching, and BLAS-library efficiency.Forward time is measured on a GTX1080 GPU with batch size 64 to compare the two acceleration measures.
- Ablation Study: Accuracy fluctuation stays below 0.3% when the pruning interval varies from one to ten epochs, while pruning 18% or 40% FLOPs can exceed the unpruned baseline.Using ℓ1-norm instead of the default distance yields 93.87 ± 0.22% accuracy, compared with 93.73 ± 0.23% for the baseline distance and 93.56 ± 0.13% for cosine distance.
5. Conclusion and Future Work
The paper concludes that norm-based filter pruning has limitations and introduces FPGM, which considers mutual relations between filters to accelerate deep CNNs. FPGM achieves state-of-the-art performance in several benchmarks, with future work targeting combinations with other acceleration methods.
- FPGM considers mutual relations between filters rather than relying on the norm-based criterion.
- FPGM achieves state-of-the-art performance in several benchmarks.
- Future work will explore combining FPGM with matrix decomposition and low-precision weights.