Source-linked AI summary
Group Fisher Pruning for Practical Network Compression
Liyang Liu, Shilong Zhang, Zhanghui Kuang, Aojun Zhou, Jing-Hao Xue, Xinjiang Wang, Yimin Chen, Wenming Yang, Qingmin Liao, Wayne Zhang
TL;DR
Existing pruning methods rarely address coupled channels in complicated architectures, despite their importance for practical inference efficiency. Group Fisher Pruning automatically groups coupled channels, scores them with Fisher information, and normalizes importance by memory reduction; experiments across classification and detection networks validate effective pruning with improved speed-accuracy trade-offs.
Problem
Existing pruning methods seldom handle residual, group/depth-wise convolution, and feature-pyramid structures where channels across layers are coupled and must be pruned simultaneously.
Method
Group Fisher Pruning groups coupled channels through the computation graph, evaluates single and coupled-channel importance using Fisher information, and normalizes importance by memory reduction.
Results
The method effectively prunes sophisticated networks across image classification and object detection, achieving higher or comparable accuracy with higher inference speed in residual-network ablations.
Takeaways & Limitations
The framework provides a general channel-pruning approach for complicated structures and a memory-based criterion intended to deliver more practical speedup.
Abstract
from arXiv · showhide
Network compression has been widely studied since it is able to reduce the memory and computation cost during inference. However, previous methods seldom deal with complicated structures like residual connections, group/depth-wise convolution and feature pyramid network, where channels of multiple layers are coupled and need to be pruned simultaneously. In this paper, we present a general channel pruning approach that can be applied to various complicated structures. Particularly, we propose a layer grouping algorithm to find coupled channels automatically. Then we derive a unified metric based on Fisher information to evaluate the importance of a single channel and coupled channels. Moreover, we find that inference speedup on GPUs is more correlated with the reduction of memory rather than FLOPs, and thus we employ the memory reduction of each channel to normalize the importance. Our method can be used to prune any structures including those with coupled channels. We conduct extensive experiments on various backbones, including the classic ResNet and ResNeXt, mobile-friendly MobileNetV2, and the NAS-based RegNet, both on image classification and object detection which is under-explored. Experimental results validate that our method can effectively prune sophisticated networks, boosting inference speed without sacrificing accuracy.
1. Introduction
Group Fisher Pruning addresses the difficulty of pruning coupled channels in complicated networks by grouping them automatically, evaluating their importance with Fisher information, and normalizing by memory reduction. The framework targets practical acceleration across diverse vision backbones and tasks.
- Motivation: Modern networks consume substantial inference memory and computation, limiting deployment on edge devices and cloud-service throughput.Network pruning seeks efficiency with negligible accuracy loss, typically followed by fine-tuning to recover performance.
- Motivation: Residual, group, depth-wise, and feature-pyramid structures couple channels across layers, so independent pruning can reduce FLOPs, memory access, and actual speedup.These structures require coupled channels to be preserved or pruned simultaneously.
- Method: Group Fisher Pruning automatically finds coupled channels from the computation graph and assigns them shared binary masks.The framework is designed for complicated structures rather than only sequential networks.
- Method: Channel importance is estimated from loss change using Fisher information, with coupled-channel importance obtained through the chain rule of gradient computation.For a single channel, the Fisher approximation is proportional to the squared mask gradient.
- Method: Memory reduction replaces FLOPs reduction for importance normalization because the authors seek better practical GPU speedup and accuracy-efficiency trade-offs.The method prunes channels with low importance relative to their computation-cost reduction.
- Evaluation: The framework is evaluated on ResNet, ResNeXt, MobileNetV2, and RegNet for image classification and object detection.The experiments emphasize generalization to residual, grouped/depth-wise convolution, and feature-pyramid structures.
2. Related Work
Prior pruning work includes unstructured and structured sparsity methods, heuristic and reconstruction-based importance metrics, and Taylor-based approximations. A central related-work gap is that FLOPs reduction does not consistently predict actual speedup, while complicated channel couplings remain challenging.
- Pruning paradigms: Unstructured pruning removes weights but often requires specialized libraries or hardware for efficiency, whereas structured pruning removes whole channels or filters for direct speedup.N:M fine-grained sparsity targets general-purpose GPUs, but structured methods remain more directly deployable.
- Importance metrics: Weight- and filter-norm methods rely on the assumption that smaller norms indicate less informative structures, an assumption that may fail for structured pruning.PFEC uses L1 channel-weight norms, while SFP uses L2 filter norms.
- Importance metrics: Reconstruction-based methods select channels by minimizing feature-map reconstruction error using LASSO regression or greedy selection.These methods formulate channel selection around preserving intermediate feature maps.
- Importance metrics: Exact loss-change evaluation is prohibitively expensive, motivating Taylor-based approximations that use first- or second-order derivative information.OBD and OBS estimate weight importance from second-order derivatives.
- Efficiency estimation: Many pruning methods constrain FLOPs, but equal FLOPs reductions can produce different actual speedups.The paper reports that memory-access reduction is empirically a more accurate estimator of efficiency gain.
3. Methodology
Group Fisher Pruning groups coupled channels across computation graphs, estimates their importance with Fisher information, and normalizes scores by memory or computation-cost reduction for structured pruning.
- Channel masking: Binary input-channel masks enable structured pruning by removing masked channels and corresponding outputs in parent layers.Masks are applied element-wise to convolutional or fully connected inputs, and zero-masked channels are explicitly discarded during inference.
- Fisher information: Fisher information estimates single-channel importance from the loss change caused by setting its mask to zero.At convergence, the score is proportional to the squared mask gradient, computed from sample-wise gradients during backpropagation.
- Coupled-channel grouping: Layer grouping automatically identifies coupled channels and makes them share masks so they are pruned simultaneously across layers or within grouped convolutions.Depth-first search finds parent relationships in the computation graph, while coupled-channel importance follows the chain rule through shared masks.
- Coupled-channel grouping: Group convolution pruning aggregates channel gradients within each group and then across related layers before computing squared-gradient importance.One group of channels is pruned at a time because channels within a group generally represent related features.
- Importance normalization: Importance normalization prioritizes channels with high computational overhead, using dynamically updated FLOPs reductions or memory reductions across parent layers.The algorithm periodically prunes the least important normalized channel and repeats fine-tuning and score accumulation until the target reduction is reached.
4. Experiments
Experiments show that Group Fisher Pruning handles coupled structures across classification and detection while improving the accuracy–efficiency trade-off and practical inference speed.
- Ablation Studies: Coupled-layer pruning delivered higher or comparable top-1 accuracy with much higher inference speed than isolated-channel pruning under the same FLOPs.This result held for both 50- and 101-layer residual networks.
- Ablation Studies: Memory-normalized importance achieved the best accuracy-efficiency trade-off among unnormalized, FLOPs-normalized, and memory-normalized variants.The unnormalized variant produced the worst efficiency gain and largest accuracy drop.
- Ablation Studies: The method significantly outperformed uniform-scaled MobileNetV2 baselines under different FLOPs budgets and surpassed searched half-FLOPs RegNet counterparts.For RegNet, the comparison pruned a double-FLOPs model to 50% FLOPs remained against a searched half-FLOPs model.
- Compare with SoTAs: On ImageNet, the method performed best across network structures and FLOPs constraints, outperforming layer-wise, sensitivity-based, and heuristic coupled-channel pruning methods.Its global pruning automatically learns layer pruning ratios and uses a Fisher-information metric grounded in the chain rule.
- Prune for Detection: Across COCO detection frameworks, memory normalization and coupled-channel pruning produced the highest efficiency, including 3× Faster R-CNN speedup with only 0.8% mAP drop.The method maintained average precision in many cases and sometimes exceeded the unpruned baseline.
- Prune for Detection: Pruned structures differed by task: classification retained more later-stage capacity, whereas detection preserved more early-stage channels for multiscale features.This pattern supports adaptive pruning across image classification and object detection.
5. Conclusion
The paper presents a general framework for pruning complicated network structures by grouping coupled channels, sharing masks, and using Fisher-based importance normalized by memory reduction. Experiments across residual, grouped/depth-wise convolutional, and FPN structures verify its effectiveness and speedup-oriented trade-off.
- 5. Conclusion: The framework finds coupled channels across layers, shares binary masks, and prunes them simultaneously for better accuracy-efficiency trade-offs.It addresses residual connections, GConv/DWConv, and FPN structures in detection.
- 5. Conclusion: Experiments explore pruning across residual connections, GConv/DWConv, and FPN in detection, verifying the method’s effectiveness.These structures extend evaluation beyond simple sequential networks.
- 5. Conclusion: Memory-reduction normalization is proposed instead of FLOPs normalization to deliver more inference speedup.The motivation is the memory-bound nature of GPUs and the goal of a better accuracy-speedup trade-off.