Source-linked AI summary
Dynamic Channel Pruning: Feature Boosting and Suppression
Xitong Gao, Yiren Zhao, Łukasz Dudziak, Robert Mullins, Cheng-zhong Xu
TL;DR
Deep CNNs require substantial computational and memory resources, and static pruning can permanently discard features needed for particular inputs. FBS predicts input-dependent channel saliency, dynamically boosts important channels, and skips unimportant ones while preserving the full network. It achieves approximately 5× and 2× compute savings on VGG-16 and ResNet-18, respectively, with less than 0.6% top-5 accuracy loss.
Problem
Deep CNN accuracy improvements increase memory, bandwidth, and compute demands, while static channel pruning permanently removes capabilities despite input-dependent feature importance.
Method
FBS adds small auxiliary connections that use preceding-layer features to predict channel saliency, dynamically amplify salient channels, and suppress unimportant ones during convolution.
Results
5× and 2× compute savings on VGG-16 and ResNet-18, respectively, were achieved within 0.6% loss of top-5 accuracy, surpassing examined pruning and dynamic-execution methods under the same performance constraints.
Takeaways & Limitations
FBS preserves the full CNN capabilities while accelerating many popular CNNs through conventional SGD-based training.
Takeaways & Limitations
The paper notes that a Lasso-based dynamic-layer formulation can be minimized by parameter scaling without changing outputs, making the resulting optimum infeasible in finite-precision arithmetic.
Abstract
from arXiv · showhide
Making deep convolutional neural networks more accurate typically comes at the cost of increased computational and memory resources. In this paper, we reduce this cost by exploiting the fact that the importance of features computed by convolutional layers is highly input-dependent, and propose feature boosting and suppression (FBS), a new method to predictively amplify salient convolutional channels and skip unimportant ones at run-time. FBS introduces small auxiliary connections to existing convolutional layers. In contrast to channel pruning methods which permanently remove channels, it preserves the full network structures and accelerates convolution by dynamically skipping unimportant input and output channels. FBS-augmented networks are trained with conventional stochastic gradient descent, making it readily available for many state-of-the-art CNNs. We compare FBS to a range of existing channel pruning and dynamic execution schemes and demonstrate large improvements on ImageNet classification. Experiments show that FBS can respectively provide $5\times$ and $2\times$ savings in compute on VGG-16 and ResNet-18, both with less than $0.6\%$ top-5 accuracy loss.
1 Introduction
Deep CNNs improve accuracy at substantial memory, bandwidth, and compute cost, while channel importance varies with the input. The paper proposes dynamically selecting channels so salient features are retained without permanently removing the full network’s capabilities.
- Deep CNNs demand substantial memory, bandwidth, and computation, creating deployment challenges for cloud and edge applications.
- Static channel pruning permanently removes capabilities and may fail on inputs that require discarded channels because neuron saliency is input-dependent.
- Dynamic channel pruning predicts input-specific saliency and computes only selected channels while preserving the full network.
- FBS dynamically amplifies salient output channels and suppresses unimportant ones using features from the preceding layer to predict channel saliency.
- FBS adds small auxiliary connections and trains the predictor with conventional stochastic gradient descent rather than requiring algorithmic training changes.
- FBS models applied to CIFAR-10, VGG-16, and ResNet-18 surpassed the examined channel-pruning and dynamic-execution methods at matched speed-ups.
2 Related Work
Prior efficiency methods include unstructured pruning, structured sparsity, spatial sparsity, and dynamic networks that select computation at inference. These approaches trade hardware efficiency, permanently lost capabilities, or training complexity in different ways.
- Early weight pruning creates irregular sparse connections that GPUs struggle to exploit efficiently.
- Structured sparsity removes channels or connections to produce GPU-friendly dense operations, but permanently loses the capabilities of pruned neurons.
- Dynamic networks select evaluated blocks, branches, modules, or channels according to input data during inference.
- Spatial-sparsity methods accelerate convolutions by exploiting localized salient activation regions or sharing sparsity information with expensive layers.
- Binary or multi-choice execution policies are commonly nondifferentiable, so several dynamic networks train their policy functions with reinforcement learning.
3 Feature Boosting and Suppression
FBS augments convolutional layers with low-cost, input-dependent channel saliency prediction, dynamically boosting important outputs and suppressing unimportant ones. Its sparse outputs enable convolutional computation to skip zero-valued input and output channels while retaining the full network structure.
- Dynamic layer design: A general dynamic layer computes an expensive tensor and multiplies it by a cost-effective policy tensor, allowing zero-valued slices to be skipped.The formulation also supports sparsifying the policy through an expected L1 regularizer.
- Dynamic layer design: FBS predicts output-channel importance from previous-layer features and dynamically amplifies salient channels while suppressing unimportant ones.Its auxiliary policy controls channel-wise computation at run-time rather than permanently removing channels.
- Channel saliencies: FBS uses a k-winners-take-all policy to compute only the ⌈dC_l⌉ most salient predicted channels and suppress the rest with zeros.The density d can vary, pruning C_l −⌈dC_l⌉ output channels and exposing input- and output-side sparsity to neighboring convolutions.
- Training and overhead: FBS adds tiny auxiliary connections, whose overhead is negligible relative to the potential speedup from dynamic sparsity.Unlike many non-differentiable dynamic execution policies, its formulation supports end-to-end training with stochastic gradient descent.
- Learning channel saliencies: The saliency predictor subsamples each input channel to a scalar, then uses a fully connected layer to predict channel saliencies.The experiments use the ℓ1-norm by default, equivalent to global average pooling for ReLU-activated inputs; the predictor is regularized with an L1 penalty.
4 Experiments
Experiments evaluate FBS against static pruning and dynamic execution on CIFAR-10 and ImageNet models, measuring accuracy–compute trade-offs, channel behavior, and memory costs. FBS retains accuracy better under computational constraints while reducing computation, memory accesses, and peak memory usage.
- CIFAR-10: FBS, NS, and NS+FBS are compared on M-CifarNet across accuracy and MAC budgets.The experiments incrementally increase sparsity to trace the accuracy/performance trade-off.
- CIFAR-10: 3.93× speed-up at 90.50% accuracy is achieved by FBS, compared with 3.22× for NS+FBS and 1.19× for NS.At a 2× speed-up target, accuracies are at least 91.55%, 90.90%, and 87.54%, respectively.
- CIFAR-10: FBS learns input-dependent channel skipping across M-CifarNet’s convolutional layers, with different categories producing distinct channel-use patterns.At d = 0.5, the model reaches 90.59% top-1 and 99.65% top-5 accuracy, while running approximately 4× faster and using 2.37× fewer weights.
- ImageNet ILSVRC2012: At d = 0.7 on ResNet-18, FBS uses 1.12 G MACs and reaches a 31.54% top-1 error rate, versus 1.51 G MACs and 31.70% for NS.FBS consistently outperforms NS under different speed-up constraints.
- ImageNet ILSVRC2012: FBS achieves the highest possible speed-up and lowest error rates among the dynamic execution methods examined, while improving baseline top-1 and top-5 accuracy by 1.73% and 0.46%.The baseline includes the FBS auxiliary layers but suppresses no channels, corresponding to d = 1.
- ImageNet ILSVRC2012: At 5.23× speed-up on VGG-16, FBS increases top-1 error by 1.08% and top-5 error by 0.59%.At 3.01× speed-up, the corresponding error increases are 0.44% and 0.04%.
- ImageNet ILSVRC2012: FBS reduces memory accesses and peak memory usage by avoiding suppressed weights and activations during single-image inference.The reported reductions apply across low-end and cloud application scenarios and generally improve cache utilization.
5 Conclusion
FBS reduces CNN computation while preserving model capabilities and high accuracy. It achieves substantial savings on ResNet-18 and VGG-16 while outperforming examined pruning and dynamic-execution methods under comparable constraints.
- FBS reduces required computation while maintaining high accuracies and preserving CNN capabilities.
- 2× and 5× computation savings are achieved on ResNet-18 and VGG-16, respectively, within 0.6% top-5 accuracy loss.
- Under the same performance constraints, FBS surpasses the examined structured pruning and dynamic execution methods in accuracy.
- FBS can accelerate many popular CNNs using conventional SGD without algorithmic changes to training.
A Details of M-CifarNet on CIFAR-10
The CIFAR-10 experiments use M-CifarNet and compare FBS with NS through layer-wise compute analysis and training-history visualizations. The models are minimized under a 90.5% classification-accuracy constraint.
- M-CifarNet is a custom CIFAR-10 CNN with fewer than 1.30 M parameters and 174 M MACs for 32-by-32 RGB inference.
- The CIFAR-10 training procedure uses SGD-style learning-rate decay, consecutive initialization across MAC budgets, and up to 300 epochs for convergence.
- FBS reports computed output channels per inference separately from channels remaining after unused-channel removal.
- Table 4 compares per-layer MACs for FBS, NS, and NS+FBS while maintaining at least 90.5% classification accuracy.
- As FBS fine-tuning proceeds, conv4 channel-skipping probabilities become more pronounced across the 12-epoch training history.
B Details of the ILSVRC2012 classifiers
The ImageNet classifier experiments apply FBS to ResNet-18 and a compressed VGG-16, comparing their accuracy–performance trade-offs with NS. VGG-16 is first structurally pruned to reduce its computational burden before FBS is applied.
- ResNet-18 and VGG-16 ImageNet classifiers use a training procedure similar to the CIFAR-10 experiments, with modified epoch and learning-rate schedules.
- VGG-16 is first compressed with NS, pruning 20% of channels and reducing compute to 7.90 G MACs under a 1% top-5 accuracy-degradation budget.
- FBS is applied directly to all ResNet-18 convolutional layers, with feature summation handled specially for residual blocks.
- Figure 5 compares the accuracy–performance trade-off curves of FBS and NS for ResNet-18.