Source-linked AI summary
Rethinking the Smaller-Norm-Less-Informative Assumption in Channel Pruning of Convolution Layers
Jianbo Ye, Xin Lu, Zhe Lin, James Z. Wang
TL;DR
CNN pruning commonly treats smaller norms as evidence of less informative parameters, but batch normalization and reparameterization challenge that assumption. The paper instead simplifies channel-level computation flow by sparsifying batch-normalization scales, producing constant-output channels that can be pruned. Experiments across several image-learning benchmarks show useful and competitive compact CNN models.
Problem
Norm-based channel importance is challenged because batch-normalization reparameterization can change weight norms without changing network outputs.
Method
The method uses stochastic training with ISTA and rescaling to drive batch-normalization scales γ toward zero, then prunes the resulting constant-output channels.
Results
Experiments across several image-learning benchmarks empirically demonstrate the method’s usefulness and competitiveness for building compact CNN models.
Takeaways & Limitations
Channel-level computation-graph simplification avoids introducing extra parameters or layers and supports deployment of compact models.
Takeaways & Limitations
The method lacks a strong theoretical guarantee and does not strictly preserve the network function when convolutional layers use padding, requiring fine-tuning afterward.
Abstract
from arXiv · showhide
Model pruning has become a useful technique that improves the computational efficiency of deep learning, making it possible to deploy solutions in resource-limited scenarios. A widely-used practice in relevant work assumes that a smaller-norm parameter or feature plays a less informative role at the inference time. In this paper, we propose a channel pruning technique for accelerating the computations of deep convolutional neural networks (CNNs) that does not critically rely on this assumption. Instead, it focuses on direct simplification of the channel-to-channel computation graph of a CNN without the need of performing a computationally difficult and not-always-useful task of making high-dimensional tensors of CNN structured sparse. Our approach takes two stages: first to adopt an end-to- end stochastic training method that eventually forces the outputs of some channels to be constant, and then to prune those constant channels from the original neural network by adjusting the biases of their impacting layers such that the resulting compact model can be quickly fine-tuned. Our approach is mathematically appealing from an optimization perspective and easy to reproduce. We experimented our approach through several image learning benchmarks and demonstrate its interesting aspects and competitive performance.
1 INTRODUCTION
CNN pruning seeks to remove non-critical computations, but identifying safely prunable subsets is difficult. This paper reframes pruning as simplifying channel-to-channel information flow rather than enforcing structured sparsity in dense weights.
- Model simplification can accelerate CNN computation and potentially alleviate overfitting by removing redundant computations from trained models.
- Identifying which subsets of a trained CNN should be pruned is nontrivial, with existing methods differing in their use of training data.
- The proposed method replaces direct structured sparsification of convolution or linear weights with an alternative conception that achieves the same practical goal.
- The method models CNN computation as information flow across channels and uses gates whose closed outputs become constant.
- The approach adds no extra parameters or computation graph changes, introduces only marginal training overhead, and can produce multiple compact models in one training run.
- The paper rethinks a widely used channel-pruning assumption, develops an alternative approach, and evaluates it across benchmarks.
2 RELATED WORK
Prior CNN pruning methods mainly reduce structural complexity by sparsifying weights or feature maps. Related approaches also use batch-normalization scaling factors, while this paper combines that direction with ISTA and rescaling.
- Earlier structural-pruning methods largely sparsify convolutional weights or feature maps across network layers.
- Some methods impose structured sparsity on vector components to better match specialized hardware implementations.
- A convolution without batch normalization can be converted to a near-equivalent batch-normalized form by removing its bias and setting γ and β from estimated statistics.
- A batch-normalization-based approach enforces sparsity on scaling parameter γ, blocking information through selected convolutional channels.
- Compared with related scaling-factor methods, this paper proposes ISTA updates and a rescaling trick to improve pruning robustness and speed.
3 RETHINKING THE SMALLER-NORM-LESS-INFORMATIVE ASSUMPTION
The paper argues that parameter norm is an unreliable channel-importance signal in batch-normalized CNNs because reparameterization can change norms without changing outputs. It therefore promotes sparsity in batch-normalization scales γ, while acknowledging limited theoretical guarantees.
- Model Reparameterization: Deep-network reparameterization can reduce regularization terms while preserving or improving task loss, so norm-based sparse solutions are not well-defined.
- Model Reparameterization: Gradient-based learning may inefficiently explore reparameterization trajectories, causing rapid variable growth or prolonged optimization.
- Transform Invariance: In batch-normalized convolutional layers, uniformly rescaling weights changes their norms without affecting outputs, undermining norm-based importance measures.
- Existing Lasso, group-Lasso, and thresholding-based pruning claims have theoretical gaps, and heuristic procedures may not naturally produce sparse parameterized solutions.
- The method instead enforces sparsity on batch-normalization scale parameters γ across CNN layers.
- Because γ multiplies normalized variables and subsequent batch normalization limits cross-layer reparameterization effects, its magnitude is comparable across layers.
- The authors state that the approach still lacks a strong theoretical guarantee and treat sparsity as a means to identify less important channels.
4 CHANNEL PRUNING OF BATCH-NORMALIZED CNN
The method sparsifies batch-normalization scaling parameters with ISTA, then removes channels whose outputs become constant while preserving network functionality approximately. It combines layer-aware penalties, rescaling, post-processing, and fine-tuning into an end-to-end pruning procedure.
- Constant-channel pruning: A zero scaling parameter makes its output channel constant, allowing that channel to be removed while approximately preserving network functionality.The approximation is exact without convolution padding; with padding, fine-tuning is used to address degradation.
- Constant-channel pruning: For an unnormalized subsequent convolution, constant-channel values are absorbed into the next layer’s bias while convolution is computed only over nonzero-γ channels.The reduced convolution operator selects channels indexed by nonzero γ values.
- Constant-channel pruning: When the subsequent convolution has batch normalization, its moving average is updated instead of directly absorbing the constant channel into a bias.
- ISTA training: ISTA updates batch-normalization scaling parameters γ, encouraging some γ values to become zero during stochastic training.The update uses the training-loss gradient, learning rate, and sparse penalty, with the gradient estimated from mini-batches.
- Algorithm: The algorithm scales γ and subsequent convolution weights before training, trains with SGD plus ISTA, removes zero-γ channels, rescales back, and fine-tunes the compact model.The procedure applies to training from scratch or retraining a pretrained model.
- Hyper-parameters: Layer penalties are set from per-channel memory costs, while learning rate, sparse penalty, and rescaling control convergence and sparsity progress.Larger sparse penalties produce sparser converged models, whereas excessively large penalties can prune all channels.
5 EXPERIMENTS
Experiments on CIFAR-10, ILSVRC2012, and image segmentation evaluate pruning across architectures and tasks. The method produces compact models with competitive accuracy, while performance depends on model capacity and dataset difficulty.
- CIFAR-10: On ConvNet, Model B has fewer parameters and higher accuracy than a separately trained reference model.The reference ConvNet has 224,008 parameters and 86.3% test accuracy, while Model B is smaller and more accurate.
- CIFAR-10: Over-parameterized networks can gain test generalization after substantial channel reduction, whereas saturated networks show unavoidable degradation.The experiments characterize a tradeoff between test accuracy and model efficiency.
- CIFAR-10: 37% parameter removal from ResNet-20 incurs only about 1 percent accuracy loss.A higher pruning penalty is then used to create Model B from Model A.
- ILSVRC2012: On ResNet-101, models with 2.36 × 10^7 and 1.73 × 10^7 parameters increase Top-5 error by less than 0.5%.The models are fine-tuned after pruning from a pre-trained ILSVRC2012 model.
- Image Foreground-Background Segmentation: The segmentation model saves 86% of parameters and 81% of FLOPs after pruning both network branches.The model combines an inception-like branch for foreground localization with a densenet branch for boundary refinement.
- Image Foreground-Background Segmentation: Mean IoU improves by about 2%–5% on four of five segmentation datasets but declines on the challenging DUT-Omron dataset.The reported difficulty is associated with foregrounds that might contain multiple objects.
6 CONCLUSIONS
The paper concludes with a pruning method that simplifies CNN computation graphs through ISTA-based batch-normalization scaling updates and rescaling. Experiments support its usefulness for building compact CNN models.
- The method simplifies a deep CNN’s computation graph by updating batch-normalization scaling parameters with ISTA.A γ-W rescaling trick accelerates pruning and addresses numerical difficulties.
- Across several benchmarks, the method shows usefulness and competitiveness for building compact CNN models.