Source-linked AI summary
Operation-Aware Soft Channel Pruning using Differentiable Masks
Minsoo Kang, Bohyung Han
TL;DR
The paper addresses channel pruning for reducing deep-network resource costs while avoiding the weaknesses of prune-then-fine-tune and activation-only decisions. It jointly learns differentiable masks and model parameters using BN and ReLU activation behavior, achieving strong accuracy and efficiency across benchmarks without extra fine-tuning.
Problem
Channel pruning must reduce deep-network costs, but prior methods can rely on prune-then-fine-tune procedures and ignore how BN and ReLU jointly deactivate channels.
Method
The method jointly optimizes model parameters and differentiable masks, estimating channel deactivation probabilities from activation distributions after BN and ReLU operations.
Results
The approach outperforms previous structured pruning methods and achieves strong accuracy and efficiency across multiple benchmarks and backbone networks without extra fine-tuning.
Takeaways & Limitations
Joint soft pruning provides a unified way to identify compressed networks while preserving competitive output accuracy and efficiency without a separate fine-tuning stage.
Takeaways & Limitations
The method requires searching for a suitable λ to control the target pruning ratio, although an alternative training procedure addresses this drawback.
Abstract
from arXiv · showhide
We propose a simple but effective data-driven channel pruning algorithm, which compresses deep neural networks in a differentiable way by exploiting the characteristics of operations. The proposed approach makes a joint consideration of batch normalization (BN) and rectified linear unit (ReLU) for channel pruning; it estimates how likely the two successive operations deactivate each feature map and prunes the channels with high probabilities. To this end, we learn differentiable masks for individual channels and make soft decisions throughout the optimization procedure, which facilitates to explore larger search space and train more stable networks. The proposed framework enables us to identify compressed models via a joint learning of model parameters and channel pruning without an extra procedure of fine-tuning. We perform extensive experiments and achieve outstanding performance in terms of the accuracy of output networks given the same amount of resources when compared with the state-of-the-art methods.
1. Introduction
Deep neural networks are accurate but costly for resource-constrained systems, motivating structured channel pruning. The proposed method jointly learns differentiable masks and model parameters while accounting for BN and ReLU behavior, and reports better accuracy than prior structured pruning methods.
- Deep neural networks remain difficult to deploy on mobile and portable devices because of model size, FLOPs, and power costs.
- Prior channel-pruning methods commonly prune first and fine-tune later, risking poor local minima because pruning decisions are greedy and reduce network capacity before fine-tuning.
- The proposed framework jointly optimizes model parameters and differentiable channel masks, preserving network capacity during training.
- Its pruning strategy estimates channel importance probabilistically from activation distributions while accounting for the network operations applied to each feature map.
- Experimental results show that the approach outperforms previous structured pruning methods, with component-level evidence that each proposed component improves accuracy.
2. Related Work
Deep-network compression includes matrix decomposition, quantization, and pruning, with structured pruning targeting practical inference benefits. The paper positions its method among channel-pruning approaches that avoid extra fine-tuning while addressing greedy or heuristic decisions.
- Matrix decomposition reduces computational cost by approximating pretrained weight matrices or tensors with low-rank factorizations.
- Network quantization reduces parameter precision to accelerate inference, but practical deployment may require additional low-level processing effort.
- Unstructured weight pruning removes connections but can create irregular connectivity that limits inference-speed gains without specialized hardware.
- Structured pruning removes filters, channels, or layers to improve actual inference time without special libraries or hardware support.
- Related channel-pruning methods avoid extra fine-tuning in different ways, but permanent greedy removal or heuristic strategies can lead to local optima or performance gaps.
3. Proposed Method
The method prunes channels by modeling BN outputs and subsequent ReLU deactivation, then learns differentiable masks jointly with network parameters. A sparsity-regularized objective enables soft optimization during training and binary channel removal afterward without separate fine-tuning.
- Operation-aware channel pruning: The framework models channel deactivation using both BN parameters and the subsequent ReLU operation, rather than relying on scaling alone.Channels are considered removable when most BN outputs are negative and therefore zeroed by ReLU.
- Soft channel pruning: A logistic relaxation replaces the nondifferentiable mask, while a moderate slope avoids gradients vanishing as the mask approaches hard decisions.The resulting continuous mask supports gradient-based learning and can be interpreted as a channel deactivation probability.
- Operation-aware channel pruning: The mask estimates the probability that a channel is deactivated after ReLU using a Gaussian CDF parameterized by BN statistics.The mask is defined from thresholds and the CDF of a Gaussian distribution with mean β and variance γ².
- Soft channel pruning: Gumbel-Softmax provides differentiable sampling for the channel mask instead of directly sampling a nondifferentiable Bernoulli variable.Its output approaches a Bernoulli sample as the temperature approaches zero.
- Sparsity loss: The sparsity objective maximizes the Gaussian CDF under a confidence-interval formulation, encouraging channels toward removable states.The loss uses BN affine parameters and a predefined constant to promote network sparsity.
- Joint optimization and pruning: The full objective combines task-specific classification loss with sparsity loss, jointly optimizing network parameters and masks without adding mask parameters.After training, thresholded CDF values produce binary masks, and the resulting network requires no separate fine-tuning stage.
4. Experiments
Experiments on CIFAR-10/100 and ILSVRC-12 evaluate SCP against structured pruning methods, ablate its design choices, and examine accuracy–efficiency trade-offs. SCP generally achieves strong accuracy retention and practical compression without separate fine-tuning, while sparsity controls expose accuracy–FLOPs trade-offs.
- Experimental setup: SCP is evaluated on CIFAR-10/100 and ILSVRC-12 using ResNet, DenseNet, VGGNet, and comparisons with structured pruning baselines.The evaluation includes Slimming, Variational Pruning, SFP, FPGM, and SSS across the listed datasets and architectures.
- Results on CIFAR-10/100: On CIFAR-10, SCP achieves the lowest accuracy drops across tested backbone networks and outperforms Slimming by 1.18%, 0.61%, and 0.88% points in selected cases without separate fine-tuning.The reported gains concern DenseNet-40, VGGNet-19, and VGGNet-16, respectively.
- Results on CIFAR-10/100: On CIFAR-100, SCP has less accuracy drop than Slimming and Variational Pruning in most cases, including gains of 1.95% and 0.28% points over Variational Pruning.These gains are reported for DenseNet-40 and VGGNet-16, respectively, with more compact models.
- Results on ILSVRC-12: On ILSVRC-12, SCP outperforms SSS without fine-tuning by 2.61% and 1.09% points in top-1 and top-5 accuracy drop, while exceeding FPGM by 0.70% and 0.33% in top-1 accuracy drop.The comparison reports higher compression rates than FPGM despite marginally worse top-5 accuracy drop.
- Practical efficiency: SCP reduces inference time by about 24%, from 104 ms to 79 ms, on an NVIDIA TITAN Xp with batch size 64.The measurement compares unpruned and pruned models using wall-clock inference time.
- Ablation and sensitivity analysis: Considering BN and ReLU together outperforms the SCP variant without ReLU, while increasing λ makes networks more compact and larger s lowers FLOPs but accuracy.At λ = 1 × 10^-6, the network exceeds the baseline by 0.67% points; the method also requires searching λ for a target pruning ratio unless the modified training procedure is used.
5. Conclusion
The paper presents a soft channel-pruning algorithm that jointly learns model parameters and pruning masks using stochastic gradient descent. It bases pruning on feature-map properties after batch normalization and ReLU, achieving strong accuracy and efficiency without extra fine-tuning.
- The algorithm jointly learns model parameters and pruning masks through a stochastic gradient-descent procedure.
- Its pruning strategy uses the properties of feature maps produced by successive batch normalization and ReLU operations.
- The method achieves strong accuracy and efficiency across multiple standard benchmarks and different settings without extra fine-tuning.