Source-linked AI summary
Data-Driven Sparse Structure Selection for Deep Neural Networks
Zehao Huang, Naiyan Wang
TL;DR
High computational and storage demands make state-of-the-art CNNs difficult to deploy in latency-critical applications. The paper proposes Sparse Structure Selection, which jointly trains and prunes CNN structures through sparse scaling factors optimized with modified stochastic APG, and reports promising adaptive depth and width selection across several networks and datasets.
Problem
High computational and storage demands make state-of-the-art CNNs difficult to deploy in latency-critical applications.
Method
Sparse Structure Selection jointly trains CNN weights and sparse scaling factors for neurons, groups, or blocks, using a modified stochastic Accelerated Proximal Gradient method.
Results
The method produces promising pruning results across PeleeNet, VGG, ResNet, and ResNeXt, with adaptive depth and width selection on CIFAR and ILSVRC 2012.
Takeaways & Limitations
CNN structure can be selected jointly with normal training, without extra fine-tuning or multi-stage optimization.
Takeaways & Limitations
The conclusion identifies non-convex relaxations and adaptively adjusting penalties for different structure complexities as future directions.
Abstract
from arXiv · showhide
Deep convolutional neural networks have liberated its extraordinary power on various tasks. However, it is still very challenging to deploy state-of-the-art models into real-world applications due to their high computational complexity. How can we design a compact and effective network without massive experiments and expert knowledge? In this paper, we propose a simple and effective framework to learn and prune deep models in an end-to-end manner. In our framework, a new type of parameter -- scaling factor is first introduced to scale the outputs of specific structures, such as neurons, groups or residual blocks. Then we add sparsity regularizations on these factors, and solve this optimization problem by a modified stochastic Accelerated Proximal Gradient (APG) method. By forcing some of the factors to zero, we can safely remove the corresponding structures, thus prune the unimportant parts of a CNN. Comparing with other structure selection methods that may need thousands of trials or iterative fine-tuning, our method is trained fully end-to-end in one training pass without bells and whistles. We evaluate our method, Sparse Structure Selection with several state-of-the-art CNNs, and demonstrate very promising results with adaptive depth and width selection.
1 Introduction
The paper introduces an end-to-end, data-driven framework that jointly trains and prunes CNNs by sparsifying scaling factors on selected structures. It uses modified stochastic APG optimization and demonstrates adaptive depth and width selection across several networks and datasets.
- CNN deployment remains difficult because high computational and storage costs are prohibitive for latency-critical applications.
- Existing acceleration methods include pruning, low-rank decomposition, quantization, knowledge distillation, architecture design, and sparse learning.
- Non-structured sparsity produces irregular connectivity, so group sparsity methods seek hardware-friendly CNN structures.
- Scaling factors multiply outputs of neurons, groups, or blocks, while sparsity regularization drives unimportant factors to zero for safe structural removal.
- The framework jointly optimizes CNN weights and scaling factors with a modified stochastic Accelerated Proximal Gradient method.
- The method prunes neurons, residual blocks, and groups in PeleeNet, VGG, ResNet, and ResNeXt, adaptively adjusting network depth and width on CIFAR and ILSVRC 2012.
2 Related Works
Related work spans connection pruning, neuron and group sparsity, architecture search, and residual-network analysis. Prior approaches often require iterative pruning, retraining, heuristic optimization, or specialized search procedures.
- Early and modern pruning methods remove connections using Hessian information or weight-magnitude thresholds, with threshold-based pruning requiring iterative pruning and fine-tuning.
- Neuron-level pruning can provide practical acceleration with moderate accuracy loss but remains difficult to implement end-to-end without iterative pruning and retraining.
- CNN architecture-learning methods explore designs through reinforcement learning, genetic algorithms, sparse learning, or binary optimization.
- Residual networks use skip connections to mitigate gradient vanishing or exploding issues in ultra-deep networks.
3 Proposed Method
The method jointly learns CNN weights and sparse scaling factors, then prunes neurons, groups, or residual blocks whose factors become zero. A modified stochastic APG procedure enables end-to-end structure selection during training.
- Sparse Structure Selection: Scaling factors multiply outputs of selected neurons, groups, or blocks, allowing the method to learn which structures to retain.The factors add flexibility with few additional parameters and are optimized jointly with CNN weights.
- Sparse Structure Selection: Zero scaling factors identify structures that can be safely removed because their outputs do not contribute to subsequent computation.This yields compact networks through direct structural removal rather than irregular weight sparsity.
- Sparse Structure Selection: The optimization combines CNN loss, ordinary weight regularization, and an l1 sparsity penalty on scaling factors.The l1 term is expressed as γ∥λ∥1, encouraging selected factors toward zero.
- Optimization: A modified stochastic APG method updates weights and scaling factors together using mini-batches and fixed momentum.The modification avoids the extra forward-backward computation required by a direct APG formulation.
- Block Selection: Block selection places a scaling factor after each residual block, so zero factors remove entire blocks and reduce ResNet depth.The resulting sparse factors can produce a much shallower ResNet.
- Group Selection: Group selection weights aggregated transformations with scaling factors, enabling inactive groups to be removed and combined block-plus-group pruning.Neuron pruning is a special case in which each group contains one neuron.
4 Experiments
Experiments evaluate SSS across CIFAR and ImageNet using neuron, group, and residual-block pruning in several CNN architectures. The method generally reduces parameters or FLOPs while preserving or improving reported accuracy, with optimization behavior also examined.
- Experimental setup: SSS is evaluated on CIFAR-10, CIFAR-100, and ImageNet using VGG16, ResNet, ResNeXt, and PeleeNet.The experiments target neuron, residual-block, and group pruning across plain, residual, grouped, and lightweight networks.
- 4.1 CIFAR: 30% fewer parameters and 30%–50% lower computational cost are achieved for VGG on CIFAR with minor performance loss.Parameters and FLOPs are both reported after adding scale factors after batch-normalization layers.
- 4.1 CIFAR: 2.5x speedup is obtained for ResNet-164 with about 2% performance loss on both CIFAR-10 and CIFAR-100.SSS also outperforms the baseline at similar parameter and FLOP budgets, and early-stage blocks are pruned first.
- 4.1 CIFAR: 60% fewer FLOPs are achieved for ResNeXt on CIFAR-10 with 1% higher accuracy.Combining group and block pruning is especially effective in ResNeXt-164, while shallow residual-block groups are pruned mostly.
- 4.2 ImageNet LSVRC 2012: 31% fewer FLOPs are achieved by ResNet-32 with about 2% top-1 accuracy loss, while ResNet-41 reduces FLOPs by 15% with 0.7% loss.The pruned models perform on par with original hand-crafted ResNets, and ResNet-32 and ResNet-41 outperform ResNet-34 with fewer FLOPs.
- 4.2 ImageNet LSVRC 2012: 24% top-1 error is reported for ResNeXt-41, with half the parameters and more than 20% fewer FLOPs than the original ResNet50.Three residual blocks in the conv5 stage are pruned entirely; the authors identify this pruning pattern as warranting further study.
5 Conclusions
The paper presents SSS as a data-driven framework that jointly trains and prunes CNN structures, adapting network depth and width to task and budget constraints. Future work targets broader applications and more advanced, complexity-aware sparsity regularization.
- Conclusions: SSS jointly formulates CNN training and pruning as a sparse-regularized optimization problem using scaling factors that identify removable structures.Scaling factors are driven to zero so corresponding structures can be removed.
- Conclusions: The method adaptively adjusts CNN depth and width based on available budgets and task difficulty across PeleeNet, VGG, ResNet, and ResNeXt.
- Conclusions: Future work will apply SSS to additional applications such as object detection and investigate non-convex regularization and penalties adapted to structural complexity.