Source-linked AI summary
Global Sparse Momentum SGD for Pruning Very Deep Neural Networks
Xiaohan Ding, Guiguang Ding, Xiangxin Zhou, Yuchen Guo, Jungong Han, Ji Liu
TL;DR
DNN pruning seeks to reduce model complexity for resource-constrained devices, but existing approaches require difficult sparsity tuning, non-differentiable optimization, or finetuning. The paper proposes Global Sparse Momentum SGD, which performs global on-the-fly pruning through two update rules and reports high compression, automatic layer-wise sparsity selection, and better winning tickets.
Problem
Existing pruning methods make layer-wise sparsity difficult to tune, may require unpredictable finetuning, and may not directly control compression or support end-to-end SGD.
Method
Global Sparse Momentum SGD alters momentum-SGD gradient flow by actively updating selected parameters and passively applying weight decay to the others under a target global compression ratio.
Results
GSM achieves high compression on MNIST, CIFAR-10, and ImageNet, outperforms L-OBS on ResNet-50, and finds better winning tickets.
Takeaways & Limitations
GSM automatically discovers per-layer sparsity ratios while enabling end-to-end training and lossless pruning without time-consuming retraining.
Takeaways & Limitations
Regularization-based alternatives may fail to literally zero parameters and require repeated trials because their hyper-parameters do not directly determine compression.
Abstract
from arXiv · showhide
Deep Neural Network (DNN) is powerful but computationally expensive and memory intensive, thus impeding its practical usage on resource-constrained front-end devices. DNN pruning is an approach for deep model compression, which aims at eliminating some parameters with tolerable performance degradation. In this paper, we propose a novel momentum-SGD-based optimization method to reduce the network complexity by on-the-fly pruning. Concretely, given a global compression ratio, we categorize all the parameters into two parts at each training iteration which are updated using different rules. In this way, we gradually zero out the redundant parameters, as we update them using only the ordinary weight decay but no gradients derived from the objective function. As a departure from prior methods that require heavy human works to tune the layer-wise sparsity ratios, prune by solving complicated non-differentiable problems or finetune the model after pruning, our method is characterized by 1) global compression that automatically finds the appropriate per-layer sparsity ratios; 2) end-to-end training; 3) no need for a time-consuming re-training process after pruning; and 4) superior capability to find better winning tickets which have won the initialization lottery.
1 Introduction
Existing pruning methods face difficulty choosing layer-wise sparsity, training pruned models, controlling final compression, and handling non-differentiable sparsity objectives. GSM addresses these issues through global, end-to-end gradient-flow modification and reports high compression with improved winning-ticket discovery.
- Layer-wise sparsity ratios require prior knowledge because layers differ in sensitivity to pruning and accuracy degradation.
- Pruned models can be difficult to train, and their final accuracy after finetuning is unpredictable.
- Regularization-based pruning may not directly specify the resulting compression ratio, requiring multiple hyper-parameter trials.
- Some sparsity-focused pruning problems are non-differentiable and cannot be solved with SGD in an end-to-end manner.
- GSM alters momentum-SGD gradient flow, uses a target global compression ratio, and automatically discovers appropriate per-layer sparsity ratios.
- GSM achieves high compression ratios on MNIST, CIFAR-10, and ImageNet while finding better winning tickets.
2 Related work
Momentum SGD uses accumulated past gradients to influence update direction, while DNN compression includes pruning alongside approximation, quantization, distillation, and convolutional acceleration. This paper focuses on connection pruning and uses momentum to accelerate passive updates.
- 2.1 Momentum SGD: Momentum SGD accumulates gradients from past steps, giving updates short-term memory beyond the current gradient.
- 2.1 Momentum SGD: The paper uses momentum as an accelerator for passive updates.
- 2.2 DNN pruning: Unstructured connection pruning reduces non-zero parameters and storage but does not effectively reduce computation on standard hardware and software.
- 2.2 DNN pruning: The paper focuses on connection pruning, while prior work includes magnitude pruning, connection splicing, energy-aware pruning, and second-derivative methods.
- 2.3 Other methods: Other compression and acceleration approaches include tensor decomposition, quantization, binarization, knowledge distillation, perforation, FFT, and DCT.
3 GSM: Global Sparse Momentum SGD
GSM formulates pruning as global sparsity-constrained training and alters momentum-SGD gradient flow so parameters are selected for active or passive updates. First-order loss-change estimates guide selection, while momentum-accelerated passive updates drive most parameters toward zero before global pruning.
- 3.1 Formulation: The global compression ratio directly constrains the number of retained nonzero parameters while trading model size against loss.The ratio is defined from the total parameter count and the ℓ-0 norm of the parameter collection.
- 3.2.1 Explicit trade-off as constrained optimization: Constrained optimization explicitly limits each layer’s nonzero parameters, but its non-differentiable sparsity term prevents direct solution by stochastic gradient descent.Alternating SGD with a separate non-differentiable solver, such as ADMM, is one stated remedy.
- 3.2.2 Implicit trade-off using regularizations: Magnitude regularization can plateau before parameters reach zero, and its coefficient does not directly specify the eventual compression ratio.Consequently, selecting a target ratio may require multiple coefficient trials.
- 3.3 Global sparse gradient flow via momentum SGD: At each iteration, GSM ranks parameters with a first-order Taylor metric and actively updates the Q largest-metric parameters while passively updating the remainder.The metric uses parameters and gradients from ordinary intermediate computations, avoiding second-order derivatives.
- 3.3 Global sparse gradient flow via momentum SGD: GSM automatically discovers per-layer sparsity ratios from one global target, is model-agnostic, and reduces to ordinary momentum SGD when Q = |Θ|.Its mask-based activation selection assigns exactly Q active entries across all masks.
- 3.4 GSM enables implicit reactivation and fast continuous reduction: Recomputing activation selection each iteration permits critical connections to reactivate, while momentum accelerates passive reduction toward zero and the final pruning retains the Q largest-magnitude parameters.With (1 − αη/(1−β))^k < 1 × 10^-4, the stated pruning operation causes no accuracy drop on ResNet-56 and DenseNet-40.
4 Experiments
Experiments show that GSM achieves substantial pruning while preserving accuracy across MNIST, CIFAR-10, and ImageNet, and automatically adapts sparsity to layer sensitivity. Momentum accelerates zeroing, re-selection supports training, and GSM identifies stronger winning tickets than magnitude-based pruning.
- Pruning results and comparisons: GSM maintains accuracy at 60× and 125× compression on MNIST, with only a 0.15% accuracy drop at 300× compression on LeNet-5.The evaluated models are LeNet-300-100 and LeNet-5.
- Pruning results and comparisons: 10× parameter reduction on ResNet-56 and DenseNet-40 still maintains accuracy on CIFAR-10.These models are substantially deeper and more complicated than the MNIST models.
- Pruning results and comparisons: On ImageNet with ResNet-50, GSM outperforms L-OBS by a clear margin.The paper attributes this advantage to GSM discovering appropriate layer-wise sparsity ratios from a desired global compression ratio, unlike layer-by-layer pruning at a fixed ratio.
- Automatic layer-wise sparsity ratio decision: Layer sensitivity proxies vary in the same tendency across layers, and GSM prunes sensitive layers less while pruning insensitive layers more.The cited examples include transition layers 14, 27, and 40 in DenseNet-40.
- Momentum for accelerating parameter zeroing: With target compression 8× and β = 0.98, GSM makes 87.5% of parameters smaller than 1 × 10−4 in about 150 epochs, whereas β = 0.90 leaves accuracy near 65% after 400 epochs.Larger momentum accelerates zeroing, though it initially decreases original-model accuracy before training converges.
- Implicit connection reactivation: Re-selection restores parameters that were wrongly penalized through implicit connection rewiring; removing it increases training loss, while reactivation occurs for a decreasing minority of connections.The decreasing reactivation ratio accompanies convergence and attainment of the desired sparsity ratio.
- More powerful winning lottery tickets: GSM-discovered winning tickets achieve higher eventual accuracy than tickets selected by magnitude-based pruning.The winning-ticket procedure replaces only the pruning step with GSM while keeping the other experimental settings unchanged.
5 Conclusion
GSM directly alters momentum-SGD gradient flow by splitting updates into active and passive rules, enabling end-to-end pruning with automatic layer-wise sparsity decisions and implicit rewiring.
- GSM splits momentum-SGD into active updates that preserve accuracy and passive updates that apply momentum-accelerated weight decay to redundant parameters.Passive updates push redundant parameters infinitely close to zero.
- GSM supports end-to-end training, lossless pruning, implicit connection rewiring, and automatic discovery of appropriate per-layer sparsity ratios.
- GSM can identify powerful winning tickets during DNN pruning.