Source-linked AI summary
Neural Pruning via Growing Regularization
Huan Wang, Can Qin, Yulun Zhang, Yun Fu
TL;DR
The paper addresses limited use of large regularization penalties in pruning and the related problems of pruning schedules and weight-importance scoring. It gradually raises an L2 penalty to address both problems, reporting competitive CIFAR and ImageNet results and applicability to structured and unstructured pruning.
Problem
Pruning research has focused mainly on small regularization strengths, while pruning schedule and accurate weight-importance scoring remain central problems.
Method
The paper proposes two algorithms using a growing L2 regularization scheme to improve pruning schedules and implicitly exploit Hessian information for scoring.
Results
Competitive results are reported on CIFAR and ImageNet, with the methods applying to both filter and unstructured pruning and performing comparably or better across various ImageNet speedups.
Takeaways & Limitations
The results support treating pruning schedule as an important factor alongside weight-importance scoring and using growing regularization to obtain Hessian-informed selection without explicit Hessian values.
Takeaways & Limitations
The methods use pre-specified pruning ratios, leaving layer-wise sensitivity and more proper ratio selection outside the paper's scope.
Abstract
from arXiv · showhide
Regularization has long been utilized to learn sparsity in deep neural network pruning. However, its role is mainly explored in the small penalty strength regime. In this work, we extend its application to a new scenario where the regularization grows large gradually to tackle two central problems of pruning: pruning schedule and weight importance scoring. (1) The former topic is newly brought up in this work, which we find critical to the pruning performance while receives little research attention. Specifically, we propose an L2 regularization variant with rising penalty factors and show it can bring significant accuracy gains compared with its one-shot counterpart, even when the same weights are removed. (2) The growing penalty scheme also brings us an approach to exploit the Hessian information for more accurate pruning without knowing their specific values, thus not bothered by the common Hessian approximation problems. Empirically, the proposed algorithms are easy to implement and scalable to large datasets and networks in both structured and unstructured pruning. Their effectiveness is demonstrated with modern deep neural networks on the CIFAR and ImageNet datasets, achieving competitive results compared to many state-of-the-art algorithms. Our code and trained models are publicly available at https://github.com/mingsuntse/regularization-pruning.
1 INTRODUCTION
Neural network pruning reduces the resource costs of increasingly effective deep networks, but selecting which weights to remove remains central. The paper addresses this scoring problem and introduces growing regularization to also improve pruning schedules.
- Pruning alleviates the rising storage, memory, computation, and energy costs of deep neural networks.
- Pruning methods generally use either importance criteria for one-shot removal or regularization-based training with a penalty term.
- Small regularization penalties are standard because large penalties may damage model expressivity, despite their potential to separate unimportant weights.
- The paper proposes two growing-L2 algorithms targeting pruning schedule and weight importance criterion.
- The methods are presented as easy to implement, scalable to large datasets and networks, and applicable to filter and unstructured pruning.
2 RELATED WORK
Related pruning work is organized around regularization-based sparsity learning and importance-based criteria. These approaches differ mainly in whether they emphasize the penalty scheme or the importance criterion, while other compression methods remain complementary.
- Regularization-based pruning: Regularization-based pruning uses penalties such as L0, L1, or L2 to learn sparsity, with early work emphasizing unstructured pruning.
- Importance-based pruning: Importance-based pruning selects weights using criteria such as magnitude, norms, or Hessian-derived estimates of loss increase.
- Importance-based pruning: Hessian-based criteria can be difficult to compute for large networks, motivating cheaper approximations such as K-FAC Fisher.
- Relationship between approaches: Importance-based and regularization-based pruning have no hard boundary and can be combined, but regularization methods emphasize penalty design while importance methods emphasize scoring.
- Other model compression methods: Quantization, knowledge distillation, low-rank decomposition, and efficient architecture design are orthogonal compression approaches that can complement pruning.
- Pruning schedule: Pruning schedule means how weights are removed, such as in one step or multiple steps, rather than training settings like learning-rate schedules.
3 PROPOSED METHOD
The paper formulates pruning as masking and adjusting weights, then proposes growing L2 regularization to improve pruning schedules and weight scoring. The approach uses local curvature: weights with larger curvature move less toward zero, magnifying magnitude differences for pruning.
- Problem formulation: Pruning transforms a pretrained model into a smaller masked model, followed by fine-tuning, with separate operations for selecting and adjusting weights.The mask M selects removed weights, while T2 adjusts the remaining weights.
- Scope and limitations: The method focuses on filter pruning for acceleration but states that its methodology translates to unstructured pruning by changing the weight-group definition.Filters form groups for structured pruning, while individual weights form groups for unstructured pruning.
- Pruning schedule: GReg-1: GReg-1 retains an L1-norm-selected mask but gradually increases L2 penalties on weights marked for removal instead of pruning them immediately.The method is designed to test whether pruning schedule affects accuracy when the same weights are removed.
- Pruning schedule: GReg-1: The penalty increment δλ is applied to each pruned weight, with updates every Ku iterations allowing the network to adapt between regularization changes.A smaller δλ makes the regularization process smoother.
- Algorithm: The implementation increases penalty factors at a common pace and terminates when a preset ceiling is reached before pruning and fine-tuning.The algorithms are summarized in Algorithm 1.
- Importance criterion: GReg-2: Under a local quadratic loss approximation, increasing L2 regularization shifts each converged weight according to its local Hessian curvature.The analysis assumes a positive-semidefinite Hessian and examines diagonal and two-dimensional cases.
- Importance criterion: GReg-2: Larger curvature produces a larger response ratio ri, meaning the weight moves less toward the origin when the penalty increases.For diagonal Hessians, ri increases with hii; the two-dimensional analysis preserves the same ordering despite weight dependency.
- Importance criterion: GReg-2: Growing penalties magnify magnitude discrepancies until weights naturally separate, enabling simple L1-norm scoring to identify unimportant weights more faithfully.GReg-2 exploits this separation to use Hessian information implicitly without estimating Hessian values directly.
4 EXPERIMENTAL RESULTS
Experiments evaluate growing-regularization pruning across CIFAR and ImageNet, structured and unstructured settings, focusing on pruning schedules and importance criteria. The proposed methods generally improve or match competing approaches, with larger benefits at more aggressive pruning or speedup.
- Experimental setup: Experiments use CIFAR10/100 with ResNet56/VGG19 and ImageNet with ResNet34/50, measuring speedup through FLOPs reduction.ImageNet baselines use official PyTorch pretrained models, while CIFAR baselines have accuracies comparable to the original papers.
- Pruning schedule: GReg-1: GReg-1 consistently outperforms L1+one-shot even though both remove exactly the same weights, isolating pruning schedule as a performance factor.The comparison uses identical fine-tuning schemes and uniform pruning ratios across a broad acceleration range.
- Pruning schedule: GReg-1: Larger pruning ratios produce more pronounced GReg-1 gains, whereas model plasticity makes the advantage smaller at relatively low pruning ratios such as ResNet56 r = 50%.Growing regularization helps transfer expressive power to the remaining weights when the network cannot recover through inherent plasticity alone.
- Importance criterion: GReg-2: Growing L2 regularization increases normalized filter L1-norm discrepancies over iterations, making simple L1-norm scoring increasingly faithful across models and datasets.The reported separation is illustrated for ResNet50 on ImageNet and VGG19 on CIFAR100, while the analysis links it to differing local curvature.
- ImageNet benchmarks: On ImageNet, the methods achieve comparable or better accuracy across speedups on ResNet34 and ResNet50; GReg-2’s advantage over Taylor-FO grows from 0.86% at ∼2× to 2.21% at ∼3×.At speedups below 2×, the proposed methods can improve top-1 accuracy on both ResNet variants.
- Unstructured pruning: The methods also remain effective beyond structured pruning: on ResNet50 unstructured pruning, GReg-2 is comparable to WoodFisher without Hessian approximation, while GReg-1 gives a more favorable result.GReg-2 has better absolute accuracy but a slightly worse accuracy drop than WoodFisher.
5 CONCLUSION
The paper introduces growing L2 regularization to address pruning schedule and weight-importance selection. Results on CIFAR and ImageNet support the methods’ scalability and effectiveness.
- Growing regularization raises the penalty factor to a large level, addressing pruning schedule and weight importance criterion.The methods rely on few impractical assumptions and have a sound theoretical basis.
- The experiments support pruning schedule as a pivotal factor alongside weight importance scoring.The paper argues that schedule deserves more research attention.
- Growing L2 regularization enables Hessian-based pruning without Hessian approximation.The method exploits Hessian information without requiring its specific values.
- The proposed approaches are scalable to large datasets and networks, with encouraging results on CIFAR and ImageNet.
A.1 EXPERIMENTAL SETTING DETAILS
The experiments separate pruning-ratio selection from pruning algorithms and specify training settings across network stages. ImageNet pruning uses a smaller batch size to reduce training time.
- Training settings are summarized in Table 5, including SGD momentum and weight decay.
- ImageNet pruning uses batch size 64 instead of the standard 256 to save training time.
- Pruning is treated as independent from pruning-ratio selection because ratios are viewed as reflecting model redundancy.The paper considers pruning ratios an inherent model characteristic rather than coupling them to the pruning algorithm.
- For ResNets, pruning ratios are represented by one float per convolutional stage.For ResNet56, “[0, 0.5, 0.5, 0.5]” leaves the first stage unpruned and assigns 0.5 to the other three stages.
- Higher stages receive smaller pruning ratios because their smaller spatial maps yield relatively fewer FLOPs.This heuristic targets acceleration more than compression.
B PROOF OF EQ. 5
The proof models the objective locally as a quadratic around a converged point, then analyzes the effect of increasing the L2 penalty. The updated Hessian is H + δλI.
- A converged quadratic objective is represented using its Hessian matrix H around w∗, plus a constant C.
- Increasing the L2 penalty by δλ defines a new objective for analyzing the regularization change.
- The newly converged objective is expressed around ˆw∗ using an updated Hessian ˆH and constant ˆC.
- The updated Hessian satisfies ˆH = H + δλI, encoding the L2 penalty as an isotropic diagonal shift.
- Table 6 lists pruning-ratio summaries for the experiments.
D GREG-1 + OBD
Growing-regularization pruning remains better than its one-shot counterpart when both use the OBD importance criterion. The result also links better scheduling with larger speedup settings.
- OBD is a one-shot pruning method whose Hessian-based criterion is considered more advanced than L1-norm.
- Growing-regularization pruning is consistently better than the one-shot counterpart even with the Hessian-based OBD criterion.The comparison uses the same pruning setup while replacing the L1-norm criterion with OBD.
- A better pruning schedule can provide more accuracy gain when the speedup is larger.
- The VGG19-on-CIFAR100 experiment examines whether weight separation under growing regularization generalizes across datasets and networks.
F HYPER-PARAMETERS AND SENSITIVITY ANALYSIS
The methods use several regularization and update hyper-parameters, with settings selected through validation and sensitivity analysis. The proposed schedule remains robust across tested update intervals and outperforms the one-shot scheme in the reported comparisons.
- Hyper-parameters: The methods introduce regularization ceilings τ and τ′, update intervals K_u and K_s, and granularity δλ.Their settings are summarized in Table 8.
- Hyper-parameters: τ = 1 pushes unimportant weights down, while τ′ = 0.01 applies a moderately large penalty to all weights.The ceilings are selected through validation for different regularization roles.
- Sensitivity analysis: K_u balances convergence time against iteration cost, while K_s stabilizes training under the large regularization ceiling τ = 1.The default K_u is 5 for ImageNet and 10 for CIFAR; K_s is considered less important.
- Sensitivity analysis: The proposed schedule remains significantly better than the “L1+one-shot” scheme even in the worst cases of the sensitivity analysis.This result is reported for the blue-highlighted settings.
- Sensitivity analysis: Ku = 10 is the default setting, with pruning ratios of 90% for ResNet56 and 70% for VGG19 evaluated on CIFAR10/100.Table 9 reports mean accuracy and standard deviation across three random runs while varying K_u.
G MORE RESULTS OF PRUNING SCHEDULE COMPARISON
The experiments test whether the regularization-based pruning schedule improves performance beyond a particular importance-scoring criterion. Across random filter selections, the regularization-based schedule consistently surpasses one-shot pruning with stable results.
- Motivation: The experiments ask whether regularization-based schedule gains persist independently of the weight-importance scoring criterion.Randomly selected filters provide a criterion-agnostic comparison with L1-based pruning.
- Results: 86.85% versus 87.34% for ResNet56 and 65.04% versus 66.05% for VGG19 are the mean accuracies for random versus L1-norm filter selection.The comparison serves as a sanity check for the random-selection experiments.
- Results: In every random run, regularization-based pruning significantly surpasses its one-shot counterpart.The reported accuracy standard deviations indicate stable results despite only five random runs.