PaperlayerView exact revision on arXiv

Source-linked AI summary

Layer-adaptive sparsity for the Magnitude-based Pruning

Jaeho Lee, Sejun Park, Sangwoo Mo, Sungsoo Ahn, Jinwoo Shin

arXiv:2010.07611v2cs.LG

TL;DR

Magnitude pruning works well only when its layerwise sparsity is chosen appropriately, but existing choices lack a clear consensus. LAMP uses a distortion-informed, rescaled magnitude score to choose sparsity automatically, and it consistently outperforms the evaluated baselines, including under weight rewinding.

  • Problem

    Magnitude-based pruning lacks a consensus for selecting layerwise sparsity, with choices commonly made per algorithm using heuristics or tuning.

  • Method

    LAMP uses a rescaled weight-magnitude score motivated by model-level ℓ2 distortion and performs global pruning with automatically determined layerwise sparsity.

  • Results

    LAMP consistently outperforms baseline layerwise sparsity selection schemes across diverse models and datasets, including one-shot pruning and weight-rewinding setups.

  • Takeaways & Limitations

    LAMP is presented as a “go-to” layerwise sparsity selection candidate for magnitude-based pruning within the evaluated unstructured image-classification settings.

  • Takeaways & Limitations

    The paper does not directly compare LAMP with trainable layerwise sparsity methods and focuses on unstructured sparsity.

Abstract

from arXiv · show

Recent discoveries on neural network pruning reveal that, with a carefully chosen layerwise sparsity, a simple magnitude-based pruning achieves state-of-the-art tradeoff between sparsity and performance. However, without a clear consensus on "how to choose," the layerwise sparsities are mostly selected algorithm-by-algorithm, often resorting to handcrafted heuristics or an extensive hyperparameter search. To fill this gap, we propose a novel importance score for global pruning, coined layer-adaptive magnitude-based pruning (LAMP) score; the score is a rescaled version of weight magnitude that incorporates the model-level $\ell_2$ distortion incurred by pruning, and does not require any hyperparameter tuning or heavy computation. Under various image classification setups, LAMP consistently outperforms popular existing schemes for layerwise sparsity selection. Furthermore, we observe that LAMP continues to outperform baselines even in weight-rewinding setups, while the connectivity-oriented layerwise sparsity (the strongest baseline overall) performs worse than a simple global magnitude-based pruning in this case. Code: https://github.com/jaeho-lee/layer-adaptive-sparsity

1 INTRODUCTION

Magnitude-based pruning can be highly effective when layerwise sparsity is chosen well, but existing choices lack consensus. LAMP addresses this gap with an automatically adaptive, distortion-informed score that performs consistently across evaluated settings.

  • Motivation: Magnitude-based pruning can match or outperform more complicated pruning methods when given an appropriate layerwise sparsity.Prior work achieved this using extensive hyperparameter tuning.
  • Problem: Existing magnitude-based pruning methods lack consensus on how to choose layerwise sparsity, which is often selected algorithm-by-algorithm.Global thresholds, feedback heuristics, and network-science-inspired rules represent different selection strategies.
  • Approach: LAMP selects layerwise sparsity through a model-level distortion perspective while preserving global pruning’s automatic layer adaptation.Global LAMP pruning is equivalent to magnitude pruning with automatically chosen layerwise sparsity.
  • Approach: LAMP is a rescaled magnitude score that is efficiently computable, hyperparameter-free, and independent of model-specific knowledge.Its normalization incorporates the pruning state of each layer.
  • Results: LAMP consistently outperforms baseline layerwise sparsity selection schemes across the evaluated architectures and image datasets.Experiments include VGG-16, ResNet-18/34, DenseNet-121, EfficientNet-B0, CIFAR-10/100, SVHN, and Restricted ImageNet.

2 RELATED WORK

Related work studies handcrafted, uniform, global, dynamic, and trainable layerwise sparsity schemes for magnitude-based pruning. This paper focuses on an easy-to-use selection method without modifying the training objective or extensive tuning.

  • Magnitude-based pruning: Magnitude-based pruning has been studied for model compression using iterative, uniform, and heuristic layerwise sparsity schedules.Examples include standard-deviation thresholds, gradual pruning with weight regrowth, and keeping selected layers dense.
  • Pruning at initialization: Pruning at initialization combines magnitude pruning with weight rewinding, using uniform or global sparsity and additional layer-specific heuristics.Different layerwise rules are used for small and larger networks.
  • Scope: The paper does not directly compare with trainable layerwise sparsity methods because its goal is an easy-to-use selector without training-objective changes or extensive tuning.This is an explicit scope choice for the related-work comparison.
  • Scope: The paper focuses on unstructured sparsity, which is described as less practical than structured sparsity despite recent methods aimed at bridging that gap.The authors point to recent breakthroughs as promising ways to improve practicality.

3 LAYER-ADAPTIVE MAGNITUDE-BASED PRUNING (LAMP)

LAMP rescales within-layer weight magnitudes to approximate model-level output distortion during global pruning. Its greedy score is designed to be efficient, preserve surviving connections across layers, and induce automatic layerwise sparsity.

  • Score definition: LAMP flattens each fully connected or convolutional weight tensor and orders weights by ascending magnitude before scoring them.The ordering also handles equal-magnitude weights through an index map.
  • Score definition: The LAMP score measures a connection’s relative importance among same-layer connections that remain after smaller-magnitude weights are pruned.Thus equal magnitudes can receive different scores depending on their index order.
  • Global pruning: Global pruning removes connections with the smallest LAMP scores, producing magnitude pruning with automatically selected layerwise sparsity.Within each layer, larger magnitudes receive larger scores because their denominators are smaller and numerators larger.
  • Global pruning: Unlike global magnitude pruning, LAMP guarantees at least one surviving connection in every layer because each layer has one maximum score of 1.This distinction affects how extreme global sparsity is distributed across layers.
  • Efficient computation: LAMP requires no tuned hyperparameters and adds minimal computational overhead because sorting is already typical in magnitude-pruning algorithms.The implementation uses elementary tensor operations.
  • Model-level distortion: LAMP extends this distortion perspective to the model level, where preceding layers scale inputs and succeeding layers scale pruning-induced output distortion.Because exact optimization is difficult with nonlinearities, the method uses a greedy procedure that repeatedly scores and removes one connection.
  • Distortion motivation: Layerwise magnitude pruning minimizes Frobenius distortion under a layerwise sparsity constraint, motivating its interpretation as a relaxed output-distortion minimization.The relaxation uses the Frobenius norm as an upper bound related to worst-case output distortion.
  • Efficient computation: The distortion upper bound yields a score independent of activation functions and computable in advance without rescoring after every individual pruning step.Cumulative sums of squared weights make the denominator efficient to compute.

4 EXPERIMENTS & ANALYSES

Across diverse architectures, datasets, and pruning settings, LAMP consistently delivers the strongest sparsity–accuracy tradeoff among evaluated layerwise sparsity schemes. It remains effective under one-shot pruning, weight rewinding, and SNIP comparisons.

  • Main results: 88.1% test accuracy at 1.44% surviving weights outperforms Erdős-Rényi kernel’s 77.8% on EfficientNet-B0.The comparison comes from iterative pruning and retraining on CIFAR-10.
  • Ablations: one-shot pruning, weight rewinding, and SNIP: LAMP remains effective in one-shot pruning, with only a 1.09% iterative-over-one-shot accuracy gain at 1.15% surviving weights.At the same sparsity, iterative pruning improves Uniform MP by 41.62% over one-shot pruning.
  • Ablations: one-shot pruning, weight rewinding, and SNIP: Under weight rewinding, LAMP remains beneficial, while Global can outperform Erdős-Rényi kernel in the low-sparsity regime.The authors connect this pattern to preserving larger initial-magnitude connections that may support signal propagation at initialization.
  • Ablations: one-shot pruning, weight rewinding, and SNIP: LAMP achieves performance similar to Global SNIP in the SNIP experiment.The authors suggest this may reflect a shared output-distortion-minimization spirit between LAMP and SNIP’s gradient-distortion minimization.

5 LAYERWISE SPARSITY: GLOBAL MP, ERD ˝OS-R´ENYI KERNEL, AND LAMP

The section examines layerwise sparsity patterns produced by Global MP, Erdős–Rényi kernel, and LAMP, asking whether LAMP resembles experience-based heuristics and reveals architectural structure. LAMP keeps early and late layers relatively dense while tending toward more uniform nonzero-weight counts across layers at extreme sparsity.

  • Figure 5: Figure 5 reports layerwise survival rates and nonzero-weight counts for iteratively pruned VGG-16 models on CIFAR-10.The plotted global survival rates range from 51.2% to 3.52% for survival rates and from 3.52% to 0.24% for nonzero-weight counts.
  • Observed sparsity patterns: LAMP sparsities resemble Erdős–Rényi kernel patterns and the handcrafted heuristic of preserving the first and last layers.Global MP also preserves much of the last fully connected layer, but prunes the first convolutional layer quickly.
  • Observed sparsity patterns: At 3.52% total survival, LAMP leaves approximately 79% and 62% of weights unpruned in the first and last layers, respectively.Erdős–Rényi kernel leaves both layers unpruned at this sparsity level.
  • Observed sparsity patterns: LAMP tends to keep the number of nonzero weights relatively uniform across layers at extreme sparsity levels.Erdős–Rényi kernel instead maintains a constant relative ratio across layers as global sparsity changes.
  • Implications: The authors conjecture that similar nonzero-connection counts per layer may help maximize memory capacity under a global sparsity constraint.They identify theoretical analysis of such sparse networks as a direction for future work.

6 CONCLUSION

The conclusion presents LAMP as a model-level distortion-motivated approach to choosing layerwise sparsity for magnitude-based pruning. It reports consistent gains across models and datasets, including one-shot pruning and weight-rewinding settings.

  • LAMP addresses layerwise sparsity selection for magnitude-based pruning through an ℓ2 distortion minimization perspective.
  • LAMP provides a consistent performance gain across a wide range of models and datasets.
  • LAMP performs reliably well when combined with one-shot pruning schedules or weight rewinding.

A EXPERIMENTAL SETUPS

The experimental setup specifies optimization, augmentation, normalization, and model-adaptation choices for the evaluated image-classification experiments. The listed configurations cover CIFAR-10/100, SVHN, and Restricted ImageNet settings.

  • Optimization: Except for weight rewinding, experiments use AdamW with learning rate 0.0003; weight rewinding uses vanilla Adam at the same learning rate.The setup follows PyTorch defaults for other optimizer hyperparameters.
  • Pre-processing: CIFAR-10/100 training uses random crops with padding 4 and random horizontal flips, followed by dataset normalization.
  • Pre-processing: SVHN training uses random crops with padding 2, and both training and test data are normalized.
  • Pre-processing: Restricted ImageNet uses random resized crops and horizontal flips for training, while testing uses resizing followed by a 224-pixel center crop.
  • Models: VGG-16, DenseNet-121, and EfficientNet-B0 are adapted for CIFAR-10/100 by modifying average pooling and the first fully connected layer.The convolutional layers are not modified for the 32 × 32 input resolution.

B COMPUTATIONAL AND IMPLEMENTATIONAL ASPECTS OF LAMP

LAMP computes layer-adaptive scores before applying global pruning. Its additional per-layer processing remains compatible with the O(n log n) sorting cost that dominates global magnitude-based pruning.

  • LAMP computation: LAMP first squares and sorts weight magnitudes independently within each layer, costing O(Σ_i n_i log n_i).
  • LAMP computation: For each layer, LAMP computes score denominators by summing and storing squared weight magnitudes in descending order, requiring O(n) computation.
  • LAMP computation: LAMP forms scores by dividing squared weight magnitudes by their denominators, requiring O(n) steps.
  • LAMP computation: The final LAMP step sorts and prunes as in global magnitude pruning, taking O(n log n) steps.
  • Complexity: The final sorting-and-pruning step dominates LAMP's cost and is shared with global magnitude pruning.The first three steps can be implemented in PyTorch.
  • Implementation: The cumulative-weight implementation stores shifted cumulative sums of sorted squared magnitudes before score computation.
  • Implementation: Because within-layer LAMP scores are already sorted, the global merge cost can be reduced to a merging step.

C DERIVATION OF INEQUALITY (8)

The derivation bounds model-output distortion from pruning a layer by peeling layers and applying Cauchy–Schwarz, using ReLU’s 1-Lipschitzness and zero-in-zero-out property.

  • Final bound: The resulting bound relates output distortion to the Frobenius norm of the difference between the original and pruned target-layer weights.The displayed inequality contains the factor ∥W^(i) − W̃^(i)∥F.
  • Layer peeling: The proof repeatedly peels outer layers until the target-layer perturbation is isolated in the model-output difference.The authors describe this as a simplified, modified peeling procedure and provide it for completeness.
  • Activation control: ReLU’s 1-Lipschitzness with respect to the ℓ2 norm controls activation changes during peeling.This property is cited for the first inequality in the derivation.
  • Inequality steps: Cauchy–Schwarz supplies the inequality steps used to bound the propagated perturbation.The derivation explicitly applies Cauchy–Schwarz iteratively together with the step leading to Eq. (9).
  • Activation control: The zero-in-zero-out condition σ(0) = 0 is additionally required to peel the activation functions.The proof uses this condition to compare σ(f(x; W^(1:i−1))) with σ(0).

D EXPERIMENTAL RESULTS ON LANGUAGE MODELING

The paper evaluates LAMP and baseline layerwise sparsity schemes on pruned Transformer-XL models using Penn Treebank and WikiText-2. LAMP achieves near-best performance, but its gain is marginal.

  • Model: The experiments use a simplified six-layer Transformer-XL and prune its self-attention layers, comprising approximately 7.57M parameters.The architecture removes some regularizing heuristics from the original model.
  • Training: Models are trained with AdamW, 200,000 iterations before pruning, and 50,000 retraining iterations after pruning.The setup uses learning rate 0.0003, batch size 20, and maximum sequence length 70.
  • Datasets: The language-modeling datasets are Penn Treebank and WikiText-2.The reported tables correspond to Penn Treebank and WT-2 test perplexities.
  • Results: LAMP achieves near-best performance among all considered methods on the language-modeling experiments.The results are reported in Tables 2 and 3.
  • Results: The gain from LAMP is marginal in these Transformer-XL experiments.The authors suspect this is because Transformer layers have relatively uniform widths compared with image-classification models.

E DETAILED EXPERIMENTAL RESULTS (WITH STANDARD DEVIATIONS)

The paper reports detailed image-classification experiments and examines the layerwise sparsity patterns discovered by LAMP. LAMP differentiates convolution types and prunes 3 × 3 filters more heavily, especially in later layers.

  • ResNet-50 sparsity patterns: LAMP, global MP, and Erdős–Rényi kernel pruning are compared on one-shot-pruned ImageNet-pretrained ResNet-50.The comparison reports layerwise survival ratios in Figure 6.
  • ResNet-50 sparsity patterns: LAMP assigns different sparsity ratios to 1 × 1 and 3 × 3 convolution layers, pruning 3 × 3 filters more heavily.The difference becomes more noticeable in later layers, where more weights are pruned.
  • Comparison with AMC: LAMP discovers a layerwise sparsity pattern similar to AMC without training a reinforcement-learning agent.The authors note that AMC uses iterative pruning, whereas Figure 6 uses one-shot pruning.
  • Comparison with baselines: The peaks-and-crests pattern is not prominent under global MP but is extremely evident under the Erdős–Rényi kernel method.The latter pattern is reported as even stronger than that discovered by AMC.
Loading 2010.07611v2…