Source-linked AI summary
SNIP: Single-shot Network Pruning based on Connection Sensitivity
Namhoon Lee, Thalaiyasingam Ajanthan, Philip H. S. Torr
TL;DR
Large neural networks are costly to store and compute, while existing pruning methods rely on pretraining or iterative schedules. SNIP measures task-dependent connection sensitivity at initialization, prunes once, and then trains the sparse network normally. It achieves extremely sparse models with virtually unchanged accuracy across several datasets and architectures, while enabling task relevance of retained connections to be examined.
Problem
Overparameterized neural networks incur excessive computational and memory costs, while existing pruning methods depend on pretraining and expensive iterative prune–retrain cycles.
Method
SNIP uses the normalized magnitude of data-dependent connection-sensitivity derivatives at initialization to prune redundant connections once before standard sparse-network training.
Results
SNIP obtains extremely sparse networks with virtually the same accuracy as reference or baseline models across MNIST, CIFAR-10, and Tiny-ImageNet and varied architectures.
Takeaways & Limitations
The retained connections can be tested for task relevance, and the method applies without modification across convolutional, residual, and recurrent architectures.
Takeaways & Limitations
The method assumes an initialization whose weights keep gradients in a reasonable range, and direct comparisons are constrained by differing specifications, policies, datasets, and tasks.
Abstract
from arXiv · showhide
Pruning large neural networks while maintaining their performance is often desirable due to the reduced space and time complexity. In existing methods, pruning is done within an iterative optimization procedure with either heuristically designed pruning schedules or additional hyperparameters, undermining their utility. In this work, we present a new approach that prunes a given network once at initialization prior to training. To achieve this, we introduce a saliency criterion based on connection sensitivity that identifies structurally important connections in the network for the given task. This eliminates the need for both pretraining and the complex pruning schedule while making it robust to architecture variations. After pruning, the sparse network is trained in the standard way. Our method obtains extremely sparse networks with virtually the same accuracy as the reference network on the MNIST, CIFAR-10, and Tiny-ImageNet classification tasks and is broadly applicable to various architectures including convolutional, residual and recurrent networks. Unlike existing methods, our approach enables us to demonstrate that the retained connections are indeed relevant to the given task.
1 INTRODUCTION
SNIP addresses the cost of overparameterized networks by pruning connections once before training, using task-dependent sensitivity. It targets extreme sparsity, broad architectural applicability, and evidence that retained connections matter for the task.
- Motivation: Overparameterized neural networks impose excessive computational and memory costs, motivating pruning for real-time and resource-limited applications.Compressed networks may also use model capacity more efficiently and support improved generalization bounds.
- Method: SNIP identifies task-important connections before training through a data-dependent connection-sensitivity criterion and prunes redundant connections once.The criterion is evaluated at variance-scaling initialization, after which the sparse network is trained normally.
- Method: Single-shot pruning removes the need for pretraining, complex pruning schedules, and additional hyperparameters.Training the resulting sparse network uses the standard procedure.
- Scope: The criterion is robust to architecture variations and applies without modification to convolutional, residual, and recurrent networks.Its structural focus supports use across the architectures evaluated in the paper.
- Interpretability: Varying the pruning mini-batch enables verification that retained connections are essential for the given task.This provides the paper’s interpretability evidence for the selected structure.
- Results: Across MNIST, CIFAR-10, and Tiny-ImageNet, SNIP produces extremely sparse networks with virtually the same accuracy as existing baselines across widely varying architectures.The study also examines retained-connection relevance, initialization effects, and dataset effects on saliency.
2 RELATED WORK
Prior neural-network compression research includes penalty-based and saliency-based pruning, alongside structural simplification and weight-representation methods. The paper situates its contribution within weight pruning without structural constraints.
- Classical methods: Classical pruning methods either impose sparsity penalties during training or remove weights using saliency criteria.Penalty methods use terms such as L0 or L1 norms, while saliency methods identify removable parameters by a criterion.
- Modern advances: Recent pruning work addresses deep networks’ space, time, and overfitting concerns, with weight magnitude achieving extreme sparsity without accuracy loss.Magnitude-based pruning became a de facto standard and motivated several implementations.
- Network compression in general: Network compression also includes filter pruning, structured regularization, low-rank and matrix factorization, expander graphs, and Erdős-Rényi sparsification.These approaches simplify network structure in ways distinct from unrestricted weight pruning.
- Network compression in general: Other compression methods reduce weight representation through quantization, reduced precision, or binary weights.The paper focuses instead on weight pruning free from structural constraints and compatible with further compression schemes.
3 NEURAL NETWORK PRUNING
Network pruning seeks a sparse network that preserves a dense reference model’s accuracy, but conventional criteria and iterative prune–retrain procedures create practical limitations. SNIP addresses these issues with data-dependent connection importance measured before training.
- Objective: The pruning objective is to learn a much smaller network while maintaining the standard reference network’s accuracy.It can be formulated as a constrained optimization problem limiting the number of non-zero weights.
- Existing approaches: Penalty-based and saliency-based approaches optimize pruning through procedures that remove or constrain redundant parameters.Saliency methods commonly use weight magnitude or Hessian-based criteria.
- Limitations: Magnitude- and Hessian-based criteria depend on weight scale, requiring pretraining and sensitivity to architectural choices such as normalization layers.Hessian computation is also generally intractable for large networks because the matrix is not generally diagonal or positive definite.
- Limitations: Alternating pruning and optimization creates expensive prune–retrain cycles that hinder large-scale use and raise questions about existing criteria.These cycles are a central practical limitation of conventional pruning procedures.
- SNIP: SNIP directly measures connection importance in a data-dependent way, enabling one pruning step at the beginning followed by standard sparse-network training.The authors state that sparse matrix implementations could make the method theoretically an order of magnitude faster than standard training.
4 SINGLE-SHOT NETWORK PRUNING BASED ON CONNECTION SENSITIVITY
SNIP identifies task-relevant connections before training by measuring connection sensitivity at initialization, then retains the most sensitive connections and trains the resulting sparse network once. Variance scaling and task data support robustness across architectures, while experiments report effective pruning with performance close to reference networks.
- Connection sensitivity: SNIP separates connection weights from connectivity indicators to assess each connection’s effect on the loss independently of its weight.The binary indicator c records whether a connection is active, while w contains its weight.
- Connection sensitivity: Connection sensitivity approximates each connection’s loss influence using the derivative with respect to its connectivity indicator, computable for all connections in one forward-backward pass.This derivative is an infinitesimal approximation to the discrete loss difference caused by removing a connection.
- Connection sensitivity: SNIP retains the top-κ connections by normalized derivative magnitude, preserving connections whose removal would substantially affect the loss.The retained connections are selected once before training, with ties broken arbitrarily when necessary.
- Single-shot pruning at initialization: SNIP samples a mini-batch, computes sensitivities, prunes once, and then trains the sparse network with the standard loss minimization procedure.The algorithm initializes weights, samples training data, selects top-κ connections, and performs regular training after pruning.
- Single-shot pruning at initialization: Variance scaling initialization is used to keep signal variance consistent across layers, making initialization-based saliency empirically robust to architecture variations.The authors note that poorly scaled or saturated activations can produce uninformative gradients.
- Empirical evaluation: On MNIST, CIFAR-10, and Tiny-ImageNet, SNIP produces extremely sparse networks with virtually the same accuracy as existing baselines across tested architectures.For LeNet experiments, the reported test errors remain as good as the unpruned reference across varying sparsity levels on both models.
5 EXPERIMENTS
SNIP is evaluated across standard classification benchmarks and diverse architectures, including convolutional, residual, and recurrent networks. The experiments show extreme sparsity with minimal accuracy loss, while analyses examine interpretability, initialization, and random-label fitting.
- Experiments: SNIP produces extremely sparse models with minimal or no accuracy loss across MNIST, CIFAR-10, and Tiny-ImageNet architectures.The evaluation includes a variety of network architectures and compares performance with reference or baseline models.
- LeNet results: SNIP prunes LeNet models at random initialization in a single shot, without pretraining, while retaining performance similar to the reference networks.For LeNet-300-100, accuracy loss is negligible at 90% sparsity; degradation is nearly invisible for LeNet-5-Caffe.
- LeNet results: At 98% and 99% parameter sparsity, errors degrade approximately 0.7% for LeNet-300-100 and 0.3% for LeNet-5-Caffe, respectively.At slightly lower sparsities, SNIP models can outperform the dense reference network.
- Various modern architectures: Across modern architectures, SNIP substantially reduces parameters with less than 1% accuracy loss and requires no architecture-specific modification.The tested models include deep convolutional, residual, and recurrent networks.
- Understanding pruned connections: Visualizations show that retained first-layer connections concentrate on discriminative image regions while irrelevant background connections are pruned.For class-specific mini-batches, surviving connections can reconstruct MNIST digits or Fashion-MNIST silhouettes.
- Effects of data and initialization: Variance-scaling initialization is crucial for reliable, model-agnostic saliency, especially because GRU networks perform poorly without it.VS-H achieves the best performance across the tested models, while initializer differences are marginal on LeNets.
6 DISCUSSION AND FUTURE WORK
SNIP prunes irrelevant connections before training and applies across neural-network models without architectural modifications. The authors also view connection sensitivity as a diagnostic of important connections in an untrained network, motivating future work beyond pruning.
- Figure 4 reports that the SNIP-pruned sparse model does not fit random labels.
- SNIP prunes irrelevant connections once before training and applies to varied neural-network models without modifications.
- Connection sensitivity diagnoses important connections in a purely untrained network.
- The authors identify neural-network architecture understanding, multi-task transfer learning, structural regularization, and sparse-network generalization as future directions.
A VISUALIZING PRUNED PARAMETERS ON (INVERTED) (FASHION-)MNIST
SNIP produces the same pruning results on original and inverted (Fashion-)MNIST, unlike pruning based directly on ∂L/∂w, whose patterns vary across settings.
- SNIP gives the same results on inverted (Fashion-)MNIST as on the original data.
- Using ∂L/∂w produces results different from SNIP on the original datasets.
- ∂L/∂w yields pruning patterns inconsistent across sparsity levels within and between classes.
B FITTING RANDOM LABELS: VARYING SPARSITY LEVELS
Varying sparsity changes the ability of pruned networks to fit random labels, while the same models can still learn the classification task with little accuracy loss.
- Lower sparsity levels produce lower training loss when fitting random labels.
- Networks with more parameters are more vulnerable to fitting random labels.
- All pruned models learn the classification task without losing much accuracy.
C TINY-IMAGENET
Tiny-ImageNet uses 200 classes, 64×64 images, and doubled first-convolution stride to accommodate its higher resolution than CIFAR-10. Table 4 summarizes SNIP pruning results before and after pruning.
- Tiny-ImageNet has 200 classes, with 500 training images and 50 validation images per class.
- Each Tiny-ImageNet image has spatial resolution 64×64, twice the resolution of CIFAR-10.
- All architectures double the first-convolution stride for Tiny-ImageNet following standard practice.
D ARCHITECTURE DETAILS
The architecture details specify AlexNet and VGG variants for CIFAR-10 and Tiny-ImageNet, including class counts and resolution-dependent stride changes.
- AlexNet-s uses k = 1, while AlexNet-b uses k = 2.
- The final-layer class count is c = 10 for CIFAR-10 and c = 200 for Tiny-ImageNet.
- For Tiny-ImageNet, the first convolution stride changes to [4, 4] in AlexNet and [2, 2] in VGG to accommodate higher image resolution.
- VGG-C, VGG-D, and VGG-like are described, with the second Linear layer used only in VGG-C/D.