Single Shot Structured Pruning Before Training

• Earlier pruning-before-training methods remove individual weights, which improve speed only on specialized hardware and do not reduce runtime memory footprint.Source: §1 · p. 1
• 2x faster GPU training and 3x faster CPU inference come with only 0.5% accuracy loss on CIFAR-10.Source: §1 · p. 1
• 3SP removes entire channels and hidden units before training, using one forward-backward pass on a single minibatch.The method is described as easy to implement with few hyper-parameters to tune.Source: §1 · p. 1
• Compute-aware scoring measures loss impact per unit of compute removed and raises compute reduction from 60% to 85%.It favors removing channels with either small loss impact or high computational cost.Source: §1 · p. 1
• In active learning, 3SP can help identify valuable data to acquire faster and achieve better accuracy within a time budget than an unpruned model.Source: §1 · p. 2

• SNIP prunes individual weights before training by choosing a binary mask that minimizes the change in loss, approximated from mask gradients on one training batch.It uses a first-order approximation and assumes weight importance is independent of which other weights are removed.Source: §2 · p. 2
• GraSP preserves gradient flow, but its approximation has shortcomings that are particularly problematic for structured pruning, so the paper focuses on SNIP.Source: §2 · p. 2
• Structured pruning removes entire convolutional channels or linear-layer columns, reducing activation size and model compute unlike individual-weight pruning.Source: §2 · p. 2, Figure 1 · p. 2
• Structured pruning strengthens the independence assumption: removing one channel can remove hundreds of weights, making interactions among retained mask entries more consequential.The passage gives 512 × 3 × 3 weights as an example from later VGG and ResNet layers.Source: §2 · p. 3

3.1 Structured Pruning Before Training

• 3SP assigns one mask variable to each convolutional output channel or linear hidden unit, allowing masked operations to be removed from the weight tensors.This produces a smaller model rather than applying an element-wise mask.Source: §3.1 · p. 4
• The structured score uses a continuous-mask, first-order Taylor approximation from one minibatch and assumes individual mask-entry effects add when entries are changed together.Source: §3.1 · p. 4
• Structured scores are noisier than unstructured scores, but their predicted loss changes still correlate strongly with actual changes.The paper notes that the structured approximations require stronger justification because mask-entry interactions can be larger.Source: §3.1 · p. 4
• Pruning reduces output channels and changes activation variance at initialization; the proposed weight rescaling does not exactly reproduce scratch initialization because weights are not pruned uniformly.Source: §3.1 · p. 4

3.2 Compute-Aware 3SP (3SP + CA)

• Compute-aware 3SP divides scores by normalized per-channel compute cost, while smoothing parameter λ controls how much pruning favors loss impact over layer compute differences.A larger λ makes pruning depend more on predicted loss change and less on compute cost.Source: §3.2 · pp. 4–5
• At every amount of compute pruned, 3SP outperforms uniform pruning; after 75% of FLOPs, only compute-aware 3SP removes more without pruning entire layers.Source: Figure 4 · p. 5

4 Experiments

• 3SP is comparable to unstructured methods on CIFAR-10 parameter sparsity, but a gap appears on CIFAR-100, where 3SP still beats Uniform.At 80% parameter sparsity it has similar CIFAR-10 accuracy; at 95% it is less than one percentage point below SNIP and GraSP. Reinitialization matters more on CIFAR-100.Source: Table 1 · p. 6, §4 · p. 6
• 3SP preserves performance while removing less compute than Uniform, whereas 3SP + CA with λ = 0 prunes more aggressively and reaches very high compute sparsity.At very high compute-removal levels, 3SP can be forced to remove entire layers, while the compute-aware extension can continue pruning.Source: §4 · p. 6
• 80% or 50% FLOPs reduction yields 4x or 2x faster GPU training per epoch, and 5x or 3x faster CPU prediction, respectively.The Table 2 experiments use VGG-19 on CIFAR-10; prediction uses a single-element batch on an i7 8700K CPU.Source: §4 · p. 6, Table 2 · p. 7

4.1 Active Learning with 3SP Pruning

• Within a time budget, 3SP at 50% and 3SP-CA at 80% FLOPs reduction achieve better CIFAR-10 active-learning accuracy than a full model.The pruned model trains faster and can acquire more data within the same time budget, despite pruning hurting model performance.Source: Figure 5 · p. 7, §4.1 · p. 7

5 Alternative Pruning Criterion - GraSP

• GraSP scores the predicted change in gradient magnitude to retain weights that contribute to gradient norm, rather than using SNIP’s predicted loss change.The method uses a first-order Taylor expansion and smooths outputs to avoid a trivial mask that creates a very large loss.Source: §5 · p. 7
• 91.96%, 91.6%, and 90.92% accuracy at 80%, 90%, and 95% prune ratios, respectively; structured GraSP performs worse than Uniform pruning on CIFAR-10.The authors say the naive GraSP form cannot directly apply because it computes gradients with respect to weights rather than mask variables.Source: §5 · p. 7
• Structured GraSP’s predictions are uncorrelated with the actual objective change, while unstructured predictions show correlation; the authors attribute this to noisy structured estimates.Figure 6 compares Taylor-predicted and actual normalized gradient-norm changes for VGG-19.Source: §5 · pp. 7–8, Figure 6 · p. 8

6 Related Work

• Earlier compression methods reduce evaluation cost, but the cited work does not focus on reducing computation both during and after training; SNIP and GraSP prune before training but are unstructured.The paper notes these methods cannot substantially reduce compute cost.Source: §6 · p. 8
• Wang et al. [28] use channel-wise masks with a sparsity penalty, but pruning on CIFAR-10 requires 10 training epochs at a cost similar to the original model, unlike 3SP’s one-batch scoring.The authors consider that additional cost prohibitive and do not explicitly compare results.Source: §6 · p. 8
• Post-training methods include FLOP-regularized pruning, approximate constrained optimization, reinforcement-learning model crafting, and structured masks; Wang et al. [26] use 160 full-model training epochs plus 160 fine-tuning epochs.Source: §6 · p. 8
• NAS optimizes architecture before training, whereas this method prunes with one forward-backward pass on a single batch rather than extensive retraining.The paper also notes that NAS methods have been criticized for high energy consumption [22].Source: §6 · p. 8

7 Conclusion

• 3SP speeds training by 2x and inference by 3x while losing only 0.5% accuracy.Source: §7 · p. 8
• 3SP is a single-shot structured pruning technique applied before training.Source: §7 · p. 8
• The authors report that SNIP and its structured extension have reasonable approximations, while compute-aware 3SP gives more speedups at the expense of accuracy.They characterize structured pruning before training as a feasible way to reduce training cost and architecture-selection time.Source: §7 · p. 9

Broader Impact

• The method is framed as a mechanism for reducing training energy use and carbon dioxide emissions, using one batch of backpropagation rather than extensive architecture-search evaluations.The authors connect this to energy spent on training and architecture search [22].Source: Broader Impact · p. 9
• The authors identify resource-poor teams without large datacenters as a group that may particularly benefit from a more compute-efficient method.Source: Broader Impact · p. 9
• The paper proposes compute-space pruning as a framing for future reductions in energy consumption, while noting that energy-space pruning is too hardware-dependent for this paper’s generality.Source: Broader Impact · p. 9
• Greater computational efficiency could increase deep-learning demand and thereby increase overall energy consumption, a possible instance of the Jevons Paradox.Source: Broader Impact · p. 9

A Experimental setup

• VGG-19 uses five convolutional blocks followed by average pooling and three linear layers; all elements except the final linear output and first convolution input are prunable.The convolutional blocks have widths 64, 128, 256, 512, and 512; the linear layers have 1024, 512, and class-count units.Source: A Experimental setup · p. 12
• Training runs for 160 epochs with learning-rate drops at epochs 80 and 120, using normalized CIFAR-10 data with horizontal flips and padded random crops.Results are reported on the CIFAR-10 test set and average five runs with different random seeds.Source: A Experimental setup · p. 12
• Full-model computational cost is counted with pytorch-OpCounter2, excluding preprocessing.Source: A Experimental setup · p. 12

B Wall Clock Time and Model Size Experiments

• The 50% FLOPs-reduction model is 3SP without compute awareness, while the 80% model uses compute awareness.Source: B Wall Clock Time and Model Size… · p. 12
• Training time per epoch averages five epochs after excluding the first, and prediction time averages per-point inference while processing an epoch one data point at a time.Source: B Wall Clock Time and Model Size… · p. 12
• Model size is measured by saving parameters to disk; pruning time is a single-batch forward pass, with architecture-adjustment overhead excluded and estimated below 10 ms.Source: B Wall Clock Time and Model Size… · p. 12

C Active Learning Experiments

• Starting from 1,000 labeled CIFAR-10 examples, the method selects 50 points per acquisition step using softmax entropy as the informativeness proxy, and prunes once before the timed process.Source: C Active Learning Experiments · p. 12
• The comparison uses 3SP at 50% FLOPs reduction without compute awareness and 3SP + CA at 80% reduction, with five repeats and one-standard-deviation errors.Source: C Active Learning Experiments · p. 13
• Almost 50% more data: 3SP + CA sees this much more than the unpruned model within the time budget, while Figure 7 shows both pruned configurations acquire more data.Source: Figure 7 · p. 12, C Active Learning Experiments · p. 13