Revisiting Random Channel Pruning for Neural Network Compression

1. Introduction

• The paper presents channel pruning as an underexamined empirical problem and defines random pruning by randomly selecting each layer’s pruning ratio while using a criterion to choose channels within that layer.It notes that random pruning is often used as a baseline for state-of-the-art methods [11] [12] [36] [40] [43] [48] [50].Source: §1 · p. 1
• Under the same random-pruning setting, recent channel-pruning criteria [19] [37] [45] [50] perform comparably to simple L1- and L2-norm criteria.The paper presents this as finding F1.Source: §1 · p. 1
• Random pruning finds models with comparable or sometimes superior performance to methods starting from pretrained models [9] [17] [18] [19] [23] [38] [39] [40] [46] [61].The paper presents this as finding F2 and points to Table 3.Source: §1 · p. 1
• On ImageNet classification, random pruning narrows the performance gap with advanced architecture-optimizing methods to less than 0.5%.The compared methods optimize pooling-layer placement or expand available network width.Source: §1 · p. 1
• Increasing fine-tuning epochs can substantially improve pruned-network performance, so the paper calls for standardized, clearly reported fine-tuning protocols.The findings describe high-performing pruned networks as usually involving prolonged fine-tuning.Source: §1 · pp. 1–2
• Random pruning samples channel configurations to search for compact networks and serves as a simplified, general baseline for comparing pruning methods.The paper studies both pruning pretrained networks and pruning from scratch, and calls for more efficient sampling.Source: §1 · p. 2

2. Related Works

• Structured pruning removes channels to support practical compression and acceleration, whereas unstructured pruning removes individual parameters and is harder to implement with current hardware and software.The paper situates channel pruning among compression approaches such as quantization [7] [13] and low-rank approximation [24] [32].Source: §2 · p. 2
• Pruning criteria use weight magnitudes, feature maps, loss gradients, or alternatives such as KL-divergence, simulated annealing, importance sampling, and learned Dirichlet distributions.Examples include L1/L2 norms [57], median pruning [19], feature-map methods [38], Taylor approximations [50], KL-divergence [45], simulated annealing, importance sampling [3], and Dirichlet learning [1].Source: §2 · p. 2
• Some pruning methods combine pruning with knowledge distillation between large and small networks, sometimes alongside generative adversarial learning.The cited examples include [21] [31] [34] [55] and [39].Source: §2 · p. 2
• The paper argues that importance scores alone do not specify how many parameters to prune or justify the resulting architecture, motivating random architecture search for a predefined model.It presents the search as a simple, unbiased, and general approach to comparing most pruning methods.Source: §2 · p. 3
• Unlike prior comparisons [42], this study aims to establish random pruning as a neutral baseline and reports advanced criteria as comparable to naive L1/L2 criteria.It further calls for optimized sampling to make the search more efficient.Source: §2 · p. 3

3.1. Basic concepts and formalization

• The paper studies random channel-number selection: layer pruning ratios are sampled, while within-layer channels are removed by an importance criterion.Source: §3.1 · p. 3
• The channel configuration space contains possible per-layer channel counts; pruning seeks the highest-accuracy configuration at fixed complexity, and nearby configurations can have similar accuracy, letting random search sample near an optimum.Source: §3.1 · p. 3, Figure 1 · p. 3
• The paper considers pruning pretrained networks by importance scores and pruning from scratch via parallel training of sampled sub-networks followed by architecture search.Source: §3.1 · pp. 3–4
• Severe pruning can make direct accuracy evaluation unreliable (30% MobileNetV2 computation pruning yielded 90% Top-1 error); feature matching or parallel training is used to evaluate candidates.Source: §3.1 · p. 4

3.2. Pruning criteria

• L1/L2 filter norms rank lower-magnitude channels for pruning, though batch normalization and filter distributions motivate alternatives.Source: §3.2 · p. 4
• Alternatives include geometric-median similarity, KL divergence, Taylor-based prediction-error estimates, and feature-map sensitivity; the paper compares six metrics under random pruning.Source: §3.2 · p. 4

4. Pruning Pre-trained Networks

• For a pretrained model, the pipeline scores channels, samples layerwise pruning ratios, removes lowest-scored channels, updates parameters by feature matching, evaluates candidates, then fine-tunes selected models.Source: §4 · pp. 4–5, Figure 2 · p. 5

4.1. Random sampling

• Layerwise ratios are sampled independently and retained only when FLOPs fall within threshold T of target pruning ratio γ; η limits per-layer pruning to avoid bottlenecks.Source: §4.1 · p. 5

4.2. Updating network parameter

• Each pruned network is updated by minimizing squared distance between feature maps; a least-squares X is folded into layer parameters without adding inference computation.Source: §4.2 · p. 5

5. Pruning From Scratch

• From scratch, a slimmable network trains four subnetworks per minibatch (full plus three random); candidates are validated, the top 50 briefly trained, and the best retrained from scratch.Parallel training supports direct validation of sampled candidates by interpolating the accuracy of unsampled subnetworks.Source: §5 · pp. 5–6, Figure 3 · p. 6

5.1. Designing the network pruning space

• The search space restricts channel counts to multiples of eight, rounds minimum widths to avoid narrow bottlenecks, and does not expand maximum width.These constraints reduce the number of configurations and preserve a fair comparison with pretrained-network pruning.Source: §5.1 · p. 6

6. Experimental Results

• Experiments use VGG, ResNet variants, and MobileNetV2; ImageNet uses PyTorch pretrained models, while CIFAR models are trained for 300 epochs.For pretrained models, the top five sampled architectures are fine-tuned for five epochs, then the best is fine-tuned again; the sample population is 100 and T = 0.02.Source: §6 · p. 6

6.1. Benchmarking channel pruning criteria

• Across networks and datasets, the seven pruning criteria perform closely, with no clear winner; advanced criteria do not necessarily outperform L1 or L2.At similar computational complexity, pruned networks with more parameters tend to have lower error; with more than 25 fine-tuning epochs, prolonged training can substitute for advanced criteria's benefits.Source: §6.1 · p. 7
• For the criteria benchmark, random sampling fixes FLOPs relatively closely but allows parameter counts to vary, and actual overall pruning ratios remain within 2% of target.The comparison includes L1, L2, GM, TE, KL, ES, and pruning from scratch with slimmable networks.Source: §6.1 · p. 7

6.2. Benchmarking channel pruning methods

• On CIFAR10, random pruning performs no worse than compared methods; on ImageNet it beats earlier methods under fewer fine-tuning epochs and severer pruning, and is comparable with HRank.It performs a little worse than MetaPruning, whose baseline and fine-tuning favor FLOPs reduction and whose fine-tuning is longer.Source: §6.2 · p. 8

6.3. Ablation study

• Increasing random samples from 20 to 1000 steadily lowers Top-1 and Top-5 error, without saturation in the studied range, though the accuracy gain diminishes.More samples can yield both better and worse sub-networks, motivating advanced search methods.Source: §6.3 · p. 8
• Increasing ResNet-50 fine-tuning from 25 to 100 epochs lowers Top-1 error by 0.75% and Top-5 error by 0.4%.The authors note that fine-tuning strategy should be considered when benchmarking.Source: §6.3 · p. 8

7. Conclusion

• Random pruning is presented as a simple, general, explainable baseline for benchmarking more complex pruning methods.The search applies both to learning architectures from scratch and to pretrained models using predefined channel-importance scores.Source: §7 · p. 8

• The passage is headed “Neural Network Compression.”Source: Revisiting Random Channel Pruning for · p. 12

Supplementary Material

• The supplement covers differences from prior work, a justification of the pruned-network performance upper bound, residual-block pruning with skip connections, and additional experimental results.Source: Supplementary Material · p. 12

A. Difference with Other Works

• Unlike [42], which frames pruning as architecture discovery and mostly evaluates methods pairwise against networks trained from scratch, this work proposes random pruning as a neutral baseline and compares six criteria plus one architecture-search method.Source: A. Difference with Other Works · p. 12
• Unlike ’s single-layer analysis, this study evaluates overall network performance and finds comparable results across norm-, importance-, sensitivity-, and search-based methods.The paper says the empirical study in did not support similarity across importance-, BN-, and activation-based criteria.Source: A. Difference with Other Works · p. 12
• The authors conclude that advanced pruning criteria perform comparably to naive L1/L2 norms, motivating more efficient sampling methods.They contrast this conclusion with, which encourages designing more reasonable criteria.Source: A. Difference with Other Works · p. 12

B. Upper Bounded Performance of Channel Pruning.

• Structured channel-pruned networks are upper-bounded by the original networks unless the search space expands; unlike unstructured pruning, they face an accuracy–compression trade-off.The authors cite changing pooling-layer positions or widening networks as examples of expanding the search space.Source: B. Upper Bounded Performance of… · pp. 12–13

C. Pruning Residual Blocks

• Residual-block skip connections link block inputs and outputs, so connected output channels must be pruned together using the same pruning ratio.This shared ratio applies both when pruning a pretrained network and when pruning from scratch.Source: C. Pruning Residual Blocks · pp. 13–14
• For skip-connected layers, the paper combines L1/L2/GM and Taylor scores, uses maximum empirical sensitivity, and shares masks for KL scoring.The shared masks let the same KL divergence score be computed for all skip-connected layers.Source: C. Pruning Residual Blocks · p. 14

D. More Experimental Results

• Additional CIFAR10 and CIFAR100 experiments strengthen the conclusion that channel-selection criteria are less important under random pruning.The compared criteria include L1, L2, GM, TE, KL, and ES.Source: D. More Experimental Results · p. 14
• When fine-tuning increases from 25 to 100 epochs, ResNet-50 Top-1 and Top-5 errors drop 0.75% and 0.4%, while ResNet-18 errors drop 0.97% and 0.62%, respectively.The passage identifies these changes as showing the significant influence of fine-tuning epochs.Source: D. More Experimental Results · p. 14
• At 70% pruning, sampled ResNet50 subnetworks span good and less accurate outcomes across FLOPs and parameter counts before fine-tuning.At this stage, the networks are updated only by minimizing squared differences between pruned and original feature maps.Source: D. More Experimental Results · p. 14, Figure 7 · p. 15, Figure 8 · p. 15