PaperlayerView exact revision on arXiv

Source-linked AI summary

Progressive Skeletonization: Trimming more fat from a network at initialization

Pau de Jorge, Amartya Sanyal, Harkirat S. Behl, Philip H. S. Torr, Gregory Rogez, Puneet K. Dokania

arXiv:2006.09081v5cs.CVcs.LG

TL;DR

Pruning at initialization can reduce the cost of finding sparse networks, but existing approaches may underperform random pruning at extreme sparsity. The paper defines FORCE to assess connection sensitivity after pruning and proposes progressive approximations; the authors report stronger results at extreme sparsity and at least comparable performance at moderate sparsity.

  • Problem

    Finding sparse subnetworks that can train from scratch can require training dense and intermediate networks, while existing initialization-time methods can fail at high pruning levels.

    Source: §1 · pp. 1–2

  • Method

    FORCE defines connection sensitivity on a pruned network, and the paper approximates its optimization with progressive procedures, including one that allows pruned parameters to resurrect.

    Source: §4 · pp. 4–5

  • Results

    The proposed methods are significantly better than existing approaches at extreme sparsity and at least as good at moderate sparsity.

    Source: §6 · p. 9

  • Takeaways & Limitations

    The authors report that progressive skeletonization can find trainable subnetworks at extreme sparsity, with reasonable cost for iterative methods.

    Source: §6 · p. 9

    The authors provide no corresponding theoretical guarantee for FORCE because there is no obvious link between step size and distance between masks.

    Source: §4 · p. 6

Abstract

from arXiv · show

Recent studies have shown that skeletonization (pruning parameters) of networks \textit{at initialization} provides all the practical benefits of sparsity both at inference and training time, while only marginally degrading their performance. However, we observe that beyond a certain level of sparsity (approx $95\%$), these approaches fail to preserve the network performance, and to our surprise, in many cases perform even worse than trivial random pruning. To this end, we propose an objective to find a skeletonized network with maximum {\em foresight connection sensitivity} (FORCE) whereby the trainability, in terms of connection sensitivity, of a pruned network is taken into consideration. We then propose two approximate procedures to maximize our objective (1) Iterative SNIP: allows parameters that were unimportant at earlier stages of skeletonization to become important at later stages; and (2) FORCE: iterative process that allows exploration by allowing already pruned parameters to resurrect at later stages of skeletonization. Empirical analyses on a large suite of experiments show that our approach, while providing at least as good a performance as other recent approaches on moderate pruning levels, provides remarkably improved performance on higher pruning levels (could remove up to $99.5\%$ parameters while keeping the networks trainable). Code can be found in https://github.com/naver/force.

1 INTRODUCTION

Pruning at initialization can avoid the cost of training dense networks, but existing saliency approximations become unreliable at high pruning levels. The paper introduces FORCE and progressive procedures to reassess connection importance after pruning.

1 INTRODUCTION

  • Finding sparse subnetworks that train from scratch can require training the full model and intermediate subnetworks, making discovery expensive.The cited lottery-ticket approach provides empirical evidence that such subnetworks exist.Source: §1 · p. 1
  • Pruning at initialization estimates connection importance before training and may provide sparsity benefits during both training and inference.The figure compares test accuracy on CIFAR-10 with ResNet50 across pruning methods; each point averages three runs.Source: §1 · p. 1, Figure 1 · p. 2
  • At high pruning levels, SNIP and GRASP can fall below random pruning, while the proposed progressive procedures are more robust across pruning levels.The authors argue that approximations of gradients after pruning do not hold when many weights are removed at once.Source: §1 · p. 2

2 RELATED WORK

Prior pruning work either trains dense networks before pruning or induces sparsity during training. Pruning at initialization instead seeks a sparse subnetwork before any weight update.

2 RELATED WORK

  • Train–prune–fine-tune methods require dense convergence followed by repeated pruning and fine-tuning; lottery-ticket discovery also uses a costly procedure.These methods rely on a trained network, whereas this paper seeks a trainable subnetwork with randomly initialized weights.Source: §2 · p. 2
  • Methods that induce sparsity through loss modification or dynamic pruning still train the network to select the final sparse subnetwork.The paper focuses on finding sparse subnetworks before any weight update, which it says is not directly comparable.Source: §2 · p. 2
  • Pruning at initialization is presented as offering sparsity gains at both training and inference time, and as a form of neural architecture search.The cited work frames this direction as having theoretical and practical interest.Source: §2 · p. 2
  • SNIP and GRASP estimate importance at initialization, but the paper says neither performs sufficiently well when larger compressions require pruning more weights.SNIP estimates a weight’s later training importance; GRASP uses gradient norm after pruning.Source: §2 · p. 3

3 PROBLEM FORMULATION: PRUNING AT INITIALIZATION

Pruning at initialization requires selecting a sparse support before training and keeping the network’s topology fixed during subsequent updates. Exhaustive search over candidate supports is impractical, motivating data-dependent criteria such as SNIP.

3 PROBLEM FORMULATION: PRUNING AT INITIALIZATION

  • The general pruning formulation constrains the final optimal iterate to have at most k nonzero parameters, but places no such constraint on intermediate iterates.The empirical risk is minimized over parameters subject to a constraint set.Source: §3 · p. 3
  • Pruning at initialization requires an initial parameter vector with at most k nonzero entries and constrains every later iterate to the subspace defined by that support.The support fixes the network topology throughout training.Source: §3 · p. 3
  • Exhaustively searching the possible subspaces to find the best one is impractical because it would require training neural networks.The paper therefore discusses hand-designed, data-dependent objectives to select an initialization.Source: §3 · p. 3
  • SNIP selects connections using their sensitivity at initialization, aiming to preserve parameters with maximum impact on the loss when perturbed.Its retained parameters are those with the top-k absolute values of the connection-sensitivity scores.Source: §3 · p. 3

4 FORESIGHT CONNECTION SENSITIVITY

FORCE defines connection sensitivity on a pruned network as a proxy for trainability. The paper approximates its optimization with progressive pruning, including a variant that can restore previously pruned parameters.

4 FORESIGHT CONNECTION SENSITIVITY

  • FORCE measures connection sensitivity at a pruned network and uses it as a proxy for trainability.The objective is to find a mask with maximum sensitivity after pruning.Source: §4 · p. 4
  • Unlike SNIP’s pre-pruning sensitivity, FORCE evaluates weights and gradients at the pruned network, while SNIP and GRASP can become unreliable under extreme pruning.The paper attributes this unreliability to large differences between gradients before and after pruning.Source: §4 · p. 4
  • The exact FORCE objective is difficult to optimize because it would require computing gradients for all possible subnetworks.The proposed approximations progressively increase pruning and solve an approximation at each stage.Source: §4 · p. 4
  • Iterative SNIP re-evaluates connection importance after each pruning step but cannot restore parameters already removed.For one iteration, this procedure recovers one-shot SNIP.Source: §4 · p. 5
  • FORCE allows zeroed parameters to retain gradients and potentially resurrect during progressive sparsification, enabling exploration of subnetworks.At the target sparsity, zero-valued connections are pruned; this is valid only if their removal does not adversely affect gradient flow of unpruned parameters.Source: §4 · p. 5
  • Both iterative procedures rely on small pruning steps to make the gradient approximation reliable; the authors report a simple exponential schedule works well.They report empirical robustness to the number of steps T.Source: §4 · p. 5
  • Iterative SNIP’s intermediate masks are approximate local minima, with suboptimality increasing with pruning step size; FORCE has no corresponding guarantee.The authors report that FORCE empirically often improves over Iterative SNIP, but do not establish global optimality.Source: §4 · p. 6

5 EXPERIMENTS

Experiments compare gradual pruning with other methods across datasets and architectures, then examine how saliency optimization, iteration count, and pruning cost affect results. Gradual methods are especially strong at high sparsity, with FORCE and Iterative SNIP retaining substantially more accuracy than one-shot approaches on CIFAR-10.

5.1 RESULTS ON CIFAR-10

  • At 99.9% sparsity, FORCE and Iterative SNIP obtain more than 80% accuracy, while one-shot approaches are near 10%; gradual methods also improve over other methods at moderate sparsity.On CIFAR-10, comparisons include SNIP and GRASP variants using T = 300 batches.Source: §5.1 · p. 6, Figure 2 · p. 6

5.2 RESULTS ON LARGER DATASETS

  • CIFAR-100 and Tiny ImageNet follow CIFAR-10 trends; on ImageNet, gradual pruning beats SNIP by a larger gap as sparsity rises, and FORCE removes 90% of VGG weights with less than 3% accuracy loss.On ImageNet, GRASP leads at 95% sparsity for VGG and slightly surpasses FORCE at 90% for ResNet50, but falls below random pruning at 95% for ResNet50; other methods also fall below random pruning there. The paper also reports superior FORCE performance on MobileNet-v2.Source: §5.2 · pp. 7–8, Figure 3 · p. 7, Table 1 · p. 7

5.3 ANALYSIS

  • Iterative pruning improves FORCE saliency more at higher sparsity, while at 99.5% sparsity accuracy is robust to T and reaches up to 90% for FORCE and 89% for Iterative SNIP.With T = 1, SNIP yields sub-networks unable to train at 10% accuracy; a small number of iterations already brings a large boost.Source: §5.3 · p. 8, Figure 4 · p. 8
  • At equal batch counts, FORCE-based gradual pruning is much cheaper than GRASP-MB, and iterative pruning costs negligibly more than SNIP-MB relative to training.GRASP's Hessian-vector product increases its memory and time cost.Source: §5.1 · p. 7, §5.3 · p. 8
  • FORCE tends to outperform Iterative SNIP more often, plausibly through exploration as pruned weights recover, but Iterative SNIP is likelier to yield a better mask when only a few iterations are affordable.The paper notes that FORCE may need more iterations to converge to a good subspace.Source: §5.3 · pp. 8–9
  • Early pruning often underperforms SNIP-MB for ResNet and drops sharply at high sparsity for VGG; iterative GRASP quickly yields untrainable networks, though another gradient approximation is more robust than GRASP.The early-pruning comparison does not establish that every early-pruning strategy is suboptimal.Source: §5.3 · p. 9

6 DISCUSSION

The paper reports that progressive skeletonization with FORCE and its approximations improves pruning at extreme sparsity while matching existing methods at moderate sparsity. It argues that these results motivate further study of exploration–exploitation and pruning schedules.

6 DISCUSSION

  • At extreme sparsity, existing initialization methods mostly perform below random pruning, while the proposed methods significantly outperform existing approaches and remain at least as good at moderate sparsity.The authors also report theoretical insights supporting progressive skeletonization and say iterative methods have reasonable cost relative to existing methods.Source: §6 · p. 9
  • FORCE measures connection sensitivity after pruning, and two approximations progressively optimize it to prune networks at initialization.The discussion identifies gradient-norm and FORCE optimization as objectives compatible with gradual pruning, unlike every approximation considered.Source: §6 · p. 9
  • The authors propose further study of pruning's exploration–exploitation trade-off and more efficient schedules, beyond pruning at initialization.Source: §6 · p. 9

A PRUNING IMPLEMENTATION DETAILS

The implementation evaluates pruning at initialization on CIFAR and ImageNet datasets with specified training setups, while restricting pruning to convolutional and fully connected layers. Additional plots focus on accuracy differences at higher accuracies.

A PRUNING IMPLEMENTATION DETAILS

  • CIFAR experiments train ResNet50 and VGG19 for 350 epochs, while Tiny-ImageNet runs for 300 epochs; ImageNet uses adapted official code and default settings.The CIFAR setup uses Kaiming normal initialization, SGD with momentum 0.9, and learning-rate reductions during training.Source: A PRUNING IMPLEMENTATION DETAILS · p. 12
  • SNIP and GRASP use different mini-batch selection procedures; the experiments use SNIP's simpler procedure for both methods, which may cause small discrepancies from Wang et al. (2020).GRASP's implementation selects exactly 10 examples of each class, whereas SNIP loads a batch from the dataloader.Source: A PRUNING IMPLEMENTATION DETAILS · p. 12
  • Pruning is restricted to convolutional and fully connected layers, and sparsity is reported relative to those prunable parameters.For CIFAR-10, batch normalization plus biases account for 0.2% and 0.05% of parameters in ResNet50 and VGG19, respectively.Source: A PRUNING IMPLEMENTATION DETAILS · p. 12
  • Pruning batch-normalization layers and biases is left for future experiments.Source: A PRUNING IMPLEMENTATION DETAILS · p. 13

B ADDITIONAL ACCURACY-SPARSITY PLOTS

  • The additional accuracy-sparsity plots cut the y-axis to make smaller differences at lower sparsities easier to see.Figure 5 reports test accuracies on CIFAR-10/100 and Tiny-ImageNet, averaging three prune-train-test runs with shaded standard deviations.Source: B ADDITIONAL ACCURACY-SPARSITY PLOTS · p. 13, Figure 5 · p. 13

C FURTHER ANALYSIS OF PRUNING AT INITIALIZATION

The analysis compares how iterative pruning changes FORCE saliency and examines whether saliency measured on sparsified networks reflects that of pruned networks. FORCE and Iterative SNIP show similar saliency behavior as iteration count increases, and the two measurement settings are strongly correlated.

C.1 SALIENCY VS T

  • For higher sparsity levels, FORCE obtains higher saliency as the number of iterations increases, and FORCE and Iterative SNIP show similar relative-saliency behavior.Source: §C.1 · p. 13, Figure 6 · p. 14

C.2 PRUNING VS SPARSIFICATION

  • FORCE computes saliency on sparsified networks by zeroing weights while allowing gradients through the connections; pruning also zeros their gradients.Source: §C.2 · p. 13
  • Saliency measurements on sparsified and pruned networks are strongly correlated, supporting the assumption that maximizing sparsified-network saliency also maximizes saliency for the corresponding pruned networks.Source: §C.2 · p. 14

C.3 NETWORK STRUCTURE AFTER PRUNING

The appendix analyzes network structure and pruning-mask evolution at high sparsity, and compares gradual pruning with an early-pruning baseline. Early pruning is competitive for VGG at moderate sparsity but degrades as sparsity rises, and is suboptimal for ResNet across evaluated levels.

C.3 NETWORK STRUCTURE AFTER PRUNING

  • At 99.9% sparsity, analyzed methods preserve initial and final layers and prune deep convolutional layers more heavily; FORCE has a distinct structure because it can recover pruned weights.FORCE and Iterative SNIP preserve more weights in deeper VGG19 layers than GRASP and SNIP, whereas the opposite pattern is observed for ResNet50.Source: §C.3 · p. 14, Figure 8 · p. 15
  • At extreme sparsity, ResNet50 can reach 99.99% without falling to random accuracy, whereas Iterative SNIP cannot prune VGG19 beyond 99.9%; skip connections allow fully pruned ResNet layers to retain signal flow.The authors suggest that fewer layers with more weights may be more efficient than many extremely sparse layers when high sparsity is required.Source: §C.3 · p. 14

C.4 EVOLUTION OF PRUNING MASKS

  • FORCE pruning and recovery peak before rapidly declining, indicating an exploration phase followed by convergence, while Iterative SNIP has no recovered weights and its pruning amount decays exponentially.Source: §C.4 · p. 16, Figure 9 · p. 16

C.5 COMPARISON WITH EARLY PRUNING

  • Early pruning is competitive for VGG at moderate sparsity but degrades as more weights are pruned, while it is suboptimal for ResNet at all evaluated sparsity levels.The plotted CIFAR-10 test accuracies average three prune-train-test runs; the authors caution that this does not establish that every early-pruning strategy is suboptimal.Source: Figure 10 · p. 16, §C.5 · p. 17

C.6 MOBILENET EXPERIMENTS

On MobileNet-v2, gradual pruning preserves accuracy better than one-shot methods at high sparsity, while FORCE and Iter SNIP improve on SNIP at moderate sparsity. The accompanying appendix material defines a mask-dependent smoothness condition and describes a multi-dataset, multi-network accuracy comparison.

C.6 MOBILENET EXPERIMENTS

  • On MobileNet-v2, gradual pruning preserves accuracy better than one-shot methods at high sparsity, while FORCE and Iter SNIP improve on SNIP at moderate sparsity.The figure averages three prune-train-test runs.Source: §C.6 · p. 17, Figure 11 · p. 17
  • At high sparsity, Iter SNIP surpasses FORCE on MobileNet-v2; the authors hypothesize that exploration may hurt this slim architecture.MobileNet-v2 has roughly 10× fewer parameters than ResNet or VGG.Source: §C.6 · p. 17, Figure 11 · p. 17

D LOCAL OPTIMAL MASKS

  • The appendix defines Λ-total CRS by requiring coordinate-restricted smoothness for every mask, with a mask-dependent bound Λ(c).The (p, ϵ)-local-optimal-mask definition compares a smaller mask with its preceding mask, but the supplied passage does not include its full condition.Source: D LOCAL OPTIMAL MASKS · pp. 17–18

E ITERATIVE PRUNING TO MAXIMIZE THE GRADIENT NORM

  • Figure 12 reports test accuracies across datasets and networks, averaging three prune-train-test runs and showing standard-deviation regions.The shaded regions may be too small to see.Source: Figure 12 · p. 19

E.1 MAXIMIZING THE GRADIENT NORM USING THE GRADIENT APPROXIMATION

The authors extend a gradient approximation to iterative pruning for maximizing gradient norm, while noting that the Taylor-based GRASP approximation assumes pruning is a small perturbation. Iterative GRASP beats GRASP at high sparsity, but its comparison with other methods varies by architecture.

E.1 MAXIMIZING THE GRADIENT NORM USING THE GRADIENT APPROXIMATION

  • The authors argue that GRASP's first-order Taylor approximation may fail at extreme sparsity because it assumes pruning is a small perturbation to the weights.They state their gradient approximation can also be applied to maximizing gradient norm after pruning.Source: §E.1 · p. 19
  • Iterative GRASP combines the gradient approximation with Eq. (7), but unlike FORCE, it does not recover GRASP when T = 1.The approximation assumes pruned connections have null gradients and unpruned gradients remain unchanged.Source: §E.1 · pp. 19–20
  • Iterative GRASP beats GRASP at high sparsity; on VGG19 it matches Iterative SNIP, but on ResNet50 it falls behind FORCE and Iterative SNIP as pruning increases.The authors hypothesize that gradient-only scoring, unlike FORCE's use of both gradient and weight magnitude, may explain the gap to Iterative SNIP.Source: §E.1 · p. 20

E.2 ITERATIVE GRASP (PRUNING CONSISTENCY)

The authors connect pruning consistency across sparsity levels with the behavior of iterative GRASP and SNIP. GRASP's layer allocation changes across sparsity levels, while the iterative procedure can heavily prune the initial and final layers; SNIP's preserved layers remain consistent in the reported analysis.

E.2 ITERATIVE GRASP (PRUNING CONSISTENCY)

  • Figure 13 tracks per-layer remaining weights and intermediate masks during iterative pruning at a global remaining-weight fraction of 0.01.The figure reports averages over three runs, with standard-deviation error bars.Source: Figure 13 · p. 20
  • Iterative use of GRASP's Taylor approximation yielded masks whose networks were unable to train in the authors' experiments.They then analyze GRASP's behavior relative to SNIP to investigate why the approximation cannot be used iteratively.Source: §E.2 · p. 20
  • GRASP's per-layer remaining-weight pattern is inconsistent across global sparsity levels, although both SNIP and GRASP show low variance across initializations.For GRASP, layers most preserved at high sparsity, such as the initial and last layers, are most heavily pruned at low sparsity.Source: §E.2 · p. 20
  • Iterative GRASP tends to prune nearly all initial- and final-layer weights, producing networks unable to converge, whereas iterative SNIP preserves consistent layers and reaches high accuracy.The authors frame this contrast in terms of pruning consistency across sparsity levels.Source: §E.2 · pp. 20–21
Loading 2006.09081v5…