PaperlayerView exact revision on arXiv

Source-linked AI summary

The Lottery Ticket Hypothesis: Finding Sparse, Trainable Neural Networks

Jonathan Frankle, Michael Carbin

arXiv:1803.03635v5cs.LGcs.AIcs.NE

TL;DR

Pruned neural networks are often difficult to train from scratch, raising whether smaller trainable subnetworks exist. The paper identifies such subnetworks by pruning and restoring initial weights, finding winning tickets that match or exceed dense-network accuracy while learning at least as quickly.

  • Problem

    Pruned architectures are often harder to train from scratch, motivating the search for smaller subnetworks that retain accuracy and trainability.

  • Method

    The paper identifies winning tickets by pruning smallest-magnitude weights, then resetting surviving connections to their original initialization before retraining.

  • Results

    Winning tickets consistently train at least as fast as larger networks, match their test accuracy, and can exceed their test accuracy while matching training accuracy.

  • Takeaways & Limitations

    The findings support the lottery ticket hypothesis that dense random networks contain sparse subnetworks whose initial weights make effective isolated training possible.

  • Takeaways & Limitations

    The study examines only vision-centric classification on smaller MNIST and CIFAR10 datasets, not larger datasets such as ImageNet.

Abstract

from arXiv · show

Neural network pruning techniques can reduce the parameter counts of trained networks by over 90%, decreasing storage requirements and improving computational performance of inference without compromising accuracy. However, contemporary experience is that the sparse architectures produced by pruning are difficult to train from the start, which would similarly improve training performance. We find that a standard pruning technique naturally uncovers subnetworks whose initializations made them capable of training effectively. Based on these results, we articulate the "lottery ticket hypothesis:" dense, randomly-initialized, feed-forward networks contain subnetworks ("winning tickets") that - when trained in isolation - reach test accuracy comparable to the original network in a similar number of iterations. The winning tickets we find have won the initialization lottery: their connections have initial weights that make training particularly effective. We present an algorithm to identify winning tickets and a series of experiments that support the lottery ticket hypothesis and the importance of these fortuitous initializations. We consistently find winning tickets that are less than 10-20% of the size of several fully-connected and convolutional feed-forward architectures for MNIST and CIFAR10. Above this size, the winning tickets that we find learn faster than the original network and reach higher test accuracy.

1 INTRODUCTION

The introduction argues that dense, randomly initialized networks contain sparse subnetworks whose original initializations enable training to match or exceed the dense network’s accuracy and speed. It presents pruning-and-resetting as a way to identify these “winning tickets” and motivates the lottery ticket hypothesis and conjecture.

  • Evidence for the hypothesis: Pruning consistently uncovers smaller subnetworks that train at least as fast as larger counterparts while reaching similar test accuracy.The finding is reported across fully connected MNIST and convolutional CIFAR10 architectures.
  • Lottery ticket hypothesis: Winning tickets are subnetworks that, when trained from their original initialization, match the original network’s test accuracy in at most the same number of iterations.The lottery ticket hypothesis defines these subnetworks as appropriately initialized masks of dense networks.
  • Initialization: Winning tickets lose their performance advantage when their surviving parameters are randomly reinitialized, indicating that their success depends on initialization as well as structure.The paper therefore characterizes them as combinations of weights and connections that have won the initialization lottery.
  • Identifying winning tickets: The identification algorithm trains a network, prunes its smallest-magnitude weights, and resets surviving connections to their original initialization.Iterative pruning repeatedly trains, prunes, and resets, and finds matching-accuracy tickets at smaller sizes than one-shot pruning.
  • Lottery ticket conjecture: The lottery ticket conjecture proposes that SGD seeks well-initialized subsets, while dense networks are easier to train because they contain more possible subnetworks.The introduction also reports winning tickets that learn faster, achieve higher test accuracy, and generalize better than their original networks.

2 WINNING TICKETS IN FULLY-CONNECTED NETWORKS

Experiments on fully connected Lenet-300-100 networks trained on MNIST show that pruning can uncover winning tickets that learn faster and achieve higher test accuracy than the original network. These advantages depend on preserving the tickets’ original initialization, while one-shot pruning can also identify effective tickets.

  • Iterative pruning: The study applies iterative magnitude pruning to Lenet-300-100 networks trained on MNIST, resetting surviving connections to their original initializations.Weights with the lowest magnitudes are removed layer by layer.
  • Iterative pruning: 38% earlier early stopping occurs at Pm = 21%, where winning tickets learn faster than the original network.Learning speeds up as Pm decreases from 100% to 21%, then slows with further pruning.
  • Iterative pruning: More than 0.3 percentage points of test accuracy improvement occurs at Pm = 13.5%, before accuracy declines with further pruning.Accuracy returns to the original network’s level at Pm = 3.6%.
  • Iterative pruning: Up to 0.35 percentage points of test accuracy improvement remains at iteration 50,000 despite training accuracy reaching 100% for nearly all networks.This result indicates improved generalization rather than only faster optimization.
  • Random reinitialization: At Pm = 21%, winning tickets reach minimum validation loss 2.51x faster than randomly reinitialized tickets, whose accuracy declines after little pruning.The original initialization withstands and benefits from pruning, whereas random reinitialization progressively harms performance.
  • One-shot pruning: One-shot pruning identifies winning tickets: when 67.5% ≥Pm ≥17.6%, they reach minimum validation accuracy earlier than the original network.Unlike iterative pruning, one-shot pruning avoids repeated training during ticket discovery.

3 WINNING TICKETS IN CONVOLUTIONAL NETWORKS

On CIFAR10 convolutional networks, iterative pruning uncovers winning tickets that train faster and achieve higher test accuracy than the original networks. These winning tickets persist under dropout, whose gains combine complementarily with iterative pruning.

  • Experimental setup: Conv-2, Conv-4, and Conv-6 are scaled-down VGG-family networks with two, four, or six convolutional layers followed by two fully-connected layers.Max-pooling occurs after every two convolutional layers.
  • Finding winning tickets: 3.5x faster is the best improvement in reaching minimum validation loss for Conv-2 at Pm = 8.8% and Conv-4 at Pm = 9.2%; Conv-6 improves 2.5x at Pm = 15.1%.Test accuracy improves by up to 3.4 percentage points for Conv-2, 3.5 for Conv-4, and 3.3 for Conv-6.
  • Finding winning tickets: At 20,000, 25,000, and 30,000 iterations, respectively, all networks reach 100% training accuracy when Pm ≥2%, while winning tickets retain higher test accuracy.These iteration counts correspond to the original networks’ final training iterations.
  • Random reinitialization: Randomly reinitialized networks take increasingly longer to learn as pruning continues, and their test accuracy drops more quickly than that of winning tickets.For Conv-2 and Conv-4, early-stopping test accuracy initially remains steady and even improves at moderate pruning levels.
  • Dropout: With dropout rate 0.5, winning tickets remain discoverable; dropout raises initial test accuracy by 2.1, 3.0, and 2.4 percentage points for Conv-2, Conv-4, and Conv-6, respectively.Iterative pruning adds up to 2.3, 4.6, and 4.7 percentage points, respectively, suggesting complementary interaction with dropout.

4 VGG AND RESNET FOR CIFAR10

Across VGG-19 and Resnet-18 on CIFAR10, the lottery ticket hypothesis continues to hold under practical training techniques, while winning-ticket discovery depends on pruning and learning-rate choices. VGG-19 benefits from warmup at a higher learning rate, and global pruning yields smaller winning tickets in the deeper networks.

  • Experimental setting: VGG-19 and Resnet-18 experiments use batch normalization, weight decay, decreasing learning-rate schedules, and augmented training data.These architectures are intended to evoke networks and techniques used in practice.
  • Pruning strategy: Global pruning identifies smaller winning tickets for Resnet-18 and VGG-19 than pruning each layer separately.The deeper networks are pruned by collectively removing the lowest-magnitude weights across convolutional layers.
  • VGG-19: At learning rate 0.1, iterative pruning does not find winning tickets for VGG-19, whereas at learning rate 0.01 it does.At 0.1, pruned-network performance is no better than random reinitialization.
  • VGG-19: Warmup over k = 10000 iterations improves VGG-19's unpruned test accuracy by about one percentage point at learning rate 0.1 and enables winning tickets with Pm ≥1.5%.The learning rate increases linearly from 0 to 0.1 during warmup.
  • Resnet-18: Resnet-18 is a 20-layer CIFAR10 convolutional network with residual connections and 271,000 parameters, trained for 30,000 iterations.Training uses SGD with momentum 0.9 and learning-rate drops by a factor of 10 at 20,000 and 25,000 iterations.

5 DISCUSSION

The discussion emphasizes that winning-ticket success depends on both fortuitous initialization and sparse structure, while these subnetworks can generalize better and achieve original-network accuracy with fewer parameters. It also situates the findings relative to random reinitialization and questions their implications for optimization.

  • The importance of winning ticket initialization: Randomly reinitializing a winning ticket makes it learn more slowly and achieve lower test accuracy, highlighting the importance of its original initialization.The passage notes that the initialization may place the ticket in a loss-landscape region especially amenable to the chosen optimization algorithm.
  • The importance of winning ticket initialization: Winning-ticket weights move further during training than other weights, arguing against the explanation that they begin close to their final trained values.This suggests the initialization benefit is connected to the optimization algorithm, dataset, and model.
  • The importance of winning ticket structure: Winning-ticket structure may encode an inductive bias customized to the learning task because the sparse architectures are discovered using extensive training data.The passage connects this hypothesis to work showing that deep-network structure determines which data can be separated.
  • The improved generalization of winning tickets: Winning tickets reliably exceed the original network’s test accuracy while matching its training accuracy, with accuracy first increasing and then decreasing as pruning intensifies.The discussion characterizes this pattern as an Occam’s Hill: the original model may have excessive complexity, whereas extreme pruning may leave too little.
  • Implications for neural network optimization: Winning tickets can match the accuracy of the original unpruned network with significantly fewer parameters, connecting the finding to questions about overparameterization and neural-network optimization.The passage cites results that sufficiently overparameterized two-layer relu networks trained with SGD converge to global optima.

6 LIMITATIONS AND FUTURE WORK

The study is limited to small vision-classification datasets and sparse pruning, while deeper networks require learning-rate warmup to reveal winning tickets. Future work targets more efficient and hardware-aware discovery methods and investigates the initialization and optimization factors underlying winning tickets.

  • Dataset and computational scope: The study considers only vision-centric classification on MNIST and CIFAR10, not larger datasets such as Imagenet, because iterative pruning requires training networks 15 or more times consecutively for multiple trials.Future work will explore more efficient methods for finding winning tickets in resource-intensive settings.
  • Pruning methods: Sparse pruning reduces parameter counts, but its resulting architectures are not optimized for modern libraries or hardware.Future work will study structured pruning for hardware-optimized networks and non-magnitude pruning to find smaller tickets or identify them earlier.
  • Initialization properties: Winning tickets match unpruned-network performance at sizes too small for randomly initialized networks to do the same.Future work will examine how their initializations, together with pruned-network inductive biases, make them particularly adept at learning.
  • Deeper networks: On Resnet-18 and VGG-19, iterative pruning finds winning tickets only when networks use learning rate warmup.Future work will investigate why warmup is necessary and whether improvements to the identification scheme can eliminate these hyperparameter modifications.

7 RELATED WORK · B ITERATIVE PRUNING STRATEGIES

The paper situates its approach among parameter-reduction methods before, during, and after training, then compares two iterative pruning strategies that differ in whether weights are reset or retained between pruning rounds. Resetting after each pruning round yields higher validation accuracy and faster early stopping at smaller network sizes.

  • 7 RELATED WORK: The broader motivation is that overparameterized networks can be reduced while preserving accuracy, despite having capacity to memorize training data.Distillation and pruning rely on parameter reduction, while related work argues that networks naturally learn simpler functions.
  • 7 RELATED WORK: Before training, prior work reduces architectures through engineered compact networks, low-rank factorization, or optimization in randomly sampled parameter subspaces.Squeezenet and MobileNets are an order of magnitude smaller than standard architectures, while other methods factorize weights or restrict optimization while retaining update access to all parameters.
  • 7 RELATED WORK: After training, distillation trains small networks to mimic large models, while pruning compresses models for resource-constrained deployment.The paper instead studies why training needs the overparameterized networks that make pruning possible.
  • B ITERATIVE PRUNING STRATEGIES: Strategy 1 iteratively initializes a masked network, prunes s% of parameters, resets surviving weights to θ0, and repeats until sufficiently pruned.The mask begins as m = 1|θ|, and each pruning round updates it to retain (Pm − s)% of parameters.
  • B ITERATIVE PRUNING STRATEGIES: Strategy 2 iteratively initializes and trains for j iterations, prunes s% of parameters, continues training with the pruned weights, and ultimately resets weights to θ0.Its distinguishing feature is retaining already-trained weights between pruning rounds before the final reset to the original initialization.
  • B ITERATIVE PRUNING STRATEGIES: The strategies differ after each pruning round: Strategy 2 retrains from trained weights, whereas Strategy 1 resets to initial weights; both finally reset surviving weights to the original initialization.This distinction defines the comparison of iterative pruning with resetting versus continued training.
  • B ITERATIVE PRUNING STRATEGIES: In all comparisons on Lenet and Conv-2/4/6, Strategy 1 maintains higher validation accuracy and faster early-stopping times at smaller network sizes.Figures 9 and 10 compare the strategies using the selected hyperparameters from Appendices G and H.

C EARLY STOPPING CRITERION

The paper measures learning speed by the iteration of minimum validation loss, when validation loss bottoms out before overfitting increases it. Winning tickets reach this early-stopping point sooner and achieve higher test accuracy there.

  • Early-stopping criterion: Learning speed is measured by the iteration at which the minimum-validation-loss early-stopping criterion ends training.This criterion serves as a proxy for how quickly networks learn.
  • Early-stopping criterion: Validation and test loss initially decrease, reach a minimum, and then increase as the model overfits the training data.The criterion identifies the bottom of this loss curve.
  • Early-stopping criterion: Networks that reach early stopping sooner are considered to have learned faster, matching the ordering of experiments reaching a particular test-accuracy threshold.This agreement supports using early-stopping iteration as a learning-speed measure.
  • Winning-ticket results: Winning tickets arrive at early stopping sooner and reach higher test accuracy at the minimum-validation-loss iteration.The paper reports both early-stopping speed and test accuracy at that point to contextualize learning speed.

D TRAINING ACCURACY FOR LOTTERY TICKET EXPERIMENTS … F.2 WINNING TICKET INITIALIZATIONS (SGD)

The appendices report training and test accuracy views for lottery-ticket experiments, compare original-initialization tickets with reinitialized and random sparse networks, and examine winning-ticket initializations under Adam and SGD. Across these analyses, iterative pruning preserves initialization patterns associated with surviving connections, especially in later layers.

  • D TRAINING ACCURACY FOR LOTTERY TICKET EXPERIMENTS: The appendix augments Figures 4 and 5 with early-stopping iteration, test accuracy, training accuracy, and final-training test accuracy.These figures cover LeNet on MNIST and Conv-2, Conv-4, and Conv-6.
  • E COMPARING RANDOM REINITIALIZATION AND RANDOM SPARSITY: The comparison evaluates original-initialization tickets, randomly reinitialized tickets, and random sparse subnetworks matched for parameter count.The three network types correspond to blue, orange, and green in Figure 14.
  • E COMPARING RANDOM REINITIALIZATION AND RANDOM SPARSITY: For fully connected LeNet on MNIST, randomly reinitialized networks outperform random sparsity, whereas convolutional networks show no significant difference.Figure 14 reports this comparison across the paper’s major experiments.
  • F EXAMINING WINNING TICKETS: The winning-ticket structure analysis studies LeNet trained on MNIST using glorot initialization and Adam optimization unless otherwise stated.The appendix examines ticket structure to understand why heavily pruned tickets learn effectively.
  • F.1 WINNING TICKET INITIALIZATION (ADAM): As pruning increases, the second hidden and output layers become increasingly bimodal, while the first hidden layer maintains its initialization distribution.The bimodal peaks lie on opposite sides of 0, and the second hidden layer retains more positive than negative initializations.
  • F.1 WINNING TICKET INITIALIZATION (ADAM): Surviving second-hidden-layer and output-layer connections tend to have higher-magnitude initializations, whereas input-layer connections preserve their distribution.The result follows pruning connections with the lowest magnitudes after training.
  • F.2 WINNING TICKET INITIALIZATIONS (SGD): With SGD learning rate 0.8, winning-ticket initializations show bimodal distributions across all layers, and high-magnitude initializations are more likely to survive pruning.The peaks occur on opposite sides of 0, paralleling the Adam-optimized tickets.

F.3 REINITIALIZING FROM WINNING TICKET INITIALIZATIONS … G HYPERPARAMETER EXPLORATION FOR FULLY-CONNECTED NETWORKS

Across Sections F.3–F.7, the paper tests how winning-ticket initializations, pruning timing, weight changes, connectivity, and noise affect subnetworks. Section G motivates broader hyperparameter exploration to explain the selected settings and test whether lottery-ticket patterns generalize.

  • F.3 REINITIALIZING FROM WINNING TICKET INITIALIZATIONS: Winning tickets reinitialized from their winning-ticket initialization distribution perform little better than those randomly reinitialized from the original distribution, with similar results for SGD-trained tickets.The experiment samples parameters per layer from the winning-ticket initialization distribution D_m and compares them with reinitialization from D.
  • F.4 PRUNING AT ITERATION 0: Pruning at iteration 0 removes low-magnitude weights before training, and SGD-trained winning tickets show similar results to the reported Adam experiment.The passage interprets initially small weights as candidates for immediate pruning and reports a corresponding pretraining magnitude-pruning experiment.
  • F.5 COMPARING INITIAL AND FINAL WEIGHTS IN WINNING TICKETS: Winning-ticket weights are more likely than other weights to increase in magnitude, distinguishing their optimization trajectories in the unpruned network.The comparison examines the magnitude difference between initial and final weights and notes that this distinction may help identify winning tickets earlier.
  • F.6 WINNING TICKET CONNECTIVITY: Incoming connectivity remains approximately proportional to layer-wide pruning for both Adam and SGD, so units retain relatively even sparsity.The study contrasts incoming and outgoing connections when examining how winning tickets distribute surviving edges across units.
  • F.6 WINNING TICKET CONNECTIVITY: Adam-trained networks retain far more outgoing connections for certain units than others, suggesting that some features are more useful than others.Outgoing-connection distributions are less smooth than incoming-connection distributions, especially in the fully connected MNIST setting.
  • F.7 ADDING NOISE TO WINNING TICKETS: Gaussian noise perturbations provide a less extreme test of winning-ticket robustness than random reinitialization, which substantially slows learning and reduces eventual test accuracy.The experiment adds noise whose layerwise standard deviations are multiples 0.5, 1, 2, and 3 of the initialization standard deviations.
  • G HYPERPARAMETER EXPLORATION FOR FULLY-CONNECTED NETWORKS: Hyperparameter exploration explains the settings selected in the main paper and evaluates whether lottery-ticket experiment patterns extend to other hyperparameter choices.The exploration concerns the Lenet architecture used in Section 2.

G.1 EXPERIMENTAL METHODOLOGY … H.1 EXPERIMENTAL METHODOLOGY

The appendices evaluate iterative lottery-ticket pruning across fully connected MNIST and convolutional CIFAR10 networks while varying optimization, pruning rates, initialization, and network size. Across these settings, the lottery-ticket pattern is robust, but accuracy, learning speed, and sparsity depend on the experimental choices.

  • G.1 EXPERIMENTAL METHODOLOGY: The MNIST experiments use Lenet with hidden layers of 300 and 100 units, training on 55,000 examples after reserving 5,000 for validation.MNIST has 60,000 training and 10,000 test examples.
  • G.2 LEARNING RATE: Learning rates between 0.0002 and 0.002 achieve similar validation accuracy on the original Adam-trained network and preserve similar performance during pruning.Learning-rate selection also required good early-stopping behavior on the unpruned network and matching that performance with few parameters.
  • G.3 OTHER OPTIMIZATION ALGORITHMS; G.3.1 SGD: SGD learning rates 0.4 and 0.8 reach early-stopping in a similar number of iterations as the best Adam learning rates, 0.0012 and 0.002.The lottery-ticket pattern appears across all tested SGD learning rates, including those failing early stopping within 50,000 iterations.
  • G.3.2 MOMENTUM: With momentum 0.9, learning rates between 0.025 and 0.1 maintain high validation accuracy and faster learning for the longest number of pruning iterations.Learning rate 0.025 achieves the highest validation accuracy on the unpruned network.
  • G.4 ITERATIVE PRUNING RATE: Pruning rates 0.1 and 0.2 outperform rates 0.4 and above in learning speed and validation accuracy, retaining both advantages to smaller network sizes.The iterative experiment prunes each layer independently and resets weights to their original initializations after each pruning step.
  • G.5 INITIALIZATION DISTRIBUTION: A Gaussian initialization with standard deviation 0.1 maintains high validation accuracy and low early-stopping times for the longest, approximately matching Gaussian Glorot initialization.The lottery-ticket pattern continues across all tested standard deviations.
  • G.6 NETWORK SIZE: At a fixed number of remaining weights, initially larger networks produce more accurate winning tickets, learn only marginally faster, and return to original accuracy at 9,000–15,000 weights.Early-stopping iterations generally vary little across initial network sizes after pruning to the same number of weights.
  • H HYPERPARAMETER EXPLORATION FOR CONVOLUTIONAL NETWORKS; H.1 EXPERIMENTAL METHODOLOGY: The CIFAR10 Conv-2, Conv-4, and Conv-6 networks use VGG-style convolutional modules, Gaussian Glorot initialization, Adam, and separately selected pruning rates for convolutional and fully connected layers.CIFAR10 uses 45,000 training examples after reserving 5,000 for validation; validation and test performance are measured retroactively.

H.2 LEARNING RATE … H.4 ITERATIVE PRUNING RATE

Across optimization and pruning-rate experiments, winning-ticket behavior generally persisted, although extreme learning rates and SGD produced important deviations. The selected hyperparameters balanced validation accuracy, early-stopping time, and final sparsity across Conv-2, Conv-4, and Conv-6.

  • H.2 LEARNING RATE: 0.0002 was selected for Conv-2, improving validation accuracy by 3.3 percentage points when pruning reached 3% of the original size.It combined the highest initial validation accuracy with fast early-stopping times and sustained accuracy during pruning.
  • H.2 LEARNING RATE: 0.0003 was selected for Conv-4 and Conv-6, balancing accuracy and early-stopping improvements under substantial pruning.For Conv-4, accuracy improved 3.7 percentage points to 78.6% with 5.4% of weights remaining, while early-stopping improved 4.27x with 11.1% remaining; for Conv-6, accuracy improved 2.4 points to 81.5% with 9.31% remaining, and early-stopping improved 2.61x when pruned to 11.9%.
  • H.2 LEARNING RATE: Across nearly all learning-rate combinations, early-stopping times decreased or remained controlled while validation accuracy was maintained or increased during pruning.At the very highest learning rates, this lottery-ticket pattern failed or held only briefly, especially for Conv-6.
  • H.3.1 SGD: SGD with Glorot initialization made Conv-2 and Conv-4 challenging to train, but winning tickets appeared at all depicted learning rates.For every learning rate, early-stopping times first decreased and later increased; Conv-6 validation accuracy first increased and later decreased with pruning.
  • H.3.1 SGD: Conv-2 and Conv-4 accuracy initially declined with SGD pruning, then rose to match or surpass the unpruned networks while reaching early-stopping in about the same or fewer iterations.These pruned networks therefore constituted winning tickets despite differing from the usual accuracy pattern.
  • H.3.2 MOMENTUM: With SGD momentum 0.9, the lottery-ticket pattern generally continued, but very low learning rates caused initial accuracy declines and very high rates prevented early-stopping improvements.At low rates, accuracy later exceeded the unpruned network; at high rates such as 0.005 and 0.008 for Conv-2 and Conv-4, early-stopping remained stable before increasing.
  • H.4 ITERATIVE PRUNING RATE: 10% convolutional pruning per iteration was selected for Conv-2 and Conv-4, and 15% for Conv-6, because rates between 10% and 20% generally preserved the lottery-ticket pattern.The selection criterion sought the lowest parameter counts while maintaining validation accuracy at or above the original and early-stopping times at or below the original.

H.5 LEARNING RATES (DROPOUT) … I.4 LEARNING RATE

Dropout and pruning choices strongly affect lottery-ticket outcomes: suitable learning rates and convolutional pruning improve results, while deeper VGG-19 and Resnet-18 networks require lower learning rates and global pruning. The appendix also specifies their architectures, training procedures, initialization, and pruning schedules.

  • H.5 LEARNING RATES (DROPOUT): 82.6% average validation accuracy is reached by dropout-trained Conv-4 winning tickets pruned to 7.6%, with early-stopping times improving by up to 1.58x.The unpruned Conv-4 network reaches 77.6% average validation accuracy.
  • H.5 LEARNING RATES (DROPOUT): 84.8% average validation accuracy is reached by dropout-trained Conv-6 winning tickets pruned to 10.5%, while learning-rate effects favor 0.0002 over 0.0003.The unpruned Conv-6 architecture reaches 81.3% average validation accuracy, compared with 81.5% for the strongest cited nondropout pruned result.
  • H.5 LEARNING RATES (DROPOUT): Dropout-trained networks generally preserve the lottery-ticket accuracy pattern, but Conv-2 is difficult to train consistently and does not clearly improve early-stopping times.Dropout requires three times as many iterations as training without dropout because networks take longer to learn.
  • H.6 PRUNING CONVOLUTIONS VS. PRUNING FULLY-CONNECTED LAYERS: Pruning convolutions alone produces higher test accuracy and faster learning, whereas pruning fully-connected layers alone generally worsens accuracy and slows learning.The comparison considers pruning convolutions, fully-connected layers, or both while measuring parameters remaining.
  • I.1 GLOBAL PRUNING: VGG-19 and Resnet-18 experiments use global pruning, which finds smaller winning tickets than layer-wise pruning under the cited VGG-19 hyperparameters.For VGG-19 with learning rate 0.1 and warmup to iteration 10,000, winning tickets occur at Pm ≥1.5% globally versus Pm ≥6.9% layer-wise.
  • I.1 GLOBAL PRUNING: At learning rate 0.1, neither VGG-19 nor Resnet-18 yields winning tickets; at 0.01, the lottery-ticket pattern reemerges, including with layer-wise pruning.Warmup to a higher learning rate also enables winning tickets in the cited experiments.
  • I.3 RESNET-18 DETAILS: Resnet-18 has 20 total layers and uses batch normalization, weight decay 0.0001, SGD with momentum (0.9), and global 20% convolutional pruning.The 2560 residual-downsampling parameters and 640 fully-connected output parameters are not pruned.

I.5 WARMUP ITERATION

Linear learning-rate warmup enables winning-ticket discovery at higher learning rates, with longer warmup generally improving eventual accuracy before diminishing returns. The experiments select k = 20000 for Resnet-18 and k = 10000 for VGG-19.

  • Warmup iteration: Linear warmup enables finding winning tickets for VGG-19 and Resnet-18 at higher learning rates, which can yield higher accuracy.Warmup allows VGG-19 to use its original learning rate and Resnet-18 to increase its learning rate from 0.01 to 0.03.
  • Resnet-18: For Resnet-18, accuracy improves rapidly below k = 5000, then shows diminishing returns above k = 5000.The experiments select k = 20000, which achieves the highest validation accuracy.
  • VGG-19: For VGG-19, accuracy improves rapidly below k = 5000, then shows diminishing returns above k = 5000.The experiments select k = 10000 because larger values provide little benefit.
Loading 1803.03635v5…