Deconstructing Lottery Tickets: Zeros, Signs, and the Supermask

• Lottery-ticket subnetworks address a sparsity challenge: although pruned networks can work well, sparse networks trained directly often underperform their dense counterparts.Neural-network over-parameterization enables compression through pruning, matrix factorization, and other approaches.Source: §1 · p. 1
• The lottery-ticket procedure trains a dense network, selects weights by a mask criterion, resets retained weights to their initial values, and sets pruned weights to zero and freezes them.The process can be repeated for iterative pruning.Source: §1 · pp. 1–2
• Frankle & Carbin [5] reported that pruning 95%–99.5% of weights caused no performance drop, while pruning 50%–90% often outperformed the unpruned networks.They also found that the subnetworks trained well only when rewound to their original initial weights; reinitializing the same topology with new weights trained poorly.Source: §1 · p. 1
• The study varies mask criteria, mask-1 actions, and mask-0 actions to investigate why lottery-ticket networks work, and discovers Supermasks that yield above-chance performance in untrained networks.The authors describe Supermasks as masks applied to untrained networks.Source: §1 · p. 2

2 Mask criteria

• The nine criteria score weights using final values, initial values, both, or weight movement; movement is more likely than magnitude_increase to keep sign-changing weights.The large_final criterion from [5] keeps weights with large final magnitude.Source: §2 · p. 2, Figure 2 · p. 3
• large_final and magnitude_increase perform best; magnitude_increase is slightly more accurate on Conv2 and Conv4, while large_init_large_final and movement can match or exceed original-network performance.Figure 3 reports test accuracy at early stopping across FC, Conv2, Conv4, and Conv6.Source: Figure 3 · p. 3, §2 · p. 4
• Experiments follow [5] iterative pruning on a fully connected MNIST network and Conv2, Conv4, and Conv6 CIFAR-10 networks; generalization to harder problems remains open.Accuracy is measured at early stopping, with means over five runs and min–max bands where shown; reports that the original LT algorithm does not generalize to ResNet on ImageNet.Source: §2 · pp. 3–4
• Among the first six criteria, each opposing pair straddles the random baseline: when one performs better, its counterpart performs worse.The first six criteria form three opposing pairs.Source: Figure 2 · p. 3, §2 · p. 4

3 Mask-1 actions: the sign-ificance of initial weights

• Holding other treatments constant as in [5], the study compares distribution-based reinitialization, reshuffling, and ±α constants on original networks against rewind and random-pruning baselines.For constant reinitialization, α is the layer’s original initialization standard deviation; experiments use the large_final mask criterion with iterative pruning.Source: §3 · p. 4
• All three reinitialization variants train better when kept weights retain their original signs, and constant-valued weights perform well; optimizers struggle to cross zero between signs, contrary to [5].The authors suggest the basin of attraction is large within the correct sign quadrant.Source: §3 · p. 4

4 Mask-0 actions: masking is training

• The proposed mask-0 action matches original Lottery Ticket performance at low pruning rates and performs better at high rates; applying it to all weights performs even better.The rule zeros weights moving toward zero and otherwise freezes them at initialization; the second variant also applies it to one-masked weights.Source: §4 · pp. 5–6
• Networks perform significantly better when pruned weights are frozen at zero than at their random initial values, though initial-value freezing may do better at high pruning rates.The comparison uses the large_final mask criterion.Source: §4 · p. 5, Figure 5 · p. 6
• These results support the hypothesis that zeroing helps because selected weights were already moving toward zero during training.The large_final criterion biases the mask toward weights that had been moving toward zero.Source: §4 · pp. 5–6
• Viewing masking as training explains why large_final and magnitude_increase work well: they bias pruned weights toward their prior-round final values, making zero a better initialization for selected weights.The value assigned to pruned weights contributes to the performance of pruned networks.Source: §4 · p. 6

5 Supermasks

• A large_final mask raises an untrained network to nearly 40% MNIST accuracy, while a randomly initialized untrained network or randomly masked network performs at chance.The paper calls masks that produce better-than-chance accuracy without training the underlying weights “Supermasks”; their information reaches the initial network through a zero-one mask.Source: §5 · p. 6, Figure 6 · p. 7
• Large_final_same_sign masks yield 80% test accuracy on MNIST and 24% on CIFAR-10 without training; applying them to signed constants reaches up to 86% and 41%, respectively.The criterion selects large final-magnitude weights that kept the same sign during training; one-shot pruning is used for these heuristic Supermasks.Source: §5 · p. 7
• A Supermask can also enable network compression by storing a binary mask and a single random seed to reconstruct the full weights.The section presents these masks as findable through simple criteria.Source: §5 · p. 7

5.1 Optimizing the Supermask

• Training masks while freezing weights at their random initialization yields Supermasks, with effective weights defined as w′ = wi ⊙ g(m).The mask transformation samples binary values from Bern(S(m)); stochastic sampling helps training and mitigates the bias of masks initialized at the same value. Mask sparsity can range from 7% to 89%.Source: §5.1 · pp. 7–8

5.2 Dynamic Weight Rescaling

• Learned Supermasks reach up to 95.3% test accuracy on MNIST and 65.4% on CIFAR-10, with performance close to training the full network.Dynamic weight rescaling—multiplying each layer’s weights by its total weight count divided by the number of ones in the mask—significantly improves masked-network performance.Source: §5.1 · p. 7, §5.2 · p. 8

6 Conclusion

• The paper proposes that pruning works well when pruned weights are set close to their final values, and introduces alternative freezing schemes and mask criteria that meet or exceed current approaches.It studies the mask criterion, the treatment of kept weights during retraining (mask-1 action), and the treatment of pruned weights (mask-0 action).Source: §6 · pp. 8–9
• Only the signs, not the relative magnitudes, of the original initialization are crucial to Lottery Ticket network performance.Source: §6 · p. 9
• The masking procedure can be viewed as a training operation, revealing Supermasks that produce partially working networks without training.Source: §6 · p. 9

S1 Architectures and training hyperparameters

• Training used the same hyperparameters as [5], with no additional tuning.The architectures and relevant training hyperparameters were based on experiments in [5].Source: S1 Architectures and training… · p. 12
• Validation was randomly split from the training data: MNIST used 55,000 training and 5,000 validation examples, while CIFAR-10 used 45,000 and 5,000.Training/test splits were provided for both datasets.Source: S1 Architectures and training… · p. 12
• Networks were trained up to 24 times with iterative pruning, requiring more computation than regular training procedures.Source: S1 Architectures and training… · p. 12
• Each experiment used a single NVIDIA GeForce GTX 1080 Ti GPU.Source: S1 Architectures and training… · p. 12
• Experiments were parallelized by running multiple experiments on multiple GPUs.Source: S1 Architectures and training… · p. 12
• Table S1 describes the study’s architectures, including 3×3 convolutional layers with max pooling, followed by fully connected layers.The table caption says fully connected layer sizes are from [12], and initializations are Glorot Normal [7] with ReLu activations.Source: Table S1 · p. 12

S2 Further mask criteria details

• Figure S1 shows convergence speed and performance for all mask criteria on FC with MNIST and Conv2, Conv4, and Conv6 with CIFAR-10.The supplied passage identifies the models and datasets but does not report specific comparative outcomes.Source: S2 Further mask criteria details · p. 12

S3 Further mask-0 action details

• Both control experiments performed significantly worse than “set pruned to init or 0,” and the reversed treatment performed worse than both other controls.The controls randomly froze an equal number of pruned weights at zero or set weights that moved away from zero to zero; the other controls were random freezing and freezing all pruned weights at their initial values.Source: S3 Further mask-0 action details · pp. 12–13

S4 Further mask-1 action details

• Figure S4 reports convergence speed for various reinitialization methods on FC for MNIST and Conv2, Conv4, and Conv6 for CIFAR-10.Source: S4 Further mask-1 action details · p. 13
• Figure S4 reports performance for various reinitialization methods on FC for MNIST and Conv2, Conv4, and Conv6 for CIFAR-10.Source: S4 Further mask-1 action details · p. 13
• The S4 comparisons cover FC on MNIST and Conv2, Conv4, and Conv6 on CIFAR-10.Figure S4 presents the convergence speed and performance of various reinitialization methods across these settings.Source: S4 Further mask-1 action details · p. 13

S5 Further heuristic Supermask details

• Despite better test accuracy at iteration 0, large_final_same_sign slightly underperforms large_final when the network is fully trained.The passage suggests this may occur because the criterion randomly selects weights that changed signs when there are not enough same-sign weights.Source: S5 Further heuristic Supermask details · p. 13
• The large_final_same_sign criterion has better test accuracy than large_final at iteration 0.This advantage is not maintained when the underlying network is fully trained.Source: S5 Further heuristic Supermask details · p. 13
• When there are not enough same-sign weights, large_final_same_sign randomly selects weights that changed signs to keep.The passage suggests this may explain its slight underperformance after full training.Source: S5 Further heuristic Supermask details · p. 13

Training details

• Mask-only training freezes regular weights and biases, uses SGD with 0.9 momentum, and applies early stopping because Conv4 and Conv6 significantly overfit.Learning rates for FC, Conv2, Conv4, and Conv6 are 100, 100, 50, and 20, respectively; iteration counts are 2000, 2000, 1000, and 800. Evaluation averages accuracy over 10 independent Bernoulli samples.Source: S6 Further training details for… · p. 13

Mask initialization

• Mask values are initialized uniformly per layer to a shared constant from -5 to 5, but values starting too low can cause some networks to learn nothing.Different per-layer initializations or an L1 loss are proposed as future ways to control pruning rates or alleviate cold starts.Source: S6 Further training details for… · p. 13

Mask criteria

• large_final and magnitude_increase show the best convergence speed and accuracy, with magnitude_increase slightly more accurate on Conv2 and Conv4; criteria using small weights perform worse than random.The figure reports results for four networks at various pruning rates.Source: Figure S1 · p. 14

Mask motion

• large_final preferentially moves decreasing weights toward zero while resetting increasing weights to initialization, explaining its Supermasks and motivating large_final_same_sign.large_final selects large final weights rather than weight motion directly; decreased weights are more likely to have small final magnitudes.Source: Figure S3 · p. 15

1-actions

• “Reshuffle, init sign” and “constant, init sign” perform similarly to the rewind baseline, with no significant performance difference between rewind (large_final) and constant, init sign.The comparison includes four networks and various pruning rates.Source: Figure S4 · p. 16

Supermask performance

• With no network training, large_final_same_sign produces the highest-performing heuristic Supermask by a wide margin across the evaluated network structures and tasks.The comparison applies masks to the same underlying network with weights frozen at initialization or a signed constant; the figure also compares additional mask criteria.Source: Figure S5 · p. 17, Figure S6 · p. 18