Source-linked AI summary
Deconstructing Lottery Tickets: Zeros, Signs, and the Supermask
Hattie Zhou, Janice Lan, Rosanne Liu, Jason Yosinski
TL;DR
Lottery Ticket networks can outperform dense models, but the mechanisms behind their dependence on masks and initialization were unclear. This paper ablates mask criteria and the treatment of kept and pruned weights, finding that signs and near-final values matter, and that masks can themselves yield partially working untrained networks. Supermasks reach up to 86% on MNIST and 41% on CIFAR-10 without training.
Problem
The paper investigates why Lottery Ticket networks perform well, why masks and initial weights are coupled, and why selecting large weights is effective.
Method
The paper ablates mask criteria, kept-weight retraining actions, and pruned-weight retraining actions, then evaluates Supermasks on frozen random networks.
Results
Supermasks using signed constants reach up to 86% accuracy on MNIST and 41% on CIFAR-10 without training.
Takeaways & Limitations
The paper concludes that initial signs are crucial rather than relative magnitudes, masking can act like training, and Supermasks can produce partially working networks without training.
Takeaways & Limitations
Supermask pruning experiments use one-shot rather than iterative pruning, and learned mask initialization was constant across layers.
Abstract
from arXiv · showhide
The recent "Lottery Ticket Hypothesis" paper by Frankle & Carbin showed that a simple approach to creating sparse networks (keeping the large weights) results in models that are trainable from scratch, but only when starting from the same initial weights. The performance of these networks often exceeds the performance of the non-sparse base model, but for reasons that were not well understood. In this paper we study the three critical components of the Lottery Ticket (LT) algorithm, showing that each may be varied significantly without impacting the overall results. Ablating these factors leads to new insights for why LT networks perform as well as they do. We show why setting weights to zero is important, how signs are all you need to make the reinitialized network train, and why masking behaves like training. Finally, we discover the existence of Supermasks, masks that can be applied to an untrained, randomly initialized network to produce a model with performance far better than chance (86% on MNIST, 41% on CIFAR-10).
1 Introduction
Lottery Ticket networks showed that aggressively pruned subnetworks can train from scratch and sometimes outperform dense models, but only under tightly coupled initialization conditions. This paper decomposes the algorithm into mask selection and weight-treatment steps to explain those results and introduces Supermasks.
- Sparse networks created by pruning often work well, but training sparse networks directly can underperform dense counterparts.
- 95 percent to 99.5 percent pruning caused no performance drop, while 50 percent to 90 percent pruning often outperformed unpruned networks.
- Lottery Ticket subnetworks trained well only when rewound to the specific initial weights used before pruning.
- The paper studies alternate mask criteria, mask-1 actions, and mask-0 actions through ablation studies.The algorithm trains weights, ranks them by a mask criterion, resets kept weights, freezes pruned weights, and may repeat iteratively.
- These ablations reveal insights into Lottery Ticket behavior and uncover Supermasks that produce above-chance performance on untrained networks.
2 Mask criteria
The paper evaluates mask criteria based on final weights, initial weights, their combination, and weight movement. Several criteria match or exceed the original network, with magnitude_increase performing as well as or better than large_final.
- Mask criteria map each weight’s initial and final values to a score that determines which weights are kept or pruned.The original large_final criterion uses M(wi, wf) = |wf|.
- The experiments compare criteria based on final weights, initial weights, combined values, movement, and random selection.The study uses an FC network on MNIST and Conv2, Conv4, and Conv6 networks on CIFAR-10.
- The results measure early-stopping test accuracy across four networks and multiple pruning rates, averaging five runs.
- Magnitude_increase performs as well as large_final and is significantly better in some cases.
- Large_final remains competitive, but magnitude_increase, large_init_large_final, and movement can also match or exceed the original network.
3 Mask-1 actions: the sign-ificance of initial weights
The paper tests whether Lottery Ticket rewinding requires the original weight magnitudes or only other properties of the initialization. Across reinitialization variants, preserving initial signs is the common factor associated with good training.
- The experiments ask why random reinitialization harms Lottery Ticket training and which components of the original initialization matter.
- Reinit, reshuffle, and constant variants alter the values of kept, trainable weights before subnetwork training.Constant variants assign kept weights positive or negative constants, producing layer values of −α, 0, or α.
- None of the three variants alone trains as well as the original Lottery Ticket network.
- All three variants improve when new kept weights retain the signs of their original initial values.
- Keeping signs allows even constant-valued kept weights to train well, suggesting that relative initial magnitudes are not crucial.The paper describes the basin of attraction as large within the correct sign quadrants, with difficulty crossing the zero barrier between signs.
4 Mask-0 actions: masking is training
Setting pruned weights to zero improves lottery-ticket performance because the selected weights tend to be moving toward zero. This supports viewing masking as a training operation rather than merely removing connections.
- Experimental separation: The experiment independently varies setting pruned weights to zero and freezing them, clarifying which component contributes to lottery-ticket performance.It reproduces iterative train/mask/rewind cycles while changing the treatment of pruned weights.
- Zero values matter: Freezing pruned weights at zero performs significantly better than freezing them at random initial values, although initial values can be better at high pruning levels.The high-pruning difference is consistent with many zeros creating dead connections.
- Mask-0 action: The proposed treatment sets a pruned weight to zero when it moves toward zero and otherwise freezes it at initialization.This separates the value assigned to the pruned weight from the decision to freeze it.
- Mask-0 action: The directional treatment matches original lottery-ticket performance at low pruning rates and exceeds it at high pruning rates.The result is reported for a variant applied directly to zero-masked weights.
- Masking is training: These results support the hypothesis that zero helps because the selected weights were already moving toward zero during training.The large_final criterion biases such weights toward zero in the mask, pushing them further in their prior direction.
5 Supermasks
The paper shows that carefully chosen binary masks can make randomly initialized, untrained networks perform far above chance, and that learned masks can approach trained-network performance. The strongest heuristic uses large final magnitudes whose signs are preserved, while mask optimization further improves results.
- Heuristic Supermasks: Nearly 40% MNIST accuracy is achieved by an untrained randomly initialized network using a large_final mask, versus about 10% chance.The mask transmits training-derived information only through zeros and ones.
- Heuristic Supermasks: Supermasks are masks that produce better-than-chance accuracy without training the underlying weights.Their masks are derived from a training process, making above-chance performance plausible despite unchanged initial weights.
- Heuristic Supermasks: The large_final_same_sign criterion selects large final magnitudes whose weights retain their initial signs, while large_final_diff_sign serves as a control.Supermask criteria are evaluated against untrained and fully trained baselines using one-shot pruning.
- Heuristic Supermasks: 80% MNIST and 24% CIFAR-10 test accuracy are obtained without training using large_final_same_sign on the original initial weights.Replacing initial weights with signed constants raises performance to up to 86% on MNIST and 41% on CIFAR-10.
- Implications: Supermasks also offer a compression method because a binary mask and one random seed suffice to reconstruct the full network weights.
- Optimizing the Supermask: Training binary masks with SGD reaches up to 95.3% test accuracy on MNIST and 65.4% on CIFAR-10 while freezing the randomly initialized weights.The effective weights are the fixed initial weights multiplied elementwise by a binary mask sampled from sigmoid mask parameters.
- Dynamic Weight Rescaling: Learned Supermasks show striking improvement over heuristic masks and achieve performance close to training the full network.Dynamic weight rescaling significantly improves masked-network performance.
6 Conclusion
The conclusion connects LT performance to three separable pruning components and argues that signs, final-value proximity, and masking-as-training explain why sparse subnetworks can work. It also identifies Supermasks as partially functioning networks obtainable without weight training.
- 6 Conclusion: The study varies the mask criterion and the treatments of kept and pruned weights to analyze how LT-style pruning produces sparse, performant subnetworks.
- 6 Conclusion: Alternative freezing schemes and mask criteria meet or exceed current approaches when they keep pruned weights close to their final values.
- 6 Conclusion: The sign of the initial weights, rather than their relative magnitude, is the crucial initialization element for LT-network performance.
- 6 Conclusion: Masking can be viewed as a training operation, leading to Supermasks that produce partially working networks without training.
Supplementary Information for:
The supplementary information is associated with the paper “Deconstructing Lottery Tickets: Zeros, Signs, and the Supermask.”
- The paper title identifies zeros as one focus of the work.
- The paper title identifies signs as one focus of the work.
- The paper title identifies the supermask as one focus of the work.
S1 Architectures and training hyperparameters
The supplementary materials describe the study’s architectures, training settings, data splits, and computational setup, largely following prior experiments.
- Training hyperparameters: The study uses the same training hyperparameters as the prior work without additional tuning.
- Data splits: MNIST uses 55000 training and 5000 validation examples, while CIFAR-10 uses 45000 training and 5000 validation examples.Validation sets were split randomly from the training data.
- Computation: Iterative pruning required networks to be trained up to 24 times, using one NVIDIA GeForce GTX 1080 Ti per experiment with parallel execution across GPUs.
- Architectures: The architectures use 3x3 convolutional layers with max pooling and fully connected layers, or fully connected layers sized according to prior work.Initializations are Glorot Normal and activations are ReLU.
S2 Further mask criteria details
Figure S1 compares mask criteria across four network architectures, reporting both convergence speed and test accuracy at different pruning rates.
- Figure S1 evaluates mask criteria across FC on MNIST and Conv2, Conv4, and Conv6 on CIFAR-10.
- The figure reports early stopping iteration as a convergence-speed measure for each mask criterion.
- The figure also reports test accuracy across various pruning rates.
S3 Further mask-0 action details
Control experiments test whether the proposed treatment of pruned weights drives its performance advantage. Both controls perform significantly worse than setting weights that moved toward zero to zero.
- Two controls test whether the proposed zero-weight selection, rather than treatment quirks, explains the performance improvement.
- Both randomly freezing pruned weights to zero and reversing the treatment perform significantly worse than the proposed treatment.
- The reversed treatment performs worse than both random zero-freezing and freezing all pruned weights at their initial values.
S6 Further training details for learning Supermasks
Supermask training uses specialized optimization settings because mask-only networks differ from ordinary networks, while the supplementary figures compare pruning, reinitialization, and Supermask treatments across architectures.
- Mask-only networks are trained with SGD and 0.9 momentum using architecture-specific learning rates and iteration counts.The FC, Conv2, Conv4, and Conv6 learning rates are 100, 100, 50, and 20, with 2000, 2000, 1000, and 800 training iterations, respectively.
- Conv4 and Conv6 exhibit significant overfitting, so early stopping is used during training.
- Mask initialization uses one constant across layers, ranging from -5 to 5, limiting control over per-layer pruning rates.
- Pruning-performance comparisons include early stopping iteration and test accuracy across mask criteria and pruned-weight treatments.
- Additional figures compare motion selectivity, reinitialization treatments, alternative mask criteria, and Supermask accuracy across network structures.