Source-linked AI summary
One ticket to win them all: generalizing lottery ticket initializations across datasets and optimizers
Ari S. Morcos, Haonan Yu, Michela Paganini, Yuandong Tian
TL;DR
Winning-ticket initializations are expensive to discover, and their reuse across datasets and optimizers was uncertain. The paper generates tickets in one configuration and evaluates them in others, finding broad transfer within natural-image tasks, especially from larger datasets, while identifying important scope and efficiency limitations.
Problem
Winning tickets are computationally expensive to find, and it was unclear whether their benefits were specific to the generating dataset and optimizer.
Method
The study generates winning tickets in one dataset–optimizer configuration and evaluates their performance on different datasets and optimizers, using late resetting.
Results
Winning tickets transferred across natural-image datasets and optimizers, often approaching dataset-specific performance, with larger-dataset tickets transferring better.
Takeaways & Limitations
Winning tickets generated from sufficiently large datasets appear to contain inductive biases that improve training across multiple configurations.
Takeaways & Limitations
Transfer was evaluated only within the natural-image domain and object-classification task, leaving cross-domain and cross-task generalization unresolved.
Abstract
from arXiv · showhide
The success of lottery ticket initializations (Frankle and Carbin, 2019) suggests that small, sparsified networks can be trained so long as the network is initialized appropriately. Unfortunately, finding these "winning ticket" initializations is computationally expensive. One potential solution is to reuse the same winning tickets across a variety of datasets and optimizers. However, the generality of winning ticket initializations remains unclear. Here, we attempt to answer this question by generating winning tickets for one training configuration (optimizer and dataset) and evaluating their performance on another configuration. Perhaps surprisingly, we found that, within the natural images domain, winning ticket initializations generalized across a variety of datasets, including Fashion MNIST, SVHN, CIFAR-10/100, ImageNet, and Places365, often achieving performance close to that of winning tickets generated on the same dataset. Moreover, winning tickets generated using larger datasets consistently transferred better than those generated using smaller datasets. We also found that winning ticket initializations generalize across optimizers with high performance. These results suggest that winning ticket initializations generated by sufficiently large datasets contain inductive biases generic to neural networks more broadly which improve training across many settings and provide hope for the development of better initialization methods.
1 Introduction
Lottery ticket initializations may contain unusually trainable sparse subnetworks, but finding them is expensive and their generality across training conditions was unclear. This work tests reuse across datasets and optimizers and reports broad transfer within natural images.
- The lottery ticket hypothesis proposes that over-parameterized networks contain smaller subnetwork initializations that can match the full network when trained independently.
- Finding winning tickets is computationally expensive because it repeatedly trains models to convergence and prunes them.
- The central question is whether winning-ticket benefits depend on the precise architecture, optimizer, and dataset used to generate them.
- The study finds tickets that transfer across many natural-image datasets, often performing nearly as well as dataset-specific initializations.
- Tickets generated on larger datasets transfer better, and winning tickets also generalize across optimizers.
2 Related work
Related work situates this study at the intersection of lottery-ticket initialization, pruning methodology, and transfer learning. Unlike prior pruning transfer studies, the paper transfers initializations rather than learned representations.
- The lottery ticket hypothesis links network over-parameterization to the presence of trainable sparse subnetworks with competitive accuracy.
- Pruning research includes magnitude, structured, recoverable, greedy, and variational-dropout methods, with pruning choices affecting resulting ticket structure.
- Prior transfer-learning work commonly prunes a model on one dataset and fine-tunes the resulting network on another dataset or task.
- Figure 1 compares global and layerwise pruning using CIFAR-10 winning-ticket performance and layerwise pruning-rate ratios in VGG19.
3 Approach
The approach generates winning tickets through iterative pruning and evaluates tickets from one source configuration on different target datasets or optimizers. Global pruning and late resetting are used, with output layers reinitialized when transfer changes the number of classes.
- Winning tickets are sparse subnetworks whose retained weights and pruning mask are evaluated after iterative training and pruning.
- Late resetting restores winning-ticket weights to an early training iteration k rather than iteration 0, improving tickets and removing the need for warmup in large models.
- Global pruning pools weights across layers, allowing different layerwise pruning fractions; the study uses it because it consistently outperformed local pruning.
- Global magnitude pruning preferentially removes weights from deeper layers, while preserving more of the first layer because early layers contain fewer parameters.
- The experiments use modified VGG19 and ResNet50 models.
- Transfer evaluation generates a ticket on a source configuration and tests it on a target configuration, randomly reinitializing an incompatible output layer.
- Because source and target configurations differ, each ticket is retrained on the target dataset to measure target performance.
4 Results
Winning tickets transferred across samples, natural-image datasets, architectures, and optimizers, with stronger generalization from larger, more complex source datasets and additional regularization in an over-parameterized setting.
- Experimental setup: Test accuracy at convergence was plotted against the fraction of pruned weights using six random-seed replicates and matched training epochs for each comparison.Shaded error regions represented ±1 standard deviation.
- Transfer within the same data distribution: Winning tickets generated on CIFAR-10a generalized well to CIFAR-10b for both VGG19 and ResNet50.Each half contained 25,000 training images, and comparisons included random and target-half winning tickets.
- Transfer across datasets: Six natural-image datasets spanning grayscale or color, input size, class count, and training-set size were used to evaluate cross-dataset transfer.The datasets were Fashion-MNIST, SVHN, CIFAR-10, CIFAR-100, ImageNet, and Places365.
- Transfer across datasets: Winning tickets sourced from large datasets generalized across all targets with performance close to target-dataset-specific tickets.ImageNet- and Places365-sourced tickets were highlighted as competitive across all datasets.
- Transfer across datasets: Larger, more complex source datasets consistently produced better-transferring winning tickets than smaller datasets.The effect reflected both training-set size and class count: CIFAR-100 tickets generalized better than CIFAR-10 tickets despite equal training-example counts.
- Transfer across datasets: For VGG19 on Fashion-MNIST, transferred tickets dramatically outperformed Fashion-MNIST-specific tickets at low pruning rates by avoiding severe overfitting.The authors suggest transferred tickets provide additional regularization in this highly over-parameterized setting.
- Architecture differences: Transfer success was roughly similar for VGG19 and ResNet50, but ResNet50 showed a sharper pruning cliff on large-scale datasets.ResNet50 performance rapidly degraded when approximately 5-10% of weights remained, whereas VGG19 declined only slightly even with 99.9% pruned.
- Transfer across optimizers: VGG winning tickets transferred across SGD with momentum and Adam, achieving similar performance to tickets generated with the source optimizer overall.SGD-to-Adam transfers underperformed random tickets at low pruning fractions but significantly outperformed them at high pruning fractions.
5 Discussion
Winning tickets transfer across training configurations, with larger datasets producing more generic initializations. Important limitations remain around computational cost, domain and task scope, dataset-dependent gaps, architecture topology, and the unclear source of their effectiveness.
- Winning tickets transfer across datasets and optimizers, suggesting sufficiently large-dataset tickets encode inductive biases that improve sparsified-model training more generally.The conclusion is supported by results across Figures 3 and 4.
- Larger datasets, measured by training samples and classes, consistently produce winning tickets that transfer better across datasets.
- Generating tickets remains slow because iterative pruning can require retraining the source model up to 30 times serially at extreme pruning fractions such as 0.999.The computational burden is especially substantial for larger datasets, which require significant compute per training run.
- The transfer evaluation covers only natural-image datasets and object-classification tasks, leaving cross-domain, diverse-task, and multimodal generalization unresolved.
- Transferred tickets often show a small performance gap versus dataset-specific tickets, especially when generated on small datasets, indicating some dataset-dependent inductive bias.The paper leaves the dataset-dependent and dataset-independent components unresolved.
- Because source and target topologies were fixed, a new ticket is needed for each architecture topology; parameterizing tickets for novel architectures remains future work.The experiments suggest some layers can be re-initialized without substantially damaging the ticket.
- The precise properties that make winning tickets special remain unclear despite evidence that their useful characteristics are somewhat generic.Understanding these properties is identified as important for future initialization strategies.
A.1 Model details and hyperparameters
The appendix specifies the training, architecture, optimizer, and pruning configurations used across the evaluated models.
- General hyperparameters: Models used batch size 512, distributed across 2 GPUs for smaller datasets and 16 GPUs with synchronous training for ImageNet and Places365.
- General hyperparameters: SGD used learning rate 0.1, momentum 0.9, and weight decay 0.0001, while Adam used learning rate 0.0003.The supplied passage truncates the Adam beta specification.
- VGG19: VGG19 replaced fully connected layers with average pooling, used ReLU and batch normalization throughout, and specified convolutional filter sizes through global average pooling.
- ResNet50: ResNet50 used the stated block configuration, trained for 90 epochs, and annealed learning rates by 10 at epochs 50, 65, and 80.
- Pruning parameters: All models used iterative magnitude pruning at rate 0.2 for 30 iterations, with late-reset settings varying by dataset scale.Performance was evaluated every third iteration after the sixth.
A.2 Randomized masks
The randomized-mask analysis separates information in winning-ticket weights from information in the pruning-mask structure. Winning tickets outperform random-mask controls, and preserving layerwise mask statistics matters substantially.
- Figure A1 compares random-mask performance on CIFAR-10, with error bars showing mean ± standard deviation across six random seeds.
- Randomized-mask controls test preserved, locally permuted, and globally permuted masks to distinguish exact mask structure from layerwise statistics.Local permutation preserves layerwise statistics, whereas global permutation removes information across layers.
- Global pruning produces better performance and different layerwise pruning ratios than layerwise pruning, indicating that mask statistics contain important information.
- Winning tickets outperform all random tickets; local mask permutation damages performance somewhat, while global permutation causes dramatically worse performance.The purely random mask is treated as the relevant comparison for an equivalently parameterized model trained from scratch.