Source-linked AI summary
The Lottery Tickets Hypothesis for Supervised and Self-supervised Pre-training in Computer Vision Models
Tianlong Chen, Jonathan Frankle, Shiyu Chang, Sijia Liu, Yang Zhang, Michael Carbin, Zhangyang Wang
TL;DR
It is unclear whether pre-trained models can be simplified without damaging transferability across a variety of downstream tasks. The paper applies iterative magnitude pruning to supervised and self-supervised pre-trained models to identify matching subnetworks reusable across downstream tasks. Subnetworks transfer to downstream classification with nearly the same accuracies at 67.23%, 59.04%, and 59.04% sparsity (ImageNet, simCLR, MoCo), and to detection/segmentation without sacrificing performance at 73.79%/48.80%, 48.80%/36.00%, and 73.79%/83.22% sparsity, respectively.
Problem
It is unclear whether pre-trained models can be simplified without damaging transferability across a variety of downstream tasks.
Source: §1 · p. 2Method
Results
Subnetworks transfer to downstream classification with nearly the same accuracies at 67.23%, 59.04%, and 59.04% sparsity (ImageNet, simCLR, MoCo), and to detection/segmentation without sacrificing performance at 73.79%/48.80%, 48.80%/36.00%, and 73.79%/83.22% sparsity, respectively.
Source: §1 · p. 2Takeaways & Limitations
The findings support the relevance of lottery-ticket observations in computer-vision pre-training and suggest smaller subnetworks for downstream tuning without inhibiting transfer performance.
Source: §1 · p. 2, §6 · p. 8Practical hardware speedups for training or inference remain outside the study's scope and are left for future work.
Source: §3 · p. 4
Abstract
from arXiv · showhide
The computer vision world has been re-gaining enthusiasm in various pre-trained models, including both classical ImageNet supervised pre-training and recently emerged self-supervised pre-training such as simCLR and MoCo. Pre-trained weights often boost a wide range of downstream tasks including classification, detection, and segmentation. Latest studies suggest that pre-training benefits from gigantic model capacity. We are hereby curious and ask: after pre-training, does a pre-trained model indeed have to stay large for its downstream transferability? In this paper, we examine supervised and self-supervised pre-trained models through the lens of the lottery ticket hypothesis (LTH). LTH identifies highly sparse matching subnetworks that can be trained in isolation from (nearly) scratch yet still reach the full models' performance. We extend the scope of LTH and question whether matching subnetworks still exist in pre-trained computer vision models, that enjoy the same downstream transfer performance. Our extensive experiments convey an overall positive message: from all pre-trained weights obtained by ImageNet classification, simCLR, and MoCo, we are consistently able to locate such matching subnetworks at 59.04% to 96.48% sparsity that transfer universally to multiple downstream tasks, whose performance see no degradation compared to using full pre-trained weights. Further analyses reveal that subnetworks found from different pre-training tend to yield diverse mask structures and perturbation sensitivities. We conclude that the core LTH observations remain generally relevant in the pre-training paradigm of computer vision, but more delicate discussions are needed in some cases. Codes and pre-trained models will be made available at: https://github.com/VITA-Group/CV_LTH_Pre-training.
1. Introduction
The introduction asks whether pre-trained computer-vision models can be made smaller without losing transferability across downstream tasks. It applies the lottery ticket hypothesis to supervised and self-supervised pre-training and reports transferable matching subnetworks.
1. Introduction
- The study asks whether a pre-trained model can be simplified without damaging transferability across a variety of possible downstream tasks.This differs from conventional compression focused on one dataset and task.Source: §1 · p. 2
- The paper examines matching subnetworks in both supervised and self-supervised pre-trained computer-vision models, using pre-trained weights as the initialization for sparsification.Its study aims to find subnetworks reusable across downstream tasks.Source: §1 · p. 2
- Iterative magnitude pruning identifies matching subnetworks at pre-trained weights up to 67.23%, 59.04%, and 95.60% sparsity for ImageNet, simCLR, and MoCo, respectively.The paper also reports matching subnetworks at pre-trained initialization with 73.79% to 98.20% sparsity across downstream tasks.Source: §1 · p. 2
- Matching subnetworks transfer across classification, detection, and segmentation without sacrificing performance, with classification sparsities of 67.23%, 59.04%, and 59.04% for ImageNet, simCLR, and MoCo.For detection/segmentation, the reported sparsities are 73.79%/48.80%, 48.80%/36.00%, and 73.79%/83.22%, respectively.Source: §1 · p. 2
- Subnetworks found at pre-trained initialization are more sensitive to structure perturbations, while different pre-training methods tend to yield diverse masks and perturbation sensitivities.Source: §1 · p. 2
- Pruning from larger pre-trained models can produce better transferable subnetworks, and the work frames these as a step toward smaller models for downstream tuning.The paper presents this as a way to improve efficiency without inhibiting transfer performance.Source: §1 · p. 2
2. Related Works
Related work covers pruning and lottery-ticket research, transfer between classification tasks, and supervised and self-supervised computer-vision pre-training. It also contrasts the paper with NLP subnetworks and architecture adaptation approaches.
2. Related Works
- Pruning methods include unstructured weight-magnitude sparsification and structured removal of channels, the latter targeting hardware friendliness.Source: §2 · p. 2
- Lottery-ticket research identifies sparse subnetworks that can train to match dense-network performance, with rewinding approaches extending the scale of study beyond training from scratch.Prior work also examined transfer between image-classification tasks, while universal transfer across diverse computer-vision tasks at pre-trained initialization remained unexamined.Source: §2 · pp. 2–3
- BERT work found universally transferable subnetworks at 40% to 90% sparsity, but computer vision differs in pre-training objectives, architectures, and downstream-task priors.The passage contrasts self-attention and fully connected layers with convolutional architectures, and classification shift invariance with detection location-shift equivariance.Source: §2 · p. 3
- Computer-vision pre-training includes supervised ImageNet and self-supervised approaches such as simCLR and MoCo, which can reduce the labels needed for downstream image-classifier tuning.Source: §2 · p. 3
- A prior architecture-adaptation approach pre-trains a large super-net containing weight-shared subnetworks that can operate individually.Most pre-trained networks are subsequently fine-tuned with architectures unchanged.Source: §2 · p. 3
3. Preliminaries and Setups
The experiments use ImageNet-pretrained ResNet backbones and downstream classification, detection, and segmentation tasks. Matching subnetworks are found by iterative magnitude pruning and evaluated for transfer across tasks.
Experimental settings
- ResNet-50 is the default backbone; evaluation covers five classification datasets, Pascal VOC detection and segmentation, and multiple pre-training initializations.YOLOv4 and DeepLabv3+ use ResNet-50 for detection and segmentation; ResNet-152 is compared later. The first convolution and task-specific heads are not pruned.Source: §3 · pp. 3–4
Definitions
- A subnetwork is matching when it performs no worse than the dense model under the same training algorithm and evaluation metric.The paper uses “winning ticket” for a matching subnetwork at the pre-trained initialization.Source: §3 · p. 4
- A universal subnetwork uses task-specific modules with a shared mask across a task set, which may span classification, detection, and segmentation.The training procedure is defined for a task and iteration count; the evaluation function measures its returned model.Source: §3 · p. 4
Pruning Methods
- IMP trains the dense network, removes globally smallest-magnitude weights, and repeats pruning over multiple rounds.The first convolution and task-specific heads remain unpruned in the stated setup.Source: §3 · pp. 3–4
Scope
- Practical speedups on hardware are outside this study’s scope and are left for future work.The authors also identify structured pruning as a future direction.Source: §3 · p. 4
4. Transfer of Pre-training Winning Tickets
The section tests whether winning tickets found during pre-training transfer to downstream tasks and how transfer varies by pre-training method and task. The subnetworks transfer across classification, detection, and segmentation, with behavior depending on the downstream setting.
4. Transfer of Pre-training Winning Tickets
- Pre-training itself yields winning tickets at 67.23%, 59.04%, and 95.60% sparsity for supervised ImageNet, simCLR, and MoCo, respectively.These are the reported results for subnetworks found on the pre-training tasks.Source: §4 · p. 4
4.1. Transfer to Classification Tasks
- Transferred subnetworks match dense-model performance across natural-image classification tasks; on VisDA2017, ImageNet/simCLR/MoCo reach this at 67.23%, 59.04%, and 59.04% sparsity.The natural-image datasets include CIFAR-10, CIFAR-100, SVHN, and Fashion-MNIST; their reported no-loss sparsities are higher than those for VisDA2017.Source: Figure 3 · p. 5, Figure 4 · p. 5, §4.1 · pp. 5–6
- Classification transfer depends on pre-training and dataset: self-supervised tickets excel at extreme sparsity on natural images, while supervised tickets lead on VisDA2017 up to 89.26% sparsity.At extreme sparsity on VisDA2017, MoCo produces the more transferable subnetworks.Source: §4.1 · p. 6
- Task-specific subnetworks with pre-trained weights provide an upper bound in the comparisons and can remain matching at task-dependent sparsities.The reported matching sparsities vary across CIFAR-10, CIFAR-100, SVHN, Fashion-MNIST, and VisDA2007.Source: §4.1 · p. 6
4.2. Transfer to Detection and Segmentation
- Detection and segmentation tickets transfer successfully; MoCo ranks above ImageNet, which ranks above simCLR, while downstream-found tickets outperform transferred ones.Transferred-ticket sparsities for detection are 73.79%, 48.80%, and 73.79%, and for segmentation 48.80%, 36.00%, and 83.22%, respectively, for ImageNet, simCLR, and MoCo.Source: §4.2 · p. 6
5. Analyzing Properties of Pre-training Tickets
Masks from different pre-training tasks have markedly different structures, and pre-trained subnetworks respond more sharply to mask perturbations than those found from random or early-rewinding weights. For simCLR, larger pre-trained models yield better-transfering subnetworks at equal remaining weight counts, while extreme temperature settings reduce transfer performance.
5.1. Comparing Masks from Different Pre-trainings
- At 79.03% sparsity, zero kernels cluster mainly in early layers and are most dispersed in MoCo subnetworks.Figure 7 maps kernels from input to output; bright dots mark zero kernels and dark dots mark kernels with at least one unpruned weight.Source: §5.1 · p. 6, Figure 7 · p. 7
- After five IMP rounds, masks from different pre-training tasks overlap at less than 6.55% of locations, and MoCo subnetworks retain more zero kernels until 95.60% sparsity.Figure 6 reports relative mask similarity and zero-kernel counts; overlap decreases as sparsity increases.Source: §5.1 · p. 6, Figure 6 · p. 7
5.2. Pre-training versus Random Initialization
- Masks found from random or early-rewinding weights tolerate complementary, random-pruned, and perturbed variants, whereas those found from pre-trained weights lose substantial performance under these perturbations.The pre-trained initialization is described as highly structured, with perturbations potentially destroying its intrinsic structure.Source: §5.2 · p. 7, Figure 8 · p. 8
- For randomly pruned subnetworks, pre-trained initialization benefits accuracy up to high sparsity, such as 67.23%; beyond that, initialization no longer affects performance.The comparison is reported for CIFAR-100.Source: §5.2 · p. 7, Figure 8 · p. 8
5.3. More Ablation Studies for Pre-training
- At equal remaining weights, simCLR subnetworks from ResNet-152 outperform those from ResNet-50 on CIFAR-100; IMP is proposed before downstream fine-tuning, unlike [11]’s post-fine-tuning distillation.The compared ResNet-50 is around one-third the size of ResNet-152.Source: §5.3 · p. 8, Figure 9 · p. 8
- For simCLR subnetworks at 73.79% sparsity on CIFAR-100, transfer performance is close across temperatures 0.1–0.5 but degrades at extreme temperatures such as 20.0.The temperature ablation starts at 67.23% sparsity, trains for 10 epochs, then prunes to 73.79%.Source: §5.3 · p. 8, Table 2 · p. 8
6. Conclusion
The study applies iterative magnitude pruning to supervised and self-supervised computer-vision pre-training. It finds matching subnetworks at non-trivial sparsities that can be trained to full-model performance on pre-training and downstream tasks, and discusses cross-task transferability.
6. Conclusion
- The study examines the lottery ticket hypothesis in computer-vision pre-training using supervised and self-supervised approaches.The approaches include ImageNet classification, simCLR, and MoCo.Source: §6 · p. 8
- Iterative magnitude pruning from pre-trained initializations finds matching subnetworks at non-trivial sparsity levels that can be independently trained to full-model performance.The reported performance scope includes both pre-training and downstream tasks.Source: §6 · p. 8
- The conclusion also presents a detailed discussion of matching subnetworks’ cross-task universal transferability.Source: §6 · p. 8
A.1. Pruning Algorithm
The appendix outlines iterative magnitude pruning from pre-trained weights and describes how top-1 retrieval accuracy is calculated for self-supervised pre-training.
A.1. Pruning Algorithm
- IMP trains the masked pre-trained model, prunes 20% of remaining weights, and repeats until the target sparsity is reached.The procedure returns the model with the resulting mask applied to the pre-trained weights.Source: §A.1 · p. 12
A.2. Top-1 Retrieval Accuracy
- Top-1 retrieval accuracy compares representations from two augmented views using cosine similarity and counts correct nearest-representation matches.For each representation, the procedure checks whether its most similar representation is its paired view; the accuracy formulation uses an indicator function.Source: §A.2 · p. 12
B.1. YOLOv4 Detection Results with Other Metrics
YOLOv4 detection subnetworks transfer without performance degradation relative to full unpruned models under both AP50 and AP75. The sparsity levels at which this holds differ by metric and pre-training task.
B.1. YOLOv4 Detection Results with Other Metrics
- At AP50, sparsities of 83.22%, 89.26%, and 36.00% for ImageNet, simCLR, and MoCo, respectively, preserve full-model detection performance.Figure 10 reports these IMP-subnetwork results across downstream tasks.Source: Figure 10 · p. 12, §B.1 · p. 12
- At AP75, sparsities of 73.79%, 48.80%, and 79.03% for ImageNet, simCLR, and MoCo, respectively, preserve full-model detection performance.Source: Figure 10 · p. 12, §B.1 · p. 12
B.2. Faster RCNN and SSD Detection Results
Winning tickets identified during pre-training generally fail to remain matching subnetworks for Faster RCNN and SSD detection, with limited MoCo exceptions. The simCLR detection results are substantially below reported BYOL results and are presented as appendix failure cases, though the authors say this hardly affects their main conclusions.
B.2. Faster RCNN and SSD Detection Results
- With Faster RCNN and SSD, pre-training-task winning tickets generally cease to be matching subnetworks; MoCo exceptions occur at 59.04% sparsity for Faster RCNN AP50 and 83.22%, 86.58%, and 83.22% sparsity for SSD AP, AP50, and AP75, respectively.Other conclusions align with the YOLOv4 detector results in the main text.Source: §B.2 · p. 12, Figure 11 · p. 13
- simCLR detection results remain inferior to reported BYOL results despite hyperparameter and implementation comparisons, and the authors cannot exclude sensitivity to PyTorch versus TensorFlow.The authors describe this framework-sensitivity possibility as marginal.Source: §B.2 · p. 13
- The authors place simCLR Faster RCNN and SSD detection results in the appendix as failure cases, noting that this hardly affects their main observations and conclusions.Source: §B.2 · p. 14
B.3. Ablation about Larger Pre-training Models
The simCLR model-size comparison favors heavily compressed subnetworks from the larger model, while the smaller model performs slightly better in top-1 retrieval at extreme sparsity.
B.3. Ablation about Larger Pre-training Models
- Heavily compressed simCLR ResNet-50 subnetworks outperform lightly compressed ResNet-152 subnetworks, but the smaller model has slightly better top-1 retrieval accuracy at extreme sparsity.Source: §B.3 · p. 14, Figure 12 · p. 14