Source-linked AI summary
How Well Do Self-Supervised Models Transfer?
Linus Ericsson, Henry Gouk, Timothy M. Hospedales
TL;DR
The study addresses limited, inconsistent evidence about which self-supervised representations transfer best across diverse downstream tasks. It benchmarks 13 models on 40 tasks against supervised pre-training and finds that the best self-supervised methods usually outperform supervision, while ImageNet performance is not uniformly predictive across tasks.
Problem
Self-supervised methods lack consistent large-scale comparisons across diverse downstream tasks, making their relative effectiveness difficult to assess.
Method
The authors evaluate 13 self-supervised representations on 40 transfer tasks spanning many-shot and few-shot recognition, object detection, surface normal prediction, and semantic segmentation.
Results
The best self-supervised methods usually outperform supervised pre-training across tasks, though supervised learning wins in few-shot recognition with small domain shift to ImageNet.
Takeaways & Limitations
ImageNet performance is broadly representative for natural-image recognition but is not reliably representative for unstructured image recognition and other downstream tasks.
Takeaways & Limitations
The study does not compare ImageNet-transferred representations with domain-specific self-supervised representations trained on each target dataset.
Abstract
from arXiv · showhide
Self-supervised visual representation learning has seen huge progress recently, but no large scale evaluation has compared the many models now available. We evaluate the transfer performance of 13 top self-supervised models on 40 downstream tasks, including many-shot and few-shot recognition, object detection, and dense prediction. We compare their performance to a supervised baseline and show that on most tasks the best self-supervised models outperform supervision, confirming the recently observed trend in the literature. We find ImageNet Top-1 accuracy to be highly correlated with transfer to many-shot recognition, but increasingly less so for few-shot, object detection and dense prediction. No single self-supervised method dominates overall, suggesting that universal pre-training is still unsolved. Our analysis of features suggests that top self-supervised learners fail to preserve colour information as well as supervised alternatives, but tend to induce better classifier calibration, and less attentive overfitting than supervised learners.
1. Introduction
Self-supervised representation learning addresses annotation bottlenecks by learning CNN features from freely available labels, but its transfer value across tasks remained incompletely benchmarked. This study evaluates 13 self-supervised models across 40 diverse transfer tasks against supervised pre-training.
- Motivation: Annotation-intensive computer vision progress is bottlenecked by data labeling, motivating self-supervised CNN representations trained on pretext tasks with freely available labels.These representations can support more data-efficient learning on new tasks through feature reuse or finetuning.
- Motivation: Recent self-supervised representations appear to approach or surpass fully supervised representations, raising the possibility of replacing annotation-intensive training.
- Open questions: Benchmarking remains difficult because published methods use inconsistent evaluation conditions and classification metrics do not establish transfer to detection or dense prediction.
- Study design: The study evaluates 13 self-supervised models on 40 transfer tasks spanning many-shot and few-shot classification, object detection, surface normal prediction, and semantic segmentation.The downstream datasets vary in similarity to ImageNet, where all models were pre-trained.
- Main findings: The best self-supervised methods match or outperform supervised representation learning across most tasks, while ImageNet performance is less predictive outside similar recognition settings and no method dominates overall.These findings are stated as the study’s answers to its three central benchmarking questions.
2. Related Work
Prior work spans diverse self-supervised objectives and downstream evaluations, but comparisons often vary architectures, datasets, or task coverage. This study instead compares self-supervised algorithms with a fixed architecture and dataset while broadening transfer benchmarks.
- Self-supervised learning: Self-supervised learning commonly pre-trains still-image representations on ImageNet using objectives including jigsaw puzzles, colorization, instance discrimination, and clustering.Evaluation typically transfers the learned representation to a downstream classifier.
- Self-supervised learning: Instance discrimination treats each training image as its own class and uses strong augmentation with contrastive loss to learn features resilient to view changes.
- Evaluation scope: Earlier evaluations studied architecture–pretext-task fit or pre-training scale and capacity, whereas this work compares different objectives using the same ResNet-50 architecture and ImageNet source data.
- Evaluation scope: The benchmark includes few-shot recognition, object detection, and dense prediction in addition to recognition, revealing high correlation on ImageNet-like recognition tasks but lower correlation elsewhere.
3. Preliminaries
The study evaluates thirteen self-supervised methods using common ResNet-50 backbones and transfers their features through task-specific heads or full-network finetuning. The benchmark includes few-shot transfer on CD-FSL among its downstream evaluations.
- Methods: The benchmark covers thirteen methods spanning contrastive approaches— including MoCo, SimCLR, InfoMin, and BYOL—and clustering approaches such as PCL, DeepCluster-v2, and SwAV.
- Common setup: All models use ResNet50(1×) backbones with 23.5M parameters and ImageNet pre-training on 1.28M images, while only the supervised baseline uses labels.
- Common setup: Pre-training procedures differ in training duration, augmentation, loss, and additional architectural elements, although every model uses the same backbone for evaluation.
- Transfer benchmarks: Table 3 reports 5-way 20-shot few-shot transfer on CD-FSL, a benchmark used to compare pre-trained representations across target datasets.
- Transfer procedure: For each target dataset, backbone features feed a task-specific prediction head, with either head-only optimization or full-network finetuning.
4. Experiments
Across diverse transfer settings, the strongest self-supervised models generally outperform supervised pre-training, but the leading method depends on the task and ImageNet performance is not universally predictive. Feature analyses further show differences in calibration, colour retention, and attentive focus between self-supervised and supervised representations.
- Many-shot recognition: On all but one downstream task, the best self-supervised methods outperform supervised pre-training in linear evaluation.On Aircraft and Cars, the advantage exceeds 10% absolute performance.
- Many-shot recognition: DeepCluster-v2, SwAV, and SimCLR-v2 still outperform supervision overall after finetuning, although the supervised model leads on three datasets.
- Few-shot recognition: The supervised model dominates most Kornblith few-shot datasets, winning by more than 5% on Aircraft and Cars, unlike the many-shot results.
- Few-shot recognition: Self-supervision lags for low-domain-shift few-shot transfer but consistently beats supervision for larger domain shifts.
- Object detection: SimCLR-v2 excels with frozen detection backbones and BYOL with finetuned backbones, while the best models differ from those leading recognition.
- Dense prediction: On NYUv2 surface normal estimation, SimCLR-v2 wins across metrics and outperforms supervised pre-training by around 4–10%, followed by BYOL.
- Dense prediction: PCL-v1 tops semantic segmentation despite performing poorly in recognition, while the supervised baseline is among the worst methods.
- ImageNet transferability: ImageNet performance correlates highly with many-shot recognition but increasingly less with few-shot recognition, detection, and dense prediction.
5. Discussion
The evaluation finds that strong self-supervised methods usually outperform supervised pre-training across diverse transfer tasks, but no universally best representation emerges. The study also identifies a major scope limitation: it does not compare ImageNet-trained representations with domain-specific self-supervised models.
- Self-supervised methods usually outperform supervised pre-training as a source of transferable knowledge across the evaluated tasks.Supervised representation learning wins only for few-shot recognition with small domain shift from ImageNet.
- ImageNet performance broadly represents downstream performance for natural-image recognition, but not reliably for unstructured recognition or spatially sensitive tasks.The latter include detection, surface normal prediction, and semantic segmentation.
- A universal pre-trained feature with best performance across diverse downstream tasks has not yet been realized.The authors recommend broader benchmarking to improve SSL's relevance to the wider computer vision community.
- The study does not compare ImageNet-transferred self-supervised representations with domain-specific representations trained on each target dataset.Consequently, whether domain-specific SSL is worthwhile and whether ImageNet features are truly generic remains for future work.
A.1. Pre-trained models
The pre-trained model evaluation includes implementation-specific choices for two models whose publicly available weights differed from their original reports.
- InsDis uses PyContrast weights reporting 59.5 ImageNet top-1 accuracy, versus 54.0 originally reported.PIRL likewise uses PyContrast weights because weights were unavailable from the original source.
- PIRL uses PyContrast weights reporting 61.7 ImageNet accuracy, compared with 63.6 originally reported.All other models were obtained from their original authors.
A.2. Many-shot evaluation details
Many-shot evaluation uses task-specific recognition metrics and Pascal VOC mean average precision, with validation-based regularization and optimization choices for linear evaluation and finetuning.
- Top-1 accuracy is used for six recognition datasets, mean per-class accuracy for four, and 11-point mAP for Pascal VOC 2007.The datasets are Food-101, CIFAR-10, CIFAR-100, SUN397, Stanford Cars, DTD; FGVC Aircraft, Oxford-IIIT Pets, Caltech-101, Oxford 102 Flowers; and Pascal VOC 2007.
- Caltech-101 training uses 30 randomly selected images per class, with testing on the remaining images.
- Linear classifiers use L-BFGS with softmax cross-entropy, while Pascal VOC 2007 uses one binary classifier per class because it is multi-label.
- Finetuning runs for 5000 steps with batch size 64, SGD with Nesterov momentum 0.9, cosine annealing, and grid-selected learning-rate and weight-decay settings.
A.3. Few-shot evaluation details
Few-shot episodes use frozen 2048-dimensional backbone features and a nearest-centroid classifier, evaluated on 15 query images after standardized resizing and cropping.
- Each few-shot episode fits a nearest centroid classifier on 2048-dimensional features extracted from frozen backbones.
- Episodes sample images from the combined train, validation, and test sets, without augmentation.
- Images are resized to 224 pixels along the shorter side with bicubic resampling, then center-cropped to 224 × 224.
- Each episode is evaluated using 15 query images.
A.4. Detection evaluation details
Detection transfer uses Faster R-CNN with a Feature Pyramid Network over ResNet backbone features, evaluated under frozen-backbone and full-finetuning setups.
- Detection setup: Faster R-CNN detector heads consume backbone features extracted with a Feature Pyramid Network.The backbone is based on ResNet architectures.
- Training regimes: Frozen-backbone evaluation freezes all but the final residual block, whereas full finetuning leaves the entire network trainable.
- Data and evaluation: VOC 2007 and 2012 trainval images are used for training, with evaluation on the VOC 2007 test set.
A.5. Surface normal estimation evaluation details
The surface-normal evaluation trains models with frozen backbones for 150 epochs using momentum SGD and a scheduled learning rate.
- Training setup: 150 epochs of training are performed with the full backbone frozen.
- Optimization: The optimizer uses stochastic gradient descent with momentum 0.9 and batch size 4.
- Learning-rate schedule: The learning rate follows (1 − t/T)^0.9, where t is the current epoch and T is the total number of epochs.
A.6. Semantic segmentation evaluation details
The evaluation analysis standardizes bounded metrics with logit transformations and examines statistical relationships, reconstruction fidelity, and attentive feature sensitivity.
- Statistical analysis: Bounded metrics are logit-transformed so equal percentage changes reflect differing significance across the accuracy range.The procedure is motivated by contrasting changes such as 50% to 51% and 98% to 99%.
- Statistical analysis: Correlations use transformed ImageNet top-1 accuracy and task-specific transformed recognition, detection, surface-normal, and segmentation metrics.Mean and median surface-normal errors are instead negated before correlation analysis.
- Feature analysis: Feature retention is probed by reconstructing images from backbone features with a trained reconstruction network initialized from fixed noise.
- Feature analysis: Attentive feature sensitivity is estimated by averaging root relative squared error over repeated pixel occlusions.The occlusion mask is 10 × 10 pixels, and RRSE makes distances invariant to the scale of the original features.
- Implementation details: The paper reports training details for all evaluated models, including models obtained from PyContrast rather than their original authors.