Source-linked AI summary
Self-supervised Pretraining of Visual Features in the Wild
Priya Goyal, Mathilde Caron, Benjamin Lefaudeux, Min Xu, Pengchao Wang, Vivek Pai, Mannat Singh, Vitaliy Liptchinsky, Ishan Misra, Armand Joulin, Piotr Bojanowski
TL;DR
The paper asks whether self-supervised pretraining remains effective on extremely large collections of random, uncurated, unlabeled images rather than curated datasets. It trains high-capacity models with self-supervised learning and finds competitive downstream performance, including strong few-shot learning and 84.2% ImageNet top-1 accuracy, 1% above the best existing pretrained model.
Problem
Prior computer-vision pretraining evidence largely uses curated datasets, leaving open whether self-supervision works on extremely large collections of random, uncurated, unlabeled images.
Method
The paper pretrains high-capacity RegNet models on billions of random internet images without metadata or annotations, using self-supervised learning with SwAV.
Results
84.2% top-1 accuracy on ImageNet surpassed the best existing self-supervised pretrained model by +1%, while SEER achieved 75.1% with only 10% of ImageNet.
Takeaways & Limitations
Self-supervised pretraining on random images achieves competitive downstream performance and supports strong few-shot learning beyond curated-dataset initialization.
Abstract
from arXiv · showhide
Recently, self-supervised learning methods like MoCo, SimCLR, BYOL and SwAV have reduced the gap with supervised methods. These results have been achieved in a control environment, that is the highly curated ImageNet dataset. However, the premise of self-supervised learning is that it can learn from any random image and from any unbounded dataset. In this work, we explore if self-supervision lives to its expectation by training large models on random, uncurated images with no supervision. Our final SElf-supERvised (SEER) model, a RegNetY with 1.3B parameters trained on 1B random images with 512 GPUs achieves 84.2% top-1 accuracy, surpassing the best self-supervised pretrained model by 1% and confirming that self-supervised learning works in a real world setting. Interestingly, we also observe that self-supervised models are good few-shot learners achieving 77.9% top-1 with access to only 10% of ImageNet. Code: https://github.com/facebookresearch/vissl
1. Introduction
The paper asks whether self-supervised learning can scale beyond curated datasets, and addresses this by pretraining high-capacity models on billions of random internet images without annotations. SEER models achieve strong ImageNet and few-shot results, while also transferring well to downstream tasks.
- Motivation: Self-supervised pretraining has mainly been evaluated on curated datasets, leaving its effectiveness on extremely large collections of random, uncurated, unlabeled images open.The paper frames this as an important question for using unlabeled data and continually learning from unending data streams.
- Approach: The study pretrains high-capacity models on billions of completely unconstrained internet images without metadata, filtering, or manual annotations.The analysis focuses on RegNet architectures and uses the SwAV self-supervised approach in an online setting.
- Findings: Self-supervised pretraining on uncurated data outperforms supervised ImageNet pretraining on several downstream tasks.The paper presents this as evidence that self-supervised pretraining benefits transfer even when performed on uncurated data.
2. Related Work
Prior work shows that pretraining, large datasets, and high-capacity architectures improve visual representations, but most self-supervised studies rely on curated or filtered data. This work extends that line by combining self-supervision, large-scale training, and scale-efficient RegNet architectures on uncurated images.
- Unsupervised pretraining: Self-supervised methods have produced features that surpass supervised representations on many downstream tasks, but most studies use ImageNet or filtered datasets.The cited methods include instance discrimination and cluster-assignment approaches.
- Learning at scale: Large-scale visual pretraining has used weak supervision, semi-supervision, or supervision on hundreds of millions of filtered images.Prior work also reports that pretraining on billions of images improves large architectures compared with training on smaller datasets.
- Architecture scaling: Large architectures are especially important for large pretraining datasets because limited-capacity models may underfit, particularly under contrastive learning.Contrastive methods must discriminate between dataset instances to learn useful representations.
- RegNet: RegNets provide competitive benchmark performance with efficient runtime and memory usage, making them suitable candidates for large-scale self-supervised pretraining.The paper uses RegNets to study the benefits of this model family for pretraining at scale.
3. Method
The method combines SwAV online clustering with scale-efficient RegNetY architectures and training strategies designed for billions of random images. SwAV learns augmentation-consistent cluster assignments, while RegNetY supports joint scaling of data and model capacity.
- 3.1. Self-Supervised Pretraining: SwAV trains convolutional networks without annotations by making cluster assignments consistent across multiple augmented views of each image.Features are assigned to learnable prototypes with an Optimal Transport solver that encourages uniform cluster usage, after which assignments are swapped between views.
- 3.1. Self-Supervised Pretraining: SwAV’s cluster prediction loss compares each view’s features with the other view’s cluster assignment using cross entropy over prototype similarities.The loss is defined as L(f_i^1, f_i^2) = ℓ(f_i^1, y_i^2) + ℓ(f_i^2, y_i^1).
- 3.2. Scale efficient model family: RegNetY: RegNets are designed to scale data and model capacity efficiently in memory and runtime, and RegNetY adds squeeze-and-excitation operations to the standard RegNet design.The family uses four stages of repeated residual bottleneck blocks and is parameterized to support architecture scaling.
- 3.2. Scale efficient model family: RegNetY: The study focuses on RegNetY architectures, including the RegNetY-256GF model, whose specified design has 695.5M parameters.Its four stages have depths (2, 7, 17, 1) and widths (528, 1056, 2904, 7392).
- Large-scale training: The training pipeline adapts self-supervised learning to large scale through learning-rate schedules, gradient checkpointing, mixed precision, and optimized distributed operations.The largest model uses cosine learning-rate scheduling, while checkpointing trades recomputation for reduced memory consumption and mixed precision improves throughput.
- Large-scale training: Pretraining samples random public non-EU Instagram images online without curation, preprocessing, hashtag filtering, or deduplication, using a refreshed dataset.The main setup uses six crops per image, a three-layer projection head, 16K prototypes, and training on 1B images across 512 GPUs.
4. Main Results
SEER features transfer effectively across standard and low-shot evaluations, including ImageNet, Places205, classification benchmarks, and COCO detection and segmentation. Performance benefits increase with limited data and larger model capacity, while self-supervised pretraining outperforms supervised alternatives in several comparisons.
- ImageNet classification: 84.2% top-1 accuracy on ImageNet is achieved by the largest self-supervised pretrained RegNet, surpassing the best existing pretrained model by +1%.The comparison uses RegNetY-256GF finetuned on ImageNet.
- ImageNet classification: Self-supervised pretraining has a more significant capacity-related impact than training from scratch as RegNets scale to hundreds of millions of parameters.The authors suggest scratch training may overfit ImageNet, a relatively small dataset.
- Low-shot learning: Self-supervised pretraining yields a stable 2.5% top-1 gain on Places205 across fractions of the training data.The comparison uses RegNetY-128GF and supervised ImageNet pretraining.
- Low-shot learning: 77.9% top-1 accuracy is achieved with only 10% of ImageNet, despite competing methods using all ImageNet images during pretraining.At 1% of ImageNet, the gap increases significantly because other methods pretrain on the full dataset.
- Low-shot learning: Increasing model capacity produces higher relative improvement as access to labels and images decreases, with comparable +20% relative gains in the 1% setting.The analysis compares 1%, 10%, and 100% of ImageNet, including supervised full-data reference models.
- Transfer to other benchmarks: Self-supervised features transfer better than supervised features on classification, detection, and segmentation evaluations.On COCO detection and segmentation, the advantage is 1.5−2 AP points, while architecture gaps are only 0.1−0.5 AP.
5. Ablation Studies
The ablations examine architecture, training-data scale, and self-supervised head capacity. Larger model capacity, more updates, more unique images, and a larger head improve learned feature quality under the reported evaluations.
- Experimental setting: The ablations vary architecture, capacity, pretraining data, and self-supervised head design using SwAV on random Instagram images.Models are pretrained for 1 epoch on 1B random public and non-EU Instagram images with shared hyperparameters across ablation models.
- Impact of the architecture: RegNets surpass the compared ResNet and ResNeXt architectures in linear evaluation of frozen ImageNet features.The evaluation probes pretrained features directly rather than favoring models that train well from scratch on ImageNet.
- Impact of the architecture: Increasing model capacity significantly improves feature quality with a logarithmic gain in performance, regardless of architecture.
- Impact of pretraining data: Performance steadily increases with the number of updates and does not saturate after updates corresponding to 1B images.The study uses an online RegNetY-128GF pretraining setup and evaluates frozen features with a linear ImageNet classifier.
- Impact of pretraining data: The number of updates matters more than repeatedly seeing the same images, supporting continual online pretraining without fixing the dataset.The comparison holds updates constant while varying the number of unique images for RegNetY-8GF and RegNetY-16GF models.
- Scaling the self-supervised model head: Scaling the self-supervised head improves ImageNet top-1 accuracy by +3% without changing the model or feature size.The larger head increases MLP capacity and prototypes from 3K to 16K, reducing information leakage caused by the limited original SwAV classifier.
6. Conclusion
The study finds that self-supervised pretraining on random, unlabeled images produces competitive downstream features. Its scalability depends on large datasets, efficient high-capacity architectures, and especially RegNet scaling.
- Pretraining on random images with no annotation achieves competitive performance on downstream tasks.
- The result indicates that recent self-supervised learning progress is not specific to curated training sets.
- Scaling modern self-supervised learning methods to uncurated data benefits from efficient, high-capacity architectures, with RegNet scalability playing a key role.
7. Model architectures.
The paper evaluates ResNet, ResNeXt, and RegNetY architectures with shared pretraining settings, while adapting the RegNetY projection head to each architecture's output width.
- Architectures: The architecture study compares ResNet-{50,101}, ResNeXt RX101-32x{4,8}d, and RegNetY-{8,16,32,64,128}GF models.
- ResNet and ResNeXt pretraining: ResNet and ResNeXt models use SwAV with eight crops, a 2-layer MLP projection head, 3K prototypes, and fixed temperature and Sinkhorn regularization settings.
- RegNetY pretraining: RegNetY architectures are generated using published scaling rules and span five capacities from 8GF through 128GF.
- RegNetY configurations: RegNetY-128GF uses depth 27 with w0 = 456, wa = 160.83, wm = 2.52, and group width = 264.
- RegNetY pretraining: RegNetY pretraining uses the same hyperparameters as ResNet and ResNeXt training, with different crop counts and architecture-specific 3-layer MLP heads.The head dimensions are adapted to each RegNetY architecture's output channels.