Source-linked AI summary
Are Large-scale Datasets Necessary for Self-Supervised Pre-training?
Alaaeldin El-Nouby, Gautier Izacard, Hugo Touvron, Ivan Laptev, Hervé Jegou, Edouard Grave
TL;DR
The paper asks whether self-supervised pre-training requires large datasets such as ImageNet and studies smaller, varied, and target-task data instead. It compares denoising autoencoders with joint embedding methods and finds that denoising autoencoders remain robust across pre-training data sizes and distributions. The results support competitive pre-training on datasets substantially smaller than ImageNet, including COCO.
Problem
The paper examines whether self-supervised pre-training requires large datasets such as ImageNet and whether it is robust to different image distributions.
Method
The study compares denoising autoencoders, including BEiT and SplitMask, with joint embedding methods while pre-training on smaller, varied, and target-task datasets.
Results
Denoising autoencoders show robust transfer with smaller or non-object-centric data, including competitive target-task pre-training and a +0.5 box AP gain from using COCO images alone.
Takeaways & Limitations
Large-scale datasets such as ImageNet are not necessary for self-supervised pre-training when using denoising autoencoders.
Takeaways & Limitations
Denoising autoencoding methods typically underperform instance discrimination methods such as DINO in linear probing, and BEiT showed numerical instability on sketch datasets.
Abstract
from arXiv · showhide
Pre-training models on large scale datasets, like ImageNet, is a standard practice in computer vision. This paradigm is especially effective for tasks with small training sets, for which high-capacity models tend to overfit. In this work, we consider a self-supervised pre-training scenario that only leverages the target task data. We consider datasets, like Stanford Cars, Sketch or COCO, which are order(s) of magnitude smaller than Imagenet. Our study shows that denoising autoencoders, such as BEiT or a variant that we introduce in this paper, are more robust to the type and size of the pre-training data than popular self-supervised methods trained by comparing image embeddings.We obtain competitive performance compared to ImageNet pre-training on a variety of classification datasets, from different domains. On COCO, when pre-training solely using COCO images, the detection and instance segmentation performance surpasses the supervised ImageNet pre-training in a comparable setting.
1. Introduction
The paper questions whether self-supervised pre-training requires large, curated datasets such as ImageNet. It argues that denoising autoencoders can use smaller and more varied data, including target-task images, without sacrificing competitive downstream performance.
- Motivation: High-capacity vision models often overfit on small or medium-sized datasets, motivating pre-training on large datasets before fine-tuning.This standard paradigm has achieved strong results but introduces differences between pre-training and target-task data.
- Motivation: Pre-training on ImageNet can introduce domain shift and supervision collapse, limiting alignment with downstream data and general-purpose feature learning.Supervised pre-training may focus representations on the pre-training labels while discarding information useful for other tasks.
- Research questions: The paper asks whether self-supervised pre-training needs millions of samples and whether it remains effective across different image distributions.A positive answer would support using smaller or out-of-domain datasets, including target-task training sets.
- Contributions: Denoising autoencoders are presented as more sample efficient than joint embedding techniques, enabling pre-training without relying on large-scale datasets such as ImageNet.The paper also evaluates whether these methods work directly on target-task data and on non-object-centric images.
- Contributions: Denoising autoencoders can be pre-trained on non-object-centric COCO images while retaining performance similar to ImageNet pre-training, unlike joint embedding techniques.This addresses robustness to the nature of pre-training data as well as its size.
2. Related Work
Related work spans denoising autoencoders, masked image modeling, instance discrimination, vision transformers, and studies of pre-training data. These approaches differ in their data requirements, transformations, and robustness to dataset size and distribution.
- Denoising autoencoders: Denoising autoencoders reconstruct corrupted inputs, including masked image patches, and have been explored through tasks such as in-painting, colorization, and patch de-shuffling.Masked Image Modeling adapts denoising ideas to vision by predicting missing patch information.
- Instance discrimination: Instance discrimination treats each image as its own class and learns representations by making transformed views similar while separating them from other images.These methods commonly use contrastive objectives and require many negatives through large batches or memory banks.
- Vision transformers: Vision transformers split images into patches and process the resulting embeddings as sequences, with early strong performance depending on very large pre-training collections.This motivated studying how alternative pre-training data regimes affect transfer.
- Pre-training data: Instance discrimination can learn from non-curated data, but prior work typically requires an order of magnitude more data than ImageNet.Weakly supervised pre-training on hashtag data likewise requires large amounts of data.
- Comparative studies: Table 1 compares transfer to iNaturalist-2019 across ImageNet subset sizes and COCO, reporting greater robustness for denoising autoencoders than for DINO or supervised pre-training.The comparison varies both the amount and the object-centric nature of pre-training data.
3. Analysis
The analysis tests how dataset size and image distribution affect self-supervised pre-training. Denoising autoencoders remain robust with fewer or non-object-centric images, while DINO degrades more under these changes.
- Dataset size: Denoising autoencoder pre-training is robust to reduced dataset size, unlike DINO, whose performance degrades on smaller datasets.
- Dataset size: 5% of ImageNet samples produced peak SplitMask transfer performance on iNaturalist-2019, while adding more samples provided no additional boost at fixed updates.
- Dataset size: +4 points over training from scratch was achieved using only 1,000 ImageNet samples, indicating strong sample efficiency for SplitMask.
- Training schedule: Nearly 3k pre-training epochs were crucial for strong performance with smaller subsets, although very long schedules caused slight overfitting, especially on Stanford Cars.
- Data distribution: On COCO, DINO dropped -8.3 versus full ImageNet pre-training, whereas SplitMask improved +0.7 and BEiT decreased only slightly.
- Tokenizers: Replacing BEiT’s DALL-E tokenizer with random projection, random patches, or k-means caused no significant accuracy degradation and improved relative runtime by 26%.
4. Methodology
SplitMask combines masked image modeling with contrastive matching across two independently encoded patch subsets. Its encoder processes observed patches, while a lightweight decoder reconstructs missing-patch targets and supplies global descriptors for matching.
- Split: SplitMask splits each image into disjoint patch subsets A and B, then processes them independently through a shared deep ViT encoder.
- Inpaint: A shallow decoder inpaints the missing subset through masked image modeling, using mask embeddings to represent missing-patch positions.
- Match: Global descriptors are average-pooled from decoder outputs, including observed and hallucinated patches, and matched across the two subsets.
- Encoder-decoder architecture: Unlike BEiT, the encoder processes only observed patches, while the decoder predicts visual words for masked patches using cross entropy.
- Encoder-decoder architecture: For n image patches, the encoder processes two sequences of size n/2 while the decoder processes two sequences of size n, keeping complexity similar to a standard ViT.
- Match: The descriptors are trained with a symmetric InfoNCE loss so representations remain consistent across different observed subsets.
5. Experiments
Experiments evaluate denoising autoencoder pre-training on target-task datasets spanning multiple sizes and domains, including classification, detection, instance segmentation, and semantic segmentation. Across these settings, target-data pre-training is competitive with or better than ImageNet pre-training, while very small datasets can overfit under long schedules.
- Target-task datasets span fine-grained classification, non-natural-image domains, COCO detection and instance segmentation, and ADE20k semantic segmentation.
- Training considerations: Long schedules caused overfitting on very small datasets such as Stanford Cars, while BEiT encountered numerical instability on clipart and sketch.Stanford Cars overfitting was observed beyond 5k epochs; BEiT failed to converge on clipart and sketch with long schedules such as 5000 epochs.
- Object detection and instance segmentation: +0.4 box AP resulted from pre-training a ViT-base BEiT model on COCO instead of ImageNet.
- Object detection and instance segmentation: +0.6 box AP for ViT-small and +0.3 mask AP for ViT-base were achieved by SplitMask over BEiT; the comparable setting yielded +1.1 box AP without ImageNet.
- Semantic segmentation: 20k ADE20k images yielded semantic-segmentation performance matching BEiT pre-training on ImageNet and only marginally below supervised ImageNet pre-training.
- Image classification: Target-data BEiT pre-training often outperformed ImageNet BEiT, including +1.1% accuracy on Stanford Cars and +0.7 accuracy on Food101 with ViT-small and ViT-base, respectively.Stanford Cars contains only 8k images, while Food101 is more than 10x smaller than ImageNet.
- Image classification: SplitMask further improved multiple classification results, including +3.0 accuracy on iNaturalist 2018 with ViT-base and +2.7% top-1 accuracy over supervised ImageNet pre-training on iNaturalist.For clipart, painting, and sketch, SplitMask outperformed ImageNet-pre-trained BEiT across ViT-S datasets, while supervised pre-training remained best for ViT-S and ViT-B.
6. Conclusion
The conclusion asks whether ImageNet-scale data are necessary for self-supervised pre-training and reports that denoising autoencoders remain effective with smaller and non-object-centric datasets. Direct target-data pre-training, including on COCO, supports competitive downstream performance without ImageNet.
- Smaller pre-training datasets caused no big performance drop for denoising autoencoders, unlike instance-discrimination self-supervision or supervised pre-training.
- Pre-training directly on target-task data achieved +0.5 box AP using only COCO images, which are 10x smaller than ImageNet.
A. SplitMask vs BEiT
Ablations show that SplitMask benefits from combining masked-image modeling with contrastive matching, while the full model improves over BEiT but remains weaker than instance-discrimination methods for linear probing.
- The global contrastive loss alone produced very weak performance, because it supplies no local patch-representation signal under 50% masking.
- +0.8 was the full SplitMask improvement over BEiT in the ablation, using both MIM and contrastive objectives.
- SplitMask had stronger linear-probing performance than BEiT, but both denoising autoencoders remained relatively weak versus instance-discrimination methods on this benchmark.Instance-discrimination methods' final layers are described as more aligned with classification.
- SplitMask added negligible computing overhead compared with BEiT, with only marginally higher wall-clock training time.
B. Encoder-Decoder vs BEiT
The encoder-decoder design separates general-purpose image encoding from solving the masked-image-modeling pretext task. This improves transferability of later-layer representations relative to BEiT.
- The encoder-decoder design encourages decoupling downstream-useful image features from features specific to the MIM pretext task.
- Because the encoder lacks the mask token, it must provide informative representations to the decoder rather than solve the pretext task independently.
- SplitMask improves later-layer linear-probing transferability, whereas BEiT shows a stronger drop in later-layer performance.
C. Overfitting during pre-training
On very small datasets, extending self-supervised pre-training too far can harm downstream fine-tuning performance. Stanford Cars performance drops severely beyond 5k pre-training epochs.
- C. Overfitting during pre-training: Pre-training beyond 5k epochs causes a severe drop in Stanford Cars finetuning performance.The authors caution that schedules matching ImageNet’s update count would require approximately 45k Stanford Cars epochs.
- C. Overfitting during pre-training: 45k epochs would be the Stanford Cars-equivalent schedule under the 300-epoch ImageNet update assumption.
- C. Overfitting during pre-training: Figure 6 plots Stanford Cars finetuning performance against the number of pre-training epochs using Stanford Cars images.
D. Image Classification Finetuning
The classification finetuning setup is documented through dataset-specific hyperparameters. These settings are presented in Table 9.
- D. Image Classification Finetuning: Dataset-specific finetuning hyperparameters are detailed in Table 9.
- D. Image Classification Finetuning: The finetuning procedure refers readers to Table 9 for the settings applied to each classification dataset.