PaperlayerView exact revision on arXiv

Source-linked AI summary

When Does Contrastive Visual Representation Learning Work?

Elijah Cole, Xuan Yang, Kimberly Wilber, Oisin Mac Aodha, Serge Belongie

arXiv:2105.05837v2cs.CVcs.LG

TL;DR

The paper asks how well contrastive self-supervised learning works beyond ImageNet, where evidence and best practices are limited. It evaluates multiple datasets and factors, finding modest benefits from more data, weaker transfer across domains, sensitivity to image quality, and a supervised advantage on fine-grained tasks.

  • Problem

    Although self-supervised representations are competitive with supervised ones on many tasks, their performance beyond standard datasets such as ImageNet is not well understood.

    Source: §1 · p. 1

  • Method

    The authors quantitatively evaluate contrastive self-supervised learning across four diverse large-scale datasets and vary data quantity, domain, quality, and task granularity.

    Source: §1 · pp. 1–2

  • Results

    Beyond 500k pretraining images, benefits are modest; cross-domain data does not yield more general representations, corruption has disparate effects, and supervised learning retains an edge on fine-grained tasks.

    Source: §1 · p. 2, §5 · p. 8

  • Takeaways & Limitations

    Current SSL methods show limits beyond ImageNet, particularly for fine-grained classification, and the findings point to areas for further research beyond top-1 ImageNet evaluation.

    Source: §5 · p. 8

    The experiments mainly use SimCLR, cover classification tasks, and consider datasets only up to roughly ImageNet scale.

    Source: §5 · p. 8

Abstract

from arXiv · show

Recent self-supervised representation learning techniques have largely closed the gap between supervised and unsupervised learning on ImageNet classification. While the particulars of pretraining on ImageNet are now relatively well understood, the field still lacks widely accepted best practices for replicating this success on other datasets. As a first step in this direction, we study contrastive self-supervised learning on four diverse large-scale datasets. By looking through the lenses of data quantity, data domain, data quality, and task granularity, we provide new insights into the necessary conditions for successful self-supervised learning. Our key findings include observations such as: (i) the benefit of additional pretraining data beyond 500k images is modest, (ii) adding pretraining images from another domain does not lead to more general representations, (iii) corrupted pretraining images have a disparate impact on supervised and self-supervised pretraining, and (iv) contrastive learning lags far behind supervised learning on fine-grained visual classification tasks.

1. Introduction

The paper asks when contrastive self-supervised representations work beyond ImageNet and evaluates data quantity, domain, quality, and task granularity across diverse datasets. It reports modest returns from more than 500k pretraining images, stronger performance from same-domain data, and a widening supervised advantage on fine-grained tasks.

Introduction

  • The study evaluates how data quantity, pretraining domain, image quality, and downstream task granularity shape self-supervised representation learning.It frames these questions as relevant both to understanding SSL and to domain experts with limited resources considering real-world applications.Source: §1 · pp. 1–2, Figure 1 · p. 1
  • Beyond 500k images, additional pretraining data brings modest benefits, while same-domain data is more effective and combining datasets does not yield large gains.The introduction reports that halving an ImageNet-scale unlabeled dataset from 1M to 500k images reduces downstream accuracy by only 1–2%; combining datasets or features does not significantly improve performance.Source: §1 · p. 2
  • As classification concepts become finer-grained, contrastive self-supervised performance falls further behind supervised baselines.The paper suggests that contrastive loss may yield coarse-grained features insufficient for fine-grained tasks.Source: §1 · p. 2
  • Pretraining-image corruption affects supervised and self-supervised learning differently, with self-supervised representations notably sensitive to image resolution.The introduction reports disparate effects across the two training approaches and highlights resolution sensitivity for self-supervised learning.Source: §1 · p. 2

2. Related Work

Prior work developed self-supervised methods and studied their ImageNet performance, dataset scale, and transfer across domains. This paper builds on that work with controlled comparisons across datasets and examines data quality and task granularity as well.

SSL for visual representations

  • Early self-supervised methods solved hand-designed pretext tasks such as rotation prediction and jigsaw puzzles, while newer contrastive approaches distinguish matching from non-matching pairs of augmented input images [30] [39].SimCLR identifies the matching image using cross-entropy loss.Source: §2 · p. 2
  • Other approaches use learning-to-rank [56], feature prediction, or exponential-moving-average targets; this study focuses on self-supervised visual classification.The study excludes supervised contrastive learning, language [42] and audio contrastive learning, and large-scale weak supervision.Source: §2 · p. 2

SSL beyond ImageNet

  • ImageNet has been a central SSL benchmark, and prior work evaluated ImageNet-pretrained representations on species classification, segmentation, and scene understanding.Source: §2 · p. 2
  • Studies beyond ImageNet report domain-specific gains and often find the best representations come from pretraining on data similar to the downstream task.Other work found animal- and plant-image contrastive pretraining did not outperform ImageNet pretraining for bird classification, while multi-dataset pretraining had little impact on detection and segmentation unless synthetic data was used.Source: §2 · p. 2
  • Pretraining on JFT-300M or YFCC100M did not improve over ImageNet pretraining in the standard regime [50].Source: §2 · p. 3

Analysis of SSL

  • This study uses controlled, like-for-like comparisons to separate dataset-specific effects from general patterns, extending prior work on data quantity [61] to domain, quality, and task granularity.The authors report that current SSL methods can perform much worse than supervised baselines on non-ImageNet domains, including fine-grained classification.Source: §2 · p. 3

3. Methods

The study compares four large-scale datasets with varied visual properties, then evaluates SimCLR representations using linear classifiers or fine-tuning under standard settings.

Datasets

  • Experiments use ImageNet (1.3M images, 1k classes), iNat21 (2.7M, 10k), Places365 (1.8M, 365), and GLC20 (1M, 16), spanning varied image and category types.The datasets include curated and in-the-wild images, fine- and coarse-grained categories, and object-centric images and scenes.Source: §3 · p. 3

Fixed-size subsets

  • For size-controlled experiments, the authors create nested, uniformly random subsets of 1M, 500k, 250k, 125k, and 50k images, fixed across experiments.Except in Figures 2 and 3, supervised training uses the full dataset, and each dataset's test set remains the same across training subsets.Source: §3 · p. 3

Training details

  • Experiments use a ResNet-50 backbone, primarily SimCLR, and evaluate pretrained features with linear classifiers or end-to-end fine-tuning.Unless specified otherwise, hyperparameters follow; MoCo and BYOL are also considered in Figure 3.Source: §3 · p. 3

4. Experiments

The experiments examine contrastive self-supervised learning across data quantity, domain, quality, and task granularity, using SimCLR representations and supervised-from-scratch comparisons.

4.1. Data quantity

  • The 500k-to-1M pretraining-image accuracy gap is typically under 1–2%, while 50k-to-250k often costs more than 10% top-1 accuracy.For a 1M-image dataset, using 500k images trades a small accuracy decrease for a 50% reduction in pretraining time.Source: §4.1 · p. 4
  • Fine-tuning SimCLR beats training from scratch with 10k or 50k labels, yet iNat21 retains an approximately 30% supervised gap even in the high-supervision regime.The authors identify iNat21 as a useful SSL benchmark; BYOL still leaves an approximately 25% gap to supervised performance there.Source: §4.1 · p. 4

4.2. Data domain

  • In-domain pretraining beats cross-domain pretraining in every dataset column, including a 12% boost on iNat21 over the best cross-domain alternative.The comparison uses million-image pretraining sets and full training sets for linear evaluation.Source: §4.2 · p. 4, Table 1 · p. 5
  • At equal pretraining-set sizes, pooled datasets underperform in-domain sets, and adding other-domain images typically hurts relative to in-domain pretraining.The 500k in-domain sets beat equally sized pooled sets; the 250k in-domain row beats the 1M pooled set and almost all 500k pooled sets.Source: Table 2 · p. 6, §4.2 · p. 6
  • +4.7% adding ImageNet Sup. to iNat21 SimCLR, but adding iNat21 Sup. to ImageNet SimCLR changes performance by -4.2%; SSL-only fusion shifts are smaller (+1.4% and -0.6%).The authors interpret these results as consistent with dataset semantics mattering for self-supervised learning.Source: Table 3 · p. 6, §4.2 · p. 6

4.3. Data quality

  • Downsampling ImageNet pretraining images reduces SimCLR accuracy by around 15% at 2x and 34% at 4x, while JPEG and salt-and-pepper noise have milder effects.The downsampling impact is much less dramatic for supervised pretraining.Source: Figure 4 · p. 7, §4.3 · p. 7

4.4. Task granularity

  • The supervised–SimCLR performance gap grows as labels become finer, with the largest gap on iNat21 and the smallest on Places365.The study uses hierarchy distance from the root as a granularity proxy and re-evaluates predictions at coarser levels without retraining.Source: §4.4 · p. 7, Figure 5 · p. 8

5. Conclusion

The experiments find diminishing returns from larger pretraining sets, substantial variation across domains and image quality, and a persistent supervised advantage on fine-grained classification. The authors also note that the evidence is limited to classification, datasets up to roughly ImageNet scale, and mainly one self-supervised method.

5. Conclusion

  • Beyond 500k pretraining images, returns diminish severely; peak supervised performance still requires hundreds of thousands of labeled images to surpass self-supervised representations.The conclusion also notes that the best self-supervised representations remain much worse than peak supervised performance without this scale of labeled data.Source: §5 · p. 8
  • Self-supervised pretraining on 1M images from different domains yields representations with different capabilities, and simple dataset-combination methods do not bring large gains.The supplied conclusion describes different capabilities across domains without specifying which representations excel on which tasks.Source: §5 · p. 8
  • Image resolution is critical for contrastive learning, while some image corruptions make self-supervised representations unusable and others have almost no impact.Source: §5 · p. 8
  • Supervised pretraining retains a substantial advantage over self-supervised learning on fine-grained classification.Source: §5 · p. 8
  • The study mainly evaluates one self-supervised method on classification tasks and datasets up to roughly ImageNet scale, leaving other methods, tasks, and larger scales for further study.Source: §5 · p. 8

A.1. How does task granularity affect different self-supervised learning methods?

On iNat21, all three self-supervised methods lose performance as evaluation becomes finer-grained, with BYOL still trailing supervision. Larger models can gain more from smaller pretraining-set increases, but gains from 500k to 1M images remain small.

A.1. How does task granularity affect different self-supervised learning methods?

  • 20% top-1 accuracy separates BYOL from fully supervised performance on iNat21, and all three self-supervised methods drop rapidly as evaluation becomes finer-grained.Source: §A.1 · p. 12
  • BYOL performs significantly better than SimCLR across iNat21 granularity levels, while MoCo is slightly worse than SimCLR.Source: §A.1 · p. 12

A.2. Do larger models scale better in terms of pretraining set size?

  • Gains from increasing pretraining data from 500k to 1M images are <2% for ResNet-34, ResNet-50, and ResNet-101.At 1M images, the gap between ResNet-101 and ResNet-50 is very small, although ResNet-101 gains more with each increase in set size.Source: §A.2 · p. 12
  • From 125k to 250k pretraining images, ResNet-50 and ResNet-101 gain significantly more than ResNet-34.Source: §A.2 · p. 12

A.3. Does semantic similarity explain patterns in self-supervised performance?

Across taxonomic classes, in-domain contrastive pretraining improves many classes but not all, and semantic overlap may help explain which classes benefit. The appendix also reports that resizing can shift accuracy and that class difficulty transfers differently across datasets.

A.3. Does semantic similarity explain patterns in self-supervised performance?

  • ImageNet SimCLR outperformed iNat21 SimCLR on all but 80 categories, 68 of which were animals or plants; Places365 SimCLR approached no iNat21 taxonomic class.Figure A3 compares top-1 linear classification accuracy across eight taxonomic classes.Source: §A.3 · pp. 12–13, Figure A3 · p. 13
  • In-domain SimCLR improved about 60% of classes, with iNat21's most improved classes all plants and many harmed classes resembling ImageNet categories.The authors say this pattern is consistent with semantic similarity partly governing performance, though patterns are less clear for Places365.Source: §A.3 · p. 13, Figure A4 · p. 13

A.4. Is SimCLR overly tuned for ImageNet?

  • In-domain SimCLR outperformed ImageNet SimCLR on iNat21, Places365, and GLC20 despite possible ImageNet-specific tuning.The authors conclude that any such tuning effect appears overwhelmed by domain similarity.Source: §A.4 · p. 13

A.5. What is the effect of native image resolution?

  • 1–2% top-1 accuracy can shift when pretraining uses resized rather than original ImageNet and iNat21 images.Images are resized to 224×224 after crops are selected; Table A2 analyzes initial resizing to a 256-pixel short side.Source: §A.5 · pp. 13–14, Table A2 · p. 14

A.6. Is class difficulty preserved between different representations?

  • Class difficulty was not preserved between ImageNet and iNat21 SimCLR representations, but was largely preserved between ImageNet and Places365 representations.The overall patterns held when tracking easy and hard classes in either direction between representations.Source: §A.6 · p. 14

A.7. What is the effect of within-dataset diversity?

A small within-dataset diversity experiment found opposite performance changes for Places365 and iNat21 under the authors' class-count proxy. Qualitative examples distinguish domain overlap from the more distinct GLC20 images.

A.7. What is the effect of within-dataset diversity?

  • The authors distinguish mild within-dataset class redistribution from cross-domain mixing and suggest excessive diversity may make contrastive pretraining too easy; this small one-dataset experiment warrants caution.The proposed explanation is that more homogeneous data may yield finer-grained features, potentially relevant to iNat21 but not Places365.Source: §A.7 · pp. 14–15
  • The study evaluates how image diversity within a dataset affects linear evaluation of self-supervised representations, using 100k ImageNet images distributed over different numbers of classes.Source: Table A3 · p. 15

B. Qualitative Examples

  • Some ImageNet examples are semantically similar to iNat21 and Places365 examples, whereas GLC20 images are distinct; the listed corruptions can considerably affect representation quality.The corruption examples are described as subjectively mild.Source: B. Qualitative Examples · p. 15

C.1. Datasets

The paper uses iNat21 as a fine-grained species-classification dataset and a US subset of GLC20 as a noisy, multi-label land-cover task.

C.1. Datasets

  • iNat21 has 2.7M training images across 10k species and an approximately balanced training set.Its 100k official validation images are evenly sampled from each species and used as the test set.Source: §C.1 · p. 15
  • The GLC20 subset contains 1M US images across 16 land-cover classes, with binarized labels treated as multi-label classification.The authors note that labels are noisy and primarily use GLC20 as a pretraining set.Source: §C.1 · p. 16
  • GLC20's roughly 27k-image official validation set is held out in spatial blocks to mitigate spatial autocorrelation.The dataset is designed for location-based species prediction and land-cover classification.Source: §C.1 · p. 16

C.2. Training hyperparameters

The appendix specifies pretraining and supervised-training settings, label taxonomies, and the augmentations used for contrastive pretraining and supervised models.

C.2. Training hyperparameters

  • SimCLR, MoCo, and BYOL pretraining run for 1000 epochs on 16 TPUs, with shared augmentation choices for comparability.SimCLR and BYOL use batch size 4096; MoCo uses 1024, while SimCLR uses a two-layer, 128-dimensional projection head and temperature τ = 0.1.Source: §C.2 · p. 16
  • Linear classifiers train for 90 epochs with SGD and Nesterov momentum; fine-tuning and training from scratch use separate stated settings.Fine-tuning and scratch training use 90 epochs, with different learning-rate schedules and augmentation or regularization choices.Source: §C.2 · p. 16

C.3. Taxonomies

  • ImageNet and iNat21 use hierarchical taxonomies, while Places365 has three levels from scene categories to indoor or outdoor groupings.iNat21 ranges from 10k species to three kingdoms; ImageNet uses WordNet, and Places365's hierarchy has three depth levels.Source: §C.3 · pp. 16–17

C.4. Augmentations

  • SimCLR uses horizontal flips, random crops, and color jitter; fine-tuning and scratch training use flips and crops, while linear classifiers use no augmentation.The crop covers 8%–100% of an image with aspect ratio 3/4–4/3; color-jitter strength is 1.0.Source: §C.4 · p. 17

C.5. Corruptions

The section specifies the image corruptions used to study artificially degraded pretraining data, including resizing, downsampling, JPEG compression, and salt-and-pepper noise.

C.5. Corruptions

  • Resizing sets the shorter image side to 256 pixels while preserving aspect ratio, enabling comparisons that control for image size.The implementation uses PIL.Image.resize with its default nearest-neighbor interpolation.Source: §C.5 · p. 17
  • Resize-and-downsample first resizes the shorter side to 256 pixels, then downsamples by 2x or 4x before upsampling by the same factor.The initial resize accounts for differences in dataset image sizes, which affect how strongly a fixed-factor downsampling changes an image.Source: §C.5 · p. 17
  • JPEG compression uses quality 10, while salt-and-pepper noise independently corrupts each channel pixel with probability 1/100, setting it to 0 or 1 equally often.JPEG compression uses PIL.Image.save.Source: §C.5 · p. 17
Loading 2105.05837v2…