PaperlayerView exact revision on arXiv

Source-linked AI summary

Scaling and Benchmarking Self-Supervised Visual Representation Learning

Priya Goyal, Dhruv Mahajan, Abhinav Gupta, Ishan Misra

arXiv:1905.01235v2cs.CVcs.AIcs.LG

TL;DR

The paper asks whether self-supervised visual representations continue improving when scaled beyond ImageNet-sized data and whether they can match supervised pre-training. It scales Jigsaw and Colorization across data, model capacity, and problem complexity, and evaluates them on a nine-task benchmark. Scaling matches or exceeds supervised performance on several tasks, but semantic representation quality remains limited.

  • Problem

    Self-supervised learning had not been adequately tested at scales beyond ImageNet, despite requiring no manual labels and potentially using billions of images.

  • Method

    The paper scales Jigsaw and Colorization along data size, model capacity, and problem complexity, evaluating representations with a consistent benchmark across 9 tasks.

  • Results

    Scaling outperforms ImageNet supervised pre-training on surface normal estimation and navigation, matches it on VOC07 detection, and leaves a significant semantic-classification gap.

  • Takeaways & Limitations

    Self-supervised scaling is crucial, but current pretext tasks remain insufficiently hard and do not yet produce effective high-level semantic representations.

  • Takeaways & Limitations

    The paper reports that self-supervised representations perform worse than supervised counterparts on image classification and low-shot classification.

Abstract

from arXiv · show

Self-supervised learning aims to learn representations from the data itself without explicit manual supervision. Existing efforts ignore a crucial aspect of self-supervised learning - the ability to scale to large amount of data because self-supervision requires no manual labels. In this work, we revisit this principle and scale two popular self-supervised approaches to 100 million images. We show that by scaling on various axes (including data size and problem 'hardness'), one can largely match or even exceed the performance of supervised pre-training on a variety of tasks such as object detection, surface normal estimation (3D) and visual navigation using reinforcement learning. Scaling these methods also provides many interesting insights into the limitations of current self-supervised techniques and evaluations. We conclude that current self-supervised methods are not 'hard' enough to take full advantage of large scale data and do not seem to learn effective high level semantic representations. We also introduce an extensive benchmark across 9 different datasets and tasks. We believe that such a benchmark along with comparable evaluation settings is necessary to make meaningful progress. Code is at: https://github.com/facebookresearch/fair_self_supervision_benchmark.

1. Introduction

The paper argues that self-supervised learning should exploit scalable, label-free training, and studies scaling data, model capacity, and pretext-task complexity. It introduces a nine-task benchmark and finds strong transfer on nonsemantic tasks while semantic classification still trails supervised pre-training.

  • Supervised data is becoming a bottleneck because performance grows only log-linearly with labeled-data volume.
  • Self-supervised learning trains models on auxiliary pretext tasks whose ground truth is available without manual labels.
  • The study scales Jigsaw and Colorization along data size, model capacity, and problem complexity, including up to 100× more data.
  • The benchmark evaluates representations across 9 tasks spanning semantic classification and detection, 3D geometry, and visual navigation.
  • Scaling enables self-supervised models to outperform ImageNet supervised pre-training on surface normal estimation and navigation, while semantic classification remains significantly behind.

2. Related Work

Related work distinguishes generative and discriminative unsupervised representation learning. This paper focuses on image-based discriminative self-supervision, which automatically generates labels from visual data or its structure.

  • Generative methods model the data distribution by reconstructing inputs, estimating latent variables, or using adversarial training.
  • Discriminative self-supervised methods automatically generate labels from data, using domain-agnostic cues or structural properties such as image spatial structure.
  • The paper scales image-based methods because they are easy to implement and exploit properties including spatial structure, color, illumination, and rotation.

3. Preliminaries

The paper studies two label-free image-based pretext tasks: Jigsaw puzzle solving and colorization. Both convert visual prediction problems into classification-based training objectives.

  • Both studied approaches learn representations without using supervised labels.
  • Jigsaw Self-supervision: Jigsaw divides an image into N = 9 patches, shuffles them, and trains a ConvNet to predict the applied permutation.
  • Jigsaw Self-supervision: Jigsaw reduces permutation prediction to classification over a selected set P of |P| classes.
  • Colorization Self-supervision: Colorization predicts quantized ab colors Y from lightness X in CIE Lab space, using Q = 313 color bins and a soft-encoded target ZK.

4. Scaling Self-supervised Learning

The paper studies scaling self-supervised learning across data size, model capacity, and pretext-task complexity using controlled transfer evaluations. Performance generally improves across these axes, but higher-capacity models benefit more from larger data and harder tasks.

  • Scaling setup: The investigation varies pre-training data size, model capacity, and pretext-task complexity to test whether their benefits are complementary.The setup uses fixed ConvNet features and linear SVM evaluation on VOC07.
  • Scaling the pre-training data size: Increasing YFCC pre-training data improves transfer performance for Jigsaw and Colorization on both ResNet-50 and AlexNet, while Jigsaw saturates log-linearly from 1M to 100M images.The experiments use randomly sampled YFCC-[1, 10, 50, 100] million-image subsets with the data domain fixed.
  • Scaling model capacity: The performance gap between AlexNet and ResNet-50 increases with pre-training dataset size, suggesting higher-capacity models better exploit larger datasets.Both architectures are trained on the same pre-training subsets.
  • Scaling problem complexity: For Jigsaw, increasing permutation-set size improves transfer performance, with a 5 point mAP gain for ResNet-50 versus 1.9 points for AlexNet.The pre-training data is fixed at YFCC-1M to isolate problem complexity.
  • Scaling problem complexity: Jigsaw benefits more from increasing problem complexity than Colorization, which varies by approximately 2 mAP points across K values.The paper associates Jigsaw’s gains with increased spatial structure, whereas Colorization emphasizes semantic-category and color relations.
  • Putting it together: The three scaling axes are complementary: performance increases with data size, model capacity, and problem complexity, including harder tasks at 100M images for ResNet-50.Problem-complexity gains are nearly negligible for AlexNet but significantly higher for ResNet-50.

5. Pre-training and Transfer Domain Relation

The paper examines how pre-training domain affects transfer to object-centric VOC07 and scene-centric Places205 classification. Transfer performance depends strongly on domain similarity: ImageNet is competitive for VOC07, while YFCC is competitive for Places205.

  • Evaluation setup: The study varies ImageNet and YFCC pre-training domains and evaluates fixed feature representations on VOC07 and Places205 classification.ImageNet is used without labels to isolate the effect of pre-training domain.
  • Domain relation: On VOC07 classification, Jigsaw pre-training on ImageNet-22k with 14M images transfers as well as pre-training on YFCC-100M with 100M images.VOC07 is the object-centric transfer task in this comparison.
  • Domain relation: On Places205 classification, Jigsaw pre-training on YFCC-1M with 1M images transfers as well as pre-training on ImageNet-22k with 14M images.Places205 is the scene-centric transfer task.
  • Domain relation: Colorization shows a similar pattern: ImageNet pre-training benefits VOC07 classification more than YFCC pre-training.The paper relates this pattern to the closer object-centric domain match between ImageNet and VOC07.

6. Benchmarking Suite for Self-supervision

The benchmark evaluates self-supervised representations across nine diverse tasks using consistent transfer procedures and limited fine-tuning. Results vary by task: semantic classification remains behind supervision, while detection, navigation, and surface-normal estimation are competitive or better.

  • Benchmark design: The suite evaluates representations on nine tasks spanning semantic classification and detection, scene geometry, and visual navigation.The benchmark emphasizes generalization across tasks with limited supervision and fine-tuning.
  • Benchmark design: Consistent hyperparameters and evaluation settings are intended to isolate representation quality from transfer optimization differences.The authors identify prior inconsistencies involving crops, weight rescaling, preprocessing, and fine-tuning schedules.
  • Benchmark design: A common pipeline pre-trains with Jigsaw or Colorization, extracts features from multiple layers, and evaluates them on supervised transfer tasks.For ResNet-50, features are extracted from the last layer of each residual stage.
  • Image classification: Self-supervised features show a significant accuracy gap versus supervised methods on semantic image classification despite scaling efforts.The authors attribute this comparison to both settings solving semantic image classification problems.
  • Low-shot image classification: At k=1 labeled example per class, the Places205 gap is 30 points, compared with 20 points in the full-shot setting.These results use the best-performing ResNet-50 layer, res4, on VOC07 and Places205.
  • Visual navigation: On navigation, Jigsaw ImageNet res3 features achieve higher and more sample-efficient training rewards than supervised features, while res4 and res5 perform similarly.Self-supervised ImageNet pre-training also outperforms YFCC pre-training in this task.
  • Object detection: Self-supervised initialization is competitive with ImageNet pre-training for VOC07 detection, with a very low performance gap even when fewer parameters are fine-tuned.The detection results use the reported common training schedule and compare frozen-convolution-body settings.
  • Surface normal estimation: For surface-normal estimation, Jigsaw YFCC-100M features outperform ImageNet-1k and Places205 supervised models across all reported metrics.The reported gain over Places205 supervision is 5 points on the percentage of pixels within 11.5° of ground truth.

7. Legacy Tasks and Datasets

The legacy-task evaluation reports full fine-tuning and ImageNet classification results for comparison, while excluding ImageNet linear classification from the benchmark because same-domain evaluation is biased.

  • Legacy evaluations: Legacy evaluations include full fine-tuning, which measures both representation quality and initialization or optimization effects.The authors report full fine-tuning results for VOC07 classification and object detection.
  • Legacy evaluations: With full fine-tuning, self-supervised initialization matches supervised initialization on VOC07 and VOC07+12 object detection.ImageNet self-supervised pre-training performs better than YFCC pre-training in this evaluation.
  • ImageNet classification: ImageNet linear classification is excluded because supervised baselines are trained on ImageNet-1k and many self-supervised methods are evaluated in the same pre-training domain.The authors state that identical images for pre-training and transfer accentuate evaluation bias.
  • Evaluation caveats: The reported legacy comparisons use differing fine-tuning schedules where indicated, limiting direct interpretation across entries.Table 4 marks results using a different fine-tuning schedule, while the surrounding discussion documents schedule differences.

8. Conclusion

The paper studies how data size, model capacity, and problem complexity affect self-supervised transfer, then evaluates the resulting representations across nine diverse tasks. Scaling improves several transfer outcomes, but semantic classification still trails supervised pre-training.

  • Scaling conclusions: The study scales Jigsaw and Colorization along data size, model capacity, and problem complexity, finding complementary improvements across these axes.Transfer performance increases log-linearly with data size, while representation quality improves with capacity and problem complexity.
  • Benchmark conclusions: The benchmark contains nine diverse tasks and supports evaluation of representation transfer beyond a single classification setting.The tasks cover classification, detection, surface-normal estimation, and navigation.
  • Cross-task results: Self-supervised representations outperform supervised baselines on surface-normal estimation and perform competitively or better on navigation.These comparisons are part of the paper’s benchmark conclusions.
  • Cross-task results: Self-supervised representations match supervised object-detection baselines even with little fine-tuning.The conclusion summarizes the detection result alongside the geometry and navigation findings.
  • Remaining limitations: Self-supervised representations perform worse than supervised counterparts on image classification and low-shot classification.The authors state that definitively surpassing supervised pre-training remains unresolved.
  • Remaining limitations: The authors argue that future pretext tasks should be complex enough to exploit large-scale data and increased model capacity.They characterize scaling self-supervision as crucial but insufficient for definitively surpassing supervised pre-training.

Supplementary Material

The supplementary material supplies implementation and evaluation details for the paper’s networks, transfer tasks, pre-training, benchmark experiments, legacy evaluations, and additional scaling analyses.

  • Supplementary contents: Section A provides architecture details for all self-supervised networks.
  • Supplementary contents: Section B provides architecture details for all transfer tasks.
  • Supplementary contents: Section C lists hyperparameters for self-supervised pre-training.
  • Supplementary contents: Section D lists hyperparameters for the benchmark tasks in Section 6.
  • Supplementary contents: Section E lists hyperparameters for the legacy tasks in Section 7.
  • Supplementary contents: Section F reports additional ways of increasing problem complexity for self-supervised tasks.
  • Supplementary contents: Section G reports additional results on object detection, surface-normal estimation, and image classification.

A. Model architectures for pretext tasks

The pretext-task architectures use established AlexNet and ResNet-50 setups for Jigsaw and Colorization, with architecture details provided separately for each combination.

  • AlexNet: AlexNet Jigsaw follows the architecture from [52].The full specification is referenced in Table 10.
  • AlexNet: AlexNet Colorization follows the architecture setup from [79], with SpatialBN and ReLU after every convolutional layer.The complete configuration is provided in Table 11.
  • AlexNet: The supervised AlexNet baseline uses the exact CaffeNet BVLC architecture and directly uses pretrained model weights.The supervised model was not trained separately to avoid differences in results.
  • ResNet-50: ResNet-50 Jigsaw uses an N-way Siamese ConvNet with shared weights, followed by concatenation of the branches.The architecture description specifies one Siamese branch and the remaining concatenated branch.
  • ResNet-50: ResNet-50 Colorization follows the architecture described in Table 13 and closely follows the setup in A.4.
  • ResNet-50: The remaining ResNet architecture follows the same architecture as [29].

B. Model architectures for Transfer tasks

Transfer-task evaluation uses task-specific architectures, with convolutional features downsampled to a 9k-dimensional representation in the described setups.

  • Evaluation setup: The section defines exact architectures for the evaluation tasks and benchmark suite.These architectures correspond to the benchmark procedures described in Section 6.
  • AlexNet: AlexNet evaluation uses SpatialBN and ReLU after every convolutional layer, then downsamples convolutional features to dimension 9k.The complete architecture is listed in Table 14.
  • AlexNet: Another AlexNet transfer setup downsamples convolutional layers with average pooling to produce a 9k-dimensional feature map.The exact architecture is listed in Table 15.
  • AlexNet: One evaluation setup follows the exact CaffeNet BVLC architecture and directly uses pretrained model weights.Exact architecture details are referenced in [34].
  • Transfer configurations: Tables 16 and 17 provide the exact architectures for the corresponding transfer configurations.
  • ResNet-50: The ResNet transfer architecture strictly follows the architecture in [29].

C. Pre-training Hyperparameters

Pre-training uses method- and architecture-specific schedules across ImageNet and YFCC datasets, while evaluation emphasizes standardized classifiers and matched settings.

  • Overview: Pre-training hyperparameters are specified for Jigsaw and Colorization using both AlexNet and ResNet-50.The schedules cover multiple ImageNet and YFCC dataset sizes.
  • Jigsaw: AlexNet Jigsaw follows [52] with image resizing, random cropping, grayscale and color projection augmentation, and 3x3 patch construction.Each patch is randomly cropped from an 85x85 cell after resizing the image to 256 on its shorter side.
  • Colorization: Colorization converts images to Lab space, uses 313 bins and provided priors, applies random cropping and horizontal flipping, and specifies distinct schedules by dataset.The same core preprocessing is described for AlexNet and ResNet-50 Colorization.
  • Evaluation: For VOC07 and COCO2014, frozen features are normalized and evaluated with per-class linear SVMs selected by 3-fold cross-validation.The setup uses 20 SVMs for VOC07 and 80 for COCO2014, with mean average precision reported afterward.
  • Evaluation: Dual Coordinate Descent in LIBLINEAR was chosen because SGD classifier results varied substantially with optimization hyperparameters and could converge slowly.The authors use this setting for smaller VOC07 and COCO2014 datasets as a robust comparison of image representations.

G.7. Faster R-CNN results on VOC07 and VOC07+12

The supplied passages define architectures and evaluation configurations for VOC07 and VOC07+12 detection, alongside related classification, surface-normal, and problem-complexity tables.

  • Architectures: The transfer-task architecture tables describe AlexNet and ResNet-50 configurations, including 9k-dimensional downsampled feature maps.Tables 16–17 are identified as ResNet-50 transfer architectures, while Tables 14–15 cover AlexNet transfer architectures.
  • Other tasks: Table 23 covers surface normal estimation on NYUv2 with ResNet-50 trained from res5 onward while the lower convolutional body is frozen.
  • Detection: Frozen-conv-body detection uses Faster R-CNN with ResNet-50-C4 on VOC07 and VOC07+12.Table 24 reports detection mAP under this configuration.
  • Detection: Table 25 reports VOC07 and VOC07+12 detection mAP with full fine-tuning using Faster R-CNN with ResNet-50-C4.The passage also states that the conv body is frozen for all models, creating a configuration inconsistency in the supplied text.
  • Problem complexity: Increasing Jigsaw patches from 9 to 16 at fixed |P| = 2000 does not improve VOC07 linear-SVM performance.The table frames this as a problem-complexity experiment.
  • Problem complexity: Increasing Colorization color bins changes the prediction problem's complexity and is evaluated on Places205 with fixed-feature linear classifiers.The supplied passage specifies the evaluation design but not the resulting comparison.
Loading 1905.01235v2…