How Useful is Self-Supervised Pretraining for Visual Tasks?

Alejandro Newell; Jia Deng

Source-linked AI summary

How Useful is Self-Supervised Pretraining for Visual Tasks?

Alejandro Newell, Jia Deng

arXiv:2003.14323v1Mar 31, 2020cs.CVcs.LG

TL;DR

Self-supervised pretraining is difficult to evaluate for practical use because its utility may change with label availability, task, and data properties. The paper builds a controlled synthetic benchmark to measure these factors and finds that benefits are strongest with few labels, while task-specific evaluation remains necessary.

Problem
The paper asks how to evaluate and use self-supervised pretraining when labeled data, downstream tasks, and data complexity vary.
Method
The authors evaluate recent self-supervised algorithms using synthetic datasets with controlled difficulty, diverse downstream tasks, and full-model finetuning.
Results
Self-supervised pretraining is most useful with small labeling budgets, but utility approaches zero as labels grow; performance also varies across downstream settings and linear evaluation does not correlate with utility.
Takeaways & Limitations
Practitioners should assess pretraining methods across the specific tasks and data regimes in which they intend to use them.
Takeaways & Limitations
Utility cannot be computed when pretrained accuracy exceeds the maximum observed accuracy of the scratch-trained baseline.

Abstract

from arXiv · show

Recent advances have spurred incredible progress in self-supervised pretraining for vision. We investigate what factors may play a role in the utility of these pretraining methods for practitioners. To do this, we evaluate various self-supervised algorithms across a comprehensive array of synthetic datasets and downstream tasks. We prepare a suite of synthetic data that enables an endless supply of annotated images as well as full control over dataset difficulty. Our experiments offer insights into how the utility of self-supervision changes as the number of available labels grows as well as how the utility changes as a function of the downstream task and the properties of the training data. We also find that linear evaluation does not correlate with finetuning performance. Code and data is available at \href{https://www.github.com/princeton-vl/selfstudy}{github.com/princeton-vl/selfstudy}.

1. Introduction

The paper examines when self-supervised pretraining is useful in practical settings with varying label budgets. It evaluates this utility systematically and finds benefits are strongest with few labels, while outcomes depend on task and evaluation protocol.

1. Introduction: Existing evaluations often emphasize few-shot settings or frozen linear evaluation, although many vision applications use substantial labels and require finetuning.The paper frames unrestricted finetuning as important when maximizing task accuracy matters.
1. Introduction: The paper defines utility as the ratio of additional labels a scratch-trained model needs to match a finetuned pretrained model's accuracy.Utility is zero when both models reach the same accuracy with the same label count and infinite when the scratch model cannot match it.
1. Introduction: Self-supervised pretraining may improve optimization through better representations even when many labels are available, because SGD need not reach a global optimum.This motivates testing utility beyond few-shot regimes rather than assuming abundant labels eliminate pretraining benefits.
1. Introduction: The study evaluates self-supervised methods across downstream tasks, data properties, model sizes, and label quantities to identify where they help practitioners.Its synthetic benchmark enables controlled variation in dataset complexity and task difficulty.
1. Introduction: Self-supervised pretraining is useful with small labeling budgets, but its utility tends to decrease as labels become plentiful.The most common outcome is convergence to scratch-trained performance before the baseline reaches its accuracy plateau.

2. Related Work

Prior work spans many self-supervised objectives and broad evaluations, but this paper emphasizes utility under large label budgets, full-model finetuning, and controlled image complexity. Its synthetic setting supports analysis across these dimensions.

2. Related Work: The work focuses on self-supervised pretraining rather than weak-label or semi-supervised approaches.It also does not benchmark ImageNet pretraining or methods for improving training from scratch.
2. Related Work: Self-supervised methods include reconstruction, inpainting, colorization, spatial objectives, and contrastive embedding approaches.These methods exploit cues such as missing content, spatial relations, or agreement across image views.
2. Related Work: Existing comprehensive evaluations study factors such as architecture, disentanglement, unlabeled-data scale, and self-supervised task difficulty.The paper distinguishes its focus on utility with many labels and full-model finetuning.
2. Related Work: The benchmark uses synthetic images to examine how image complexity affects self-supervision performance.Synthetic data provides controlled variation while preserving access to annotations for diverse tasks.

3. Self-supervised Pretraining

The study uses a fixed two-stage pipeline: self-supervised pretraining on unlabeled images followed by supervised finetuning for a downstream task. It varies data, models, algorithms, and tasks to measure their interaction.

3. Self-supervised Pretraining: During pretraining, a chosen self-supervised algorithm fits its loss on unlabeled images; during finetuning, a new output layer is trained for the target task.The network is not jointly trained on the self-supervised and downstream objectives.
3. Self-supervised Pretraining: The benchmark varies dataset complexity, backbone model, self-supervision algorithm, downstream task, and available labels.Each setting compares a pretrained-and-finetuned model with a baseline trained from scratch.
3. Self-supervised Pretraining: Dataset difficulty is controlled through factors such as lighting, texture, and viewpoint because difficulty affects how many labels are needed to saturate performance.The study therefore controls complexity rather than comparing only image counts.
3. Self-supervised Pretraining: Different self-supervised cues may affect downstream performance differently, including when a method becomes invariant to features needed by a target task.The benchmark consequently tests multiple algorithms and downstream settings.

4. Synthetic Benchmark

The synthetic benchmark generates controlled rendered scenes with plentiful annotations while varying object arrangement, appearance, viewpoint, and lighting. These factors create datasets of different complexity for evaluating diverse visual tasks.

4. Synthetic Benchmark: Synthetic rendering provides an endless image supply, controlled generation factors, and annotations for tasks such as depth that are difficult to collect in real images.The benchmark uses Blender-rendered ShapeNet objects and enforces disjoint model splits across training and evaluation.
4. Synthetic Benchmark: Scenes vary in object count, orientation, texture, lighting, and placement while normalizing object scale across classes and models.Single-object scenes use fixed position and distance, whereas multi-object scenes distribute objects across the camera frame.
4. Synthetic Benchmark: Dataset complexity increases as more sources of image variation are introduced, including texture, color, viewpoint, and lighting.The benchmark compares controlled changes in these factors rather than changing them indiscriminately.
4. Synthetic Benchmark: The benchmark includes single-object and multi-object settings, with multi-object scenes supporting dense outputs such as semantic segmentation and depth.Figure 3 illustrates the multi-object images alongside ground-truth segmentation and depth.
4. Synthetic Benchmark: Color conditions range from class-associated color distributions to random colors with no correlation between color and object class.The harder condition removes the class-color shortcut while retaining the same object categories.

5. Downstream Tasks

The benchmark spans semantic and geometric tasks, contrasting global predictions with dense outputs. It evaluates classification, pose estimation, semantic segmentation, and depth estimation using deliberately simple task formulations.

Task Suite: The benchmark covers object classification, pose estimation, semantic segmentation, and depth estimation across semantic/geometric and global/dense contrasts.Classification and segmentation emphasize semantic information, while pose and depth emphasize geometric information; classification and pose are global, whereas segmentation and depth are dense.
Task design: The tasks are intentionally simple to limit training hyperparameters and isolate the impact of pretraining.
Object classification: Object classification predicts one of ten ShapeNet classes from single-object images and is measured by standard classification accuracy.
Object pose estimation: Pose estimation predicts five discretized orientations for centered objects, ignoring rotationally symmetric axes, and reports classification accuracy.The five bins represent upward, forward, backward, left, and right orientations.
Semantic segmentation: Semantic segmentation uses multi-object images, coarse per-pixel supervision, cross-entropy loss, and average classification accuracy.
Depth estimation: Depth estimation uses multi-object images with coarse outputs, an L1 loss, and the δ < 1.25 accuracy metric.The metric measures the percentage of predictions within a specified ratio of ground-truth depth.

6. Pretraining Methods

The study compares four self-supervised pretraining methods spanning latent reconstruction, rotation prediction, and contrastive representation learning. The methods balance varied self-supervision styles with differing task complexity.

Methods: The benchmark compares VAE, Rotation, CMC, and AMDIM as four self-supervised pretraining algorithms.
Variational autoencoder: VAE maps images into a low-dimensional latent space as an established reconstruction-based baseline.
Rotation: Rotation pretraining predicts whether an image was rotated by 0, 90, 180, or 270 degrees.
Contrastive Multiview Coding: CMC contrasts embeddings from separated image channels against embeddings from other images using two halved networks.The method splits images into channels such as L and ab in Lab color space.
AMDIM: AMDIM contrasts representations from augmented views of the same image and from intermediate network layers.
Method selection: The chosen methods balance different self-supervision styles and task complexity, with Rotation, CMC, and AMDIM identified as high-performing on ImageNet.

7. Experiment Details

Experiments use controlled synthetic datasets for global and dense tasks, varying labeled-data availability and measuring pretraining utility against training from scratch. Utility declines toward zero as labels increase and cannot always be computed.

Datasets: The study renders 15 dataset variations at 64x64 or 128x128 resolution for single-object or multi-object tasks.Single-object datasets support classification and pose estimation, while multi-object datasets support segmentation and depth.
Model and training: Dense-task outputs are predicted from features before global pooling at 16x16 resolution, without modifying the network backbone.
Utility versus labels: As labeled data increases, pretrained-model utility tends toward zero and converges with training from scratch before performance saturates.
Utility metric: Utility measures the proportion of additional labels required for training from scratch to match a pretrained model’s accuracy.For example, 500 versus 100 labels gives U(100) = 4.
Utility limitation: Utility is undefined when the pretrained model exceeds the maximum accuracy reached by the training-from-scratch baseline.Such regions are grayed out because the required matching label count cannot be computed from the observed baseline curve.

8. Results

Self-supervised pretraining is most useful with few labels, but its utility declines as labels grow and varies substantially across tasks, data complexity, and model size. Finetuning-based evaluations reveal patterns that linear evaluation can miss.

Utility vs Number of Labeled Samples: Self-supervised utility is significant with few labels but approaches zero as labeled data grows, before baseline performance plateaus.This pattern holds across pretraining methods and downstream settings.
Utility vs Downstream Task: The best pretraining method depends on the downstream task: CMC leads classification and pose estimation, while Rotation leads segmentation and AMDIM leads depth estimation.Thus, object-classification performance does not necessarily predict performance on other tasks.
Utility vs Data Complexity: Utility changes systematically with image variation, with CMC benefiting from random color or texture but losing utility under viewpoint changes.These effects recur across multiple dataset pairs.
Linear Evaluation: Linear evaluation does not reflect finetuned performance: AMDIM has the best linear result, yet its utility is consistently below CMC and Rotation.Linear evaluation is informative for frozen models but may not identify the best finetuned model.
Utility vs Model Size: Utility is higher with a ResNet50 backbone, because the larger model improves after pretraining while the scratch baseline drops at small dataset sizes.With sufficient labels, the scratch ResNet50 eventually outperforms ResNet9.
Utility vs Amount of Finetuning: Fully finetuned models consistently reach the best accuracy, whereas freezing more network blocks reduces performance.Some freezing helps with 100–250 samples, but full finetuning remains best overall.

9. Conclusion

The study evaluates self-supervised pretraining across synthetic datasets and diverse downstream settings to identify factors affecting its utility. It finds the greatest benefits in low-data regimes, while method rankings vary across scenarios.

9. Conclusion: The greatest benefits of self-supervised pretraining occur in low-data regimes, with utility approaching zero before additional labels plateau task performance.The study also finds that performance in one setting may not reflect performance in another.

A. Additional training details

The experiments control training time, augmentation, color space, and backbone differences while examining dataset shifts and pretraining-process tuning. Viewpoint changes produce the most consistent domain-shift degradation, whereas more unlabeled data or training time yields limited downstream benefit.

Training details: Finetuning gives pretrained models as much training time as models trained from scratch, with learning schedules and weight decay selected through hyperparameter tuning.Task-specific parameters in self-supervised settings are also tuned.
Training details: All settings use standard augmentation with task-specific exceptions, including no random cropping for depth estimation because of padding and resizing issues.The augmentation suite includes cropping, flipping, brightness/color perturbations, and Cutout.
Training details: Images are presented in Lab color space across methods and datasets because CMC uses Lab input, while large-data supervised-from-scratch performance is unaffected in ablations.This choice controls differences between methods.
Dataset variations: Viewpoint changes cause the most consistent finetuning drop when absent during pretraining but present during downstream training.This comparison controls for other image factors and evaluates object classification in the low-data regime.
Tuning the pretraining process: More unlabeled images or longer pretraining produce no effect or a modest adverse effect on downstream performance in the reported CMC comparison.The comparison includes half the training time and one-eighth as many unlabeled images; Figure 10 reports downstream object classification performance.
Network backbones: CMC uses a split backbone with approximately half the original parameters, while AMDIM comparisons match backbone size as closely as possible before evaluating finetuned performance.The CMC split assigns separate processing to the L and ab channels; the AMDIM comparison controls similar feature activation sizes and total parameter count.

B.4. Results on additional datasets

Additional figures extend the evaluation across datasets, tasks, and backbone choices. The reported observation is that method strength on dense prediction varies with both data amount and model.

Additional datasets: Different methods show stronger dense-prediction performance depending on the amount of data and the model.The additional-dataset results are described as consistent with those in the main paper.
Additional datasets: Figure 10 reports downstream object-classification performance for CMC across pretraining settings labeled by epoch count and unlabeled-image count.Using 50k rather than 400k unlabeled images is paired with more epochs to match total training iterations.
Additional datasets: Figures 11–14 cover CMC training from scratch, object classification, object-classification utility, and object-pose estimation with a ResNet9 base.These figures provide results for the ResNet9 setting across the listed tasks and metrics.
Additional datasets: Figure 12 compares a ResNet50 backbone with the AMDIM backbone while matching the models in total parameter count.The number of layers and channels is chosen to control for model size.
Additional datasets: Figures 19–24 report semantic-segmentation and depth-estimation results for ResNet9 and ResNet50, including direct comparisons between backbones.The figures organize results by task, backbone, and direct backbone comparison.