Source-linked AI summary
Revisiting Self-Supervised Visual Representation Learning
Alexander Kolesnikov, Xiaohua Zhai, Lucas Beyer
TL;DR
Self-supervised visual representation learning seeks useful representations without expensive labeled data, but CNN architecture has received less attention than pretext-task design. The paper conducts a large-scale study of existing methods across architectures, uncovering architecture-specific behavior and boosting prior techniques to new state-of-the-art results. It also identifies limits in model selection and evaluation, including the lack of reliable label-free architecture selection.
Problem
Self-supervised research has emphasized pretext tasks more than CNN architecture, despite the need to learn useful visual representations without expensive labeled data.
Method
The paper revisits existing self-supervised models in a large-scale empirical study spanning multiple CNN architectures and architectural choices.
Results
The study obtains new state-of-the-art results, including context prediction outperforming published self-supervised results when paired with an appropriate CNN architecture.
Takeaways & Limitations
Pretext tasks should be considered together with underlying architectures because method and architecture rankings are not consistent across one another.
Takeaways & Limitations
The fixed model architecture cannot reliably select the best architecture, so other label-free model-selection mechanisms remain needed.
Abstract
from arXiv · showhide
Unsupervised visual representation learning remains a largely unsolved problem in computer vision research. Among a big body of recently proposed approaches for unsupervised learning of visual representations, a class of self-supervised techniques achieves superior performance on many challenging benchmarks. A large number of the pretext tasks for self-supervised learning have been studied, but other important aspects, such as the choice of convolutional neural networks (CNN), has not received equal attention. Therefore, we revisit numerous previously proposed self-supervised models, conduct a thorough large scale study and, as a result, uncover multiple crucial insights. We challenge a number of common practices in selfsupervised visual representation learning and observe that standard recipes for CNN design do not always translate to self-supervised representation learning. As part of our study, we drastically boost the performance of previously proposed techniques and outperform previously published state-of-the-art results by a large margin.
1. Introduction
Self-supervised learning reduces reliance on expensive labeled data, but representation quality depends substantially on CNN architecture as well as the pretext task. A large-scale study revisits these design choices and reports improved state-of-the-art results.
- Central observation: CNN architecture significantly affects the quality of representations learned by self-supervised techniques and therefore matters for model evaluation.The paper provides a large-scale study supporting this observation.
- Motivation: Self-supervised learning creates pretext-task supervision from unlabeled data to learn representations useful for downstream visual tasks.Examples include predicting image context or rotation, with high-level image understanding needed to solve the pretext task.
- Study motivation: Prior work mainly improved self-supervised performance by proposing novel pretext tasks, while CNN architecture received less systematic attention.The study instead examines existing pretext tasks across multiple CNN architectures.
- Key findings: Standard fully supervised architecture-design recipes do not necessarily transfer to self-supervised learning, where otherwise minor choices may substantially change performance.This challenges the assumption that architectures performing well with labels will behave similarly without labels.
- Key findings: Self-supervised representations achieve new state-of-the-art results, including context prediction surpassing published self-supervised results when paired with an appropriate CNN.The study also reports that representation quality in skip-connected CNNs does not degrade toward the model’s end, widening consistently improves quality, and SGD-based linear evaluation can require many epochs to converge.
2. Related Work
Self-supervision automatically constructs training signals for learning representations, with applications spanning modalities and tasks. This work complements pretext-task research by studying how architecture design affects existing self-supervised methods.
- Self-supervision: Self-supervision automatically creates a supervised signal for a pretext task to learn representations useful for real-world downstream tasks.The framework has applications ranging from robotics to image understanding.
- Applications: Robotics can exploit interaction outcomes and simultaneous sensory inputs across perception modalities as self-supervised signals.
- Applications: Video representation learning can use synchronized audio, video, subtitles, or temporal consistency, whereas this paper focuses on image databases.
- Pretext tasks: Image-based methods construct pretext tasks through patch relations, image rotation, clustering, inpainting, or colorization.
- This work: The study complements methods that introduce new pretext tasks by showing that existing self-supervision methods can benefit from architecture-focused insights.
- This work: Unlike related multi-task work using a modified ResNet101 without further architectural investigation, this paper focuses on architecture design’s influence on representation quality.
3. Self-supervised study setup
The study evaluates four self-supervised techniques across six CNN architectures and varying architectural details, using frozen representations and downstream linear classification. It examines both modern network designs and the standard evaluation procedure.
- Study design: The study compares six CNN models with four self-supervised approaches, then evaluates representations using specified metrics and datasets.The setup is organized around architecture selection, self-supervised techniques, evaluation metrics, and datasets.
- CNN architectures: The architectures include ResNet variants, RevNet, and batch-normalized VGG, selected to test whether self-supervision changes when modern networks replace widely used AlexNet.
- CNN architectures: ResNet models preserve representation quality toward the network’s end, motivating inclusion of RevNets because their residual units provide stronger invertibility guarantees.The paper hypothesizes that skip-connections may preserve information across depth under certain circumstances.
- CNN architectures: The study varies CNN widening factor k, producing pre-logits of size 2048, 4096, 6144, or 8192, while omitting configurations that do not fit memory.
- CNN architectures: It tests ResNet v1, ResNet v2, and a variant without ReLU before global average pooling, even though these variants perform similarly on the pretext task.
- Self-supervised techniques: The four approaches are Rotation, Exemplar, Jigsaw, and Relative Patch Location, each defining a different image-based prediction task.Rotation predicts one of four angles; Jigsaw predicts a patch permutation; Relative Patch Location predicts one of eight spatial relations.
- Evaluation: Representations are extracted from frozen networks, typically at the pre-logits level, and assessed by training linear logistic regression on ImageNet or Places205.The study also investigates a more expressive one-hidden-layer MLP evaluation model.
4. Experiments and Results
The study evaluates self-supervised representations across CNN architectures, widening factors, pretext tasks, layers, datasets, and evaluation procedures. Architecture selection and scaling substantially affect results, while several findings challenge standard assumptions about self-supervised evaluation and model design.
- Study setup: Six CNN architectures with varied widening factors were evaluated across four self-supervised techniques using pre-logits representations and linear ImageNet evaluation.The study also evaluates models on Places205 and compares selected models with prior literature using official validation splits.
- Architecture effects: Architecture rankings vary across self-supervision methods, and method rankings vary across architectures; VGG19-BN consistently performs worst despite similar supervised benchmark performance to ResNet models.RevNet50 v2 excels under Rotation, whereas relative patch location is strongest only with ResNet50 v1; these rankings generalize across ImageNet and Places205.
- Width and representation size: Increasing CNN channel count consistently improves self-supervised representation quality across full-data and low-data regimes, although width also increases final representation dimensionality.The study therefore separates widening factor from representation size and finds that both independently matter, with larger values performing better.
- Comparison to prior work: 55.4% top-1 accuracy on ImageNet is achieved by the strongest Rotation model, while context prediction reaches 51.4% and the models nearly halve the gap to fully supervised results.These gains come from selecting architectures for each self-supervision method and increasing the widening factor, with similar observations on Places205.
- Evaluation procedure: Linear evaluation remains adequate because an MLP provides only marginal improvement and leaves relative performance mostly unchanged.However, the linear evaluator is sensitive to learning-rate scheduling, and approximately 500 training epochs can yield higher accuracy.
- Layer depth and skip-connections: Representations from residual architectures improve through the final pre-logits layer, unlike VGG19-BN, while stronger RevNet invertibility boosts Rotation performance by more than 5% but not other tasks.The authors hypothesize that invertibility preserves information across depth and prevents deterioration, but the RevNet benefit is task-dependent.
5. Conclusion
The study finds that self-supervised representation quality depends strongly on CNN architecture, and that architecture and method rankings vary across one another. It also reports consistently improved existing techniques and a reduced gap to fully supervised learning.
- Architecture-design lessons from fully supervised learning do not necessarily transfer to self-supervised representation learning.
- Architecture rankings vary across self-supervised methods, while method rankings also vary across architectures.
A. Self-supervised model details
The self-supervised models use SGD with momentum and task-specific training configurations for rotation, exemplar, jigsaw, and relative patch-location learning.
- All self-supervised models are trained with SGD and momentum, using a 0.1 initial learning rate and scheduled tenfold decays.Training lasts 35 epochs, with decays after epochs 15 and 25.
- Rotation: Rotation training uses batches of 1024 images, repeating each image once for every rotation, on 128 TPU cores.
- Exemplar: Exemplar training generates eight augmented examples per image, uses triplet loss with margin 0.5, and runs on 32 TPU cores.
- Jigsaw: Jigsaw preprocessing resizes and crops images, optionally grayscales them, divides them into a 3 × 3 grid, and standardizes each patch.
- Relative Patch Location: Relative Patch Location uses the Jigsaw model’s preprocessing, representation extraction, and training setup, changing only the loss function.
B. Downstream training details
Downstream representations are evaluated with linear or MLP models using specified augmentation, patch aggregation, optimization, regularization, and batch-size settings.
- Rotation and Exemplar representations use standard resizing and random 224 × 224 crops, while patch methods extract representations from image patches.
- Patch-based evaluation averages representations from all nine colorful, standardized patches, with fixed central-grid patches at evaluation time.
- Evaluation batch sizes are 2048 for Rotation and Exemplar and 1024 for Jigsaw and Relative Patch Location, with learning-rate scaling.
- Linear models with L-BFGS: L-BFGS training uses at most 800 updates and l2 regularization with λ = 100.0.
- MLP models with SGD: The MLP has one 1000-channel hidden layer, 50% dropout, and SGD with momentum for 180 epochs.Its batch size is 512, with an initial learning rate of 0.01 and two tenfold decays.
C. Training linear models with SGD
Linear-model evaluation can require very long optimization, because top validation accuracy is reached only after training for many epochs.
- In every evaluated case, achieving top validation accuracy requires training for a very large number of epochs.
D. More Results on Places205 and ImageNet
This section provides extended evaluations across ImageNet and Places205, including reduced-data settings and additional top-5 accuracy results. It also compares published self-supervised models with the paper’s best models using linear evaluation.
- Additional evaluation settings: Extended tables report ImageNet results using 10% of the data and Places205 results using all data and a 5% random subset.These settings are covered in Tables 3–5.
- Additional evaluation settings: Table 6 adds top-5 accuracies for the paper’s best models on the public ImageNet validation set.It extends the main-paper comparison table.
- Evaluation protocol: The downstream accuracy curve uses an SGD-trained linear evaluation model on representations learned by Rotation, Exemplar, RelPatchLoc, and Jigsaw.These are the four self-supervision pretext tasks shown with the curve.
- Evaluation protocol: The comparison evaluates representations with linear logistic regression on official ImageNet and Places205 validation splits.The table also identifies the referenced architecture family as AlexNet, VGG-style, or Residual.