PaperlayerView exact revision on arXiv

Source-linked AI summary

A Cookbook of Self-Supervised Learning

Randall Balestriero, Mark Ibrahim, Vlad Sobal, Ari Morcos, Shashank Shekhar, Tom Goldstein, Florian Bordes, Adrien Bardes, Gregoire Mialon, Yuandong Tian, Avi Schwarzschild, Andrew Gordon Wilson, Jonas Geiping, Quentin Garrido, Pierre Fernandez, Amir Bar, Hamed Pirsiavash, Yann LeCun, Micah Goldblum

arXiv:2304.12210v2cs.LGcs.CV

TL;DR

Self-supervised learning can exploit vast unlabeled data, but its many methods, hyperparameters, and implementation details make the field difficult to enter and navigate. This cookbook organizes SSL foundations and families, then provides practical recipes for training and evaluation. It aims to help researchers understand the field’s design choices and train SSL methods successfully, while noting that evaluation beyond classification lacks a standardized protocol.

  • Problem

    SSL’s computational demands, intricate implementations, empirical blind spots, and lack of unified vocabulary create a high barrier to research and deployment.

  • Method

    The paper presents SSL foundations, method families, theoretical connections, practical training recipes, evaluation guidance, and researcher tips in cookbook form.

  • Results

    The cookbook covers SSL methods and practical design choices, including architectures, optimizers, hyperparameters, evaluation, and common training pitfalls.

  • Takeaways & Limitations

    The cookbook is intended to help researchers navigate SSL methods, understand their design choices, and gain practical know-how for training and exploration.

  • Takeaways & Limitations

    Evaluation beyond classification lacks a standardized protocol, with multiple existing approaches for detection and segmentation.

Abstract

from arXiv · show

Self-supervised learning, dubbed the dark matter of intelligence, is a promising path to advance machine learning. Yet, much like cooking, training SSL methods is a delicate art with a high barrier to entry. While many components are familiar, successfully training a SSL method involves a dizzying set of choices from the pretext tasks to training hyper-parameters. Our goal is to lower the barrier to entry into SSL research by laying the foundations and latest SSL recipes in the style of a cookbook. We hope to empower the curious researcher to navigate the terrain of methods, understand the role of the various knobs, and gain the know-how required to explore how delicious SSL can be.

1 What is Self-Supervised Learning and Why Bother?

Self-supervised learning (SSL) learns from unlabeled data to produce representations useful across tasks, but its many methodological and implementation choices create a high barrier to entry. The cookbook addresses this barrier by unifying SSL foundations and providing practical training recipes.

  • SSL learns generic representations from vast unlabeled data rather than relying on task-specific labeled data.This makes SSL useful when labels are costly or the downstream task is unknown in advance.
  • SSL research involves numerous choices spanning pretext tasks, training hyperparameters, and intricate implementations.The paper identifies computational cost and insufficient implementation transparency as additional barriers.
  • Empirical blind spots make it difficult to understand how SSL methods’ moving pieces affect learned representations.Addressing these blind spots requires substantial computational resources and prior hands-on experience.
  • The cookbook unifies SSL techniques through common foundations, method families, theoretical connections, and practical recipes.It covers hyperparameter choices, component assembly, evaluation, and training pitfalls.

2 The Families and Origins of SSL

Self-supervised learning has roots extending to the beginning of the deep learning era, despite its renewed prominence since 2020. Recent progress has been supported by extremely large datasets and high-memory GPUs.

  • SSL’s origins date back to the beginning of the deep learning era.
  • SSL experienced a renaissance after 2020 alongside extremely large datasets and high-memory GPUs.

2.1 Origins of SSL

Early SSL developed through several overlapping strategies for learning representations without labels, including information restoration, temporal relationships, spatial context, clustering, generative modeling, and multi-view invariance. These ideas provide foundations for contemporary SSL methods, which the paper organizes into four broad families.

  • Information restoration: Information-restoration methods remove or mask image content and train networks to reconstruct the missing information.Examples include colorization, inpainting, and masked auto-encoding with image patches.
  • Using temporal relationships in video: Video-based methods exploit temporal relationships by encouraging similar representations for the same object across nearby frames.This strategy was used for pretraining models that performed well on object detection.
  • Learning spatial context: Spatial-context methods predict object positions, orientations, or patch arrangements within an image.Examples include rotation prediction, relative patch-location prediction, and jigsaw tasks.
  • Grouping similar images together: Deep-clustering methods alternate between assigning feature-space cluster labels and updating the model to respect those assignments.
  • Generative models: Early generative approaches learned representations through layer-wise autoencoder or restricted-Boltzmann-machine pretraining, while GANs used generator–discriminator pairs.
  • Multi-view invariance: Multi-view invariance methods encourage similar representations for augmented versions of the same input.This principle became central to many modern contrastive SSL methods.
  • The paper categorizes SSL into the Deep Metric Learning, Self-Distillation, Canonical Correlation Analysis, and Masked Image Modeling families.

2.2 The Deep Metric Learning Family: SimCLR/NNCLR/MeanSHIFT/SCL

The Deep Metric Learning family evolved toward contrastive SSL by changing how positive and negative pairs are formed and by using deeper networks and predictor networks. SimCLR exemplifies this approach by aligning representations of augmented image views with a contrastive loss.

  • Deep Metric Learning encourages similarity between semantically transformed versions of an input through objectives such as contrastive loss.
  • The N-pair-mc strategy uses positive views from other samples as negatives for different pairs, improving sampling efficiency.
  • The transition from DML to SSL involved data augmentation for pair formation, deeper networks, and predictor networks.
  • SimCLR: SimCLR creates two augmented image views and encourages their representations to be similar.Its transformations include resizing, cropping, color jittering, and blurring.
  • SimCLR: SimCLR applies its contrastive loss after projecting encoded views into another space with a projector, often an MLP with ReLU.Representations before the projector have been shown to improve downstream performance.
  • SimCLR: InfoNCE and a temperature-controlled non-parametric softmax are key ingredients in the contrastive formulation used by SimCLR.The temperature changes the sharpness of prediction probabilities.

2.3 The Self-Distillation Family: BYOL/SimSIAM/DINO

The self-distillation family uses two augmented views, asymmetric student–teacher networks, and mechanisms that prevent representation collapse. Its history spans BYOL, SimSiam, DINO, iBOT, and DINOv2, with later methods adding clustering, masking, and improved training recipes.

  • Core mechanism: Self-distillation methods feed two views to two encoders and use a predictor to map one representation to the other.Collapse prevention commonly uses a moving-average update for one encoder.
  • BYOL: BYOL uses an online student updated by gradient descent and a target teacher receiving a different augmented view of the same image.The listed augmentations include resizing, cropping, color jittering, and brightness alterations.
  • SimSiam: SimSiam showed that EMA was not necessary in practice, although it provided a small performance boost.This simplified the BYOL-style objective while retaining the asymmetric architecture.
  • DINO: DINO centers student outputs and applies a softmax, while its teacher uses a moving-average encoder updated with ξ scheduled from 0.996 to 1.The softmax can be interpreted as online clustering over prototypes in the final layer.
  • Later extensions: iBOT combines DINO with masked image modeling in latent space, reconstructing teacher-embedded patches rather than image pixels.DINOv2 further improves the recipe and architecture, adds regularizers such as KoLeo, and uses 142 million pretraining images.

2.4 The Canonical Correlation Analysis Family: VICReg/BarlowTwins/SWAV/W-MSE

The canonical correlation analysis family learns representations from relationships between two views, emphasizing cross-covariance and correlated transformations. SSL methods including VICReg, Barlow Twins, SWAV, and W-MSE develop from this framework, with VICReg balancing variance, invariance, and covariance objectives.

  • CCA foundations: CCA analyzes cross-covariance between two variables and seeks transformations whose representations have zero mean and identity covariance.The output dimension d is constrained by d ≤ min(dim(X), dim(Y)).
  • Deep CCA: Deep CCA methods jointly learn two networks from two views so their outputs are maximally correlated.DCCAE extends this objective by regularizing CCA with an autoencoder.
  • SSL extensions: VICReg, Barlow Twins, SWAV, and W-MSE are SSL methods stemming from the CCA family.The passage identifies VICReg as the most recent among the listed methods.
  • VICReg: VICReg balances variance, invariance, and covariance objectives using representations from two views.Variance regularization prevents collapse, invariance aligns the views, and covariance encourages different representation dimensions to capture different information.

2.5 Masked Image Modeling

Masked image modeling (MIM) trains models to reconstruct masked or degraded image content, evolving from pixel inpainting toward latent-space targets and competitive vision performance.

  • Origins of Masked Image Modeling: Early MIM methods masked large image regions and trained encoder-decoder models to inpaint the missing pixels.Context encoders replaced masked pixels with white, while later methods used masked image patches and learned mask tokens.
  • Origins of Masked Image Modeling: BERT introduced masked language modeling by replacing text tokens with learned mask tokens and recovering the original text.The paper presents MLM as a masking-and-reconstruction strategy that remains popular for large language models.
  • Evolution of MIM: Direct pixel prediction with masked vision-transformer patches was found significantly less effective than supervised pre-training.This motivated alternative target representations and reconstruction formulations for image patches.
  • Evolution of MIM: BEiT cast MIM as regression over discrete image tokens, whereas MAE and SimMIM directly reconstruct masked image patches and surpassed BEiT on several downstream tasks.The simplified methods were reported to improve downstream image classification, semantic segmentation, and object detection performance relative to BEiT.
  • Modern Masked Image Modeling: MIM has achieved competitive performance across vision tasks and vision-language representation learning, with leading frozen-encoder methods reconstructing teacher-provided latent targets.iBOT and DINOv2 combine masked image modeling with self-distillation and reconstruct in latent space rather than from original pixels.
  • Modern Masked Image Modeling: MIM is fundamentally generative because it generates missing image parts conditional on observed content, while downstream systems can remove the decoder.Masked image models can also support generative modeling, including text-conditional generation.

2.6 A Theoretical Unification Of Self-Supervised Learning

The paper presents theoretical links among self-supervised objectives and studies representation collapse, showing that apparently different losses can share normalized criteria and representation properties.

  • Theoretical Unification: Barlow Twins’ criterion is linked to an upper bound of a contrastive loss, suggesting a connection between contrastive and covariance-based methods.Subsequent work derived the precise gap between the approaches and validated their similarities empirically at ImageNet scale.
  • Unified Contrastive Losses: A general family of contrastive losses uses monotonously increasing differentiable functions ϕ and ψ to encompass multiple objectives.The family includes generalized InfoNCE through specific choices of ϕ and ψ, with related settings recovering variations of SimCLR and DCL.
  • Unified Contrastive Losses: The unified contrastive-loss framework computes intra-view and inter-view pairwise squared distances among representations from augmented and distinct samples.Figure 8 depicts samples and augmented views passing through networks to produce representations whose distances enter the loss.
  • Hard Negative Sampling: Exponential weighting in contrastive losses already emphasizes hard-negative pairs at the batch level without explicit hard-negative sampling.Consequently, large batches are needed to observe hard negatives, increasing memory cost.
  • Dimensional Collapse: The projector maps representations into another space where the loss is computed, and theoretical work studies its role in preventing dimensional collapse.The supplied passage notes strong empirical evidence for projector benefits but limited theoretical explanation.
  • Dimensional Collapse: Dimensional collapse occurs when representation dimensions encode redundant information, producing rank-deficient projector outputs.The phenomenon can be approximated using the singular-value spectrum and has been linked by later work to downstream performance.

2.7 Pretraining Data

SSL pretraining commonly uses curated, object-centric datasets, but the paper examines transfer to less curated data and describes weakly curated retrieval as a scaling strategy.

  • Curated and Wild Data: ImageNet and similar curated datasets are common SSL pretraining sources because they are generally class-balanced and object-centric.Their simplicity makes them a useful experimental setting, although they do not represent most naturally occurring datasets.
  • Curated and Wild Data: Large uncurated datasets have been explored for SSL, but ImageNet or similarly curated pretraining remains the norm.Examples include pretraining on large uncurated data and on COCO or iNaturalist.
  • Curated and Wild Data: The paper evaluates whether ImageNet-style augmentations and tuned loss coefficients transfer to Places205 and iNaturalist18.Places205 is not object-centric, while iNaturalist18 has a power-law class distribution and requires fine-grained information.
  • Weakly Curated Data: Weakly curated training retrieves images similar to curated or smaller source datasets, producing collections that are larger and more diverse.DINOv2 used this strategy to build LVD-142M from many small and domain-specific datasets.

3 A Cook’s Guide to Successful SSL Training and Deployment

Successful SSL training depends on aligning augmentations, representations, projectors, and architecture choices with downstream tasks. The section also presents practical evidence that these design choices affect accuracy, robustness, computational cost, and representation properties.

  • 3.1 Role of Data-Augmentation: Data augmentations define the invariances learned by joint-embedding methods, but choices optimized for ImageNet may hurt other downstream tasks.ColorJitter benefits many classification tasks but is not universally beneficial; different augmentations induce different invariances.
  • 3.1.1 Role of multi-crop: Multi-crop increases positive-pair signal by comparing two large crops with additional small crops, while increasing training time and memory usage.With 2 large crops and N small crops, the invariance loss is computed 2(N −1) times; a modified SwAV setting raised training time by 25%.
  • 3.2 Role of the Projector: Projectors separate the SSL objective from the representation used downstream, and their usefulness extends to supervised training when training and downstream tasks are misaligned.The projector is typically discarded after training, while Guillotine Regularization refers more broadly to probing or cutting intermediate representations.
  • 3.2 Role of the Projector: A projector can mitigate harmful invariance constraints from semantically inconsistent augmentations, reducing the benefit of oracle filtering of augmented views.Table 2 reports that oracle filtering gives only minor gains when VICReg uses a projector.
  • 3.2 Role of the Projector: Projector output dimension influences SSL performance, but very large dimensions are not required for VICReg and comparable robustness is expected for Barlow Twins.VICReg remains more sensitive to output dimension than SimCLR while being significantly more robust than originally thought.
  • 3.2 Role of the Projector: DirectCLR shows that applying the InfoNCE objective to representation subvectors without a trainable projector can outperform SimCLR with a linear projector on ImageNet top-1 accuracy.This result demonstrates that a trainable projector is not universally necessary for strong SSL performance.

3.3 The Uniform Prior in SSL or the Failure of SSL on Unbalanced Data

SSL training involves interacting choices in objectives, optimization, architecture, and evaluation, with particularly important considerations for unbalanced data and Vision Transformers. The reviewed evidence identifies practical settings that improve stability or downstream performance while exposing scope boundaries.

  • 3.3 The Uniform Prior in SSL or the Failure of SSL on Unbalanced Data: SSL methods can perform poorly on unbalanced datasets because a hidden uniform prior may favor low-level rather than class-discriminative features.MSN adds a regularization term to alter the distribution of SSL clustering and address this limitation.
  • Training considerations: Hyperparameter differences across SSL methods make direct comparisons challenging, motivating systematic analysis of each parameter’s impact.The section aims to help practitioners identify which hyperparameters matter most for their setup.
  • 3.5.1 Role of Mini-Batch Size: A square-root learning-rate scaling rule can improve small-batch contrastive training, with gains of up to 5 points in ImageNet top-1 accuracy after 100 epochs.SimCLR can also be trained on ImageNet with a single GPU without an important performance drop, and DCL reports strong performance with smaller queues or batches.
  • Optimization: SSL optimization commonly uses LARS with base learning rates of 0.2–0.3 for several ImageNet methods, whereas MAE, DINO, and iBOT use AdamW with rates of 1e −5−5e −4.Some methods use substantially smaller learning rates for bias and batch-normalization parameters.
  • 3.5.3 Role of Weight-Decay: Weight decay can stabilize BYOL, with one hypothesis linking it to better modeling of augmentation invariance across initial conditions.The cited discussion connects weight decay to the online network and predictor’s memory of initial conditions.
  • 3.5.4 Vision Transformers Considerations: Vision Transformers are more prone to collapse and instability than standard architectures and are more sensitive to hyperparameter settings.Large-batch joint-embedding ViT training can show kNN accuracy drops when gradient L∞-norm spikes, even without a large final-accuracy decline.
  • 3.5.4 Vision Transformers Considerations: For ViT SSL pretraining, batch sizes of 1024 or 2048 appear to be a practical sweet spot, while warmup and random patch projection can improve stability.Very small batches can reduce final k-NN accuracy, whereas learned patch projection can be less stable for several methods.
  • 3.5.4 Vision Transformers Considerations: Smaller ViT patch sizes, such as 5 × 5 or 8 × 8 instead of 16 × 16, improve linear-probing accuracy in DINO pretraining.Larger patches reduce running time but increase memory usage.

3.6 Techniques for High Performance Masked Image Modeling

High-performance masked image modeling systems commonly combine masking with complementary techniques, while SSL evaluation uses increasingly complex protocols with distinct trade-offs. Label-free tools such as RankMe can support hyperparameter selection, but evaluation beyond classification remains non-standardized.

  • Techniques for High Performance Masked Image Modeling: State-of-the-art masked image modeling systems often pair MIM with normalization, distillation, or latent-space targets.ConvNextV2 combines MAE pretraining with global response normalization, while iBOT and DINO-style approaches use teacher-generated latent targets.
  • Techniques for High Performance Masked Image Modeling: Uniform masking preserves information across local windows in hierarchical pyramid vision transformers.The method hides equal amounts of information in each local window, addressing the risk that random masking removes all information from some windows.
  • Evaluation Protocols: KNN, linear probing, and full fine-tuning form common offline evaluation protocols ordered by increasing complexity.KNN uses frozen features and neighbor voting; linear probing trains a classifier on a frozen backbone, while full fine-tuning updates the model.
  • Evaluation Protocols: MLP probes reveal information unavailable to linear probes when learned features are not linearly separable.Comparing linear and nonlinear probes can indicate how readily information in the representation can be extracted.
  • Evaluation without labels: RankMe recovers most ImageNet-oracle hyperparameter-selection performance without labels, optimization, or parameters.Effective rank is the entropy of the singular-value distribution, but full rank is necessary rather than sufficient for good performance.
  • Beyond classification: Object detection and semantic segmentation lack a standardized protocol for evaluating self-supervised models.Existing alternatives include fine-tuning the encoder on a downstream task or using it as a feature extractor.

3.8 Speeding up Training

Speeding up SSL training requires managing distributed batch operations and data-processing bottlenecks, alongside hardware- and architecture-aware optimizations. FFCV-SSL and related techniques substantially reduce training time for representative SSL workloads.

  • Distributed Training: Distributed SSL training requires synchronizing batch-normalization statistics across devices.PyTorch can convert BatchNorm modules to synchronized BatchNorm so statistics are aggregated automatically.
  • Distributed Training: Losses that use batch-wide negatives or covariance require manually aggregating embeddings across devices.SimCLR uses other batch examples as negatives, while VICReg computes embedding covariance; gathered embeddings should enter the loss computation.
  • Training Configuration: Effective batch size should guide run comparisons and adaptive learning-rate scaling.A recommended rule is effective_lr = base_lr * effective_batch_size / 256.
  • Data Processing: Data processing can bottleneck SSL training because joint-embedding methods require multiple handcrafted augmentations.DALI and FFCV-SSL are presented as alternatives to torchvision for accelerating data loading.
  • Even Faster Training with FFCV and Other Speedups: 3x time speedup is enabled by FFCV-SSL relative to torchvision, with SimCLR training taking less than 2 days on one GPU.The same approach enables training in a few hours using eight GPUs.
  • Vision Transformer Speedups: 84% top-1 accuracy is achieved by Data2vec 2.0 after three hours of pretraining on 32 GPUs.Vision-transformer efficiency benefits from processing fewer patches with masked prediction objectives; bfloat16 can also accelerate training while avoiding float16 numerical-instability issues.

4 Extending Self-Supervised Learning Beyond Images and Classification

SSL extends across audio, video, text, tabular data, reinforcement learning, behavior cloning, multimodal training, and dense prediction, but domain-specific choices and limitations shape its effectiveness.

  • Audio and Video: SSL methods require domain-specific augmentations and positive-pair choices because transformations useful for images may not naturally apply to speech or other signals.Audio may pair sound with text or video, while video extends image methods across temporal clips.
  • Text and Tabular Data: Text SSL predominantly uses reconstruction objectives, whereas tabular methods combine masking, reconstruction, contrastive learning, and task-guided pair construction.Text is described as a relatively clean signal, while tabular SSL remains comparatively small and uses varied augmentation strategies.
  • Text and Tabular Data: Pretraining objectives generally improve tabular model performance, but label-using objectives perform best and existing routines produce less transferable features than supervised pretraining.These findings indicate that tabular SSL has not reached the state of the art in its domain.
  • Reinforcement Learning and Behavior Cloning: SSL supports reinforcement learning and behavior cloning, including improved sample efficiency, strong robotic control results, and effective visual, tactile, and trajectory representations.EfficientZero adds SimSiam to MuZero and establishes a new Atari100k state of the art; BYOL representations support multiple robotic imitation and manipulation tasks.
  • Multiple Modalities: Multimodal image-text pretraining yields encoders robust to semantically preserving visual changes and strong on out-of-domain generalization, while combining image-text and image-image SSL can improve linear probing.These representations may group visually dissimilar but semantically alike features, a behavior mitigated by combining modalities.
  • Localization for Dense Prediction: Localization remains a limitation for some augmented-view SSL methods, whereas transformer-specific objectives and masked autoencoding can produce representations effective for detection and segmentation.ViTs naturally provide localization through patchwise features, while CNN adaptations of masked autoencoding can also achieve competitive dense-prediction performance.

5 Conclusion

SSL has achieved many successes but remains difficult to navigate because its methods have intricate implementations and rapidly evolving choices. This breadth creates a high barrier to entry for newer researchers and practitioners.

  • 5 Conclusion: SSL remains challenging to navigate because the field contains many methods with intricate implementations and evolves rapidly.The paper frames this difficulty as a barrier for researchers and practitioners who recently joined the field.
Loading 2304.12210v2…