Bootstrap Your Own Latent: A New Approach to Self-Supervised Learning

Jean-Bastien Grill; Florian Strub; Florent Altché; Corentin Tallec; Pierre H. Richemond; Elena Buchatskaya; Carl Doersch; Bernardo Avila Pires; Zhaohan Daniel Guo; Mohammad Gheshlaghi Azar; Bilal Piot; Koray Kavukcuoglu; Rémi Munos; Michal Valko

Source-linked AI summary

Bootstrap Your Own Latent: A New Approach to Self-Supervised Learning

Jean-Bastien Grill, Florian Strub, Florent Altché, Corentin Tallec, Pierre H. Richemond, Elena Buchatskaya, Carl Doersch, Bernardo Avila Pires, Zhaohan Daniel Guo, Mohammad Gheshlaghi Azar, Bilal Piot, Koray Kavukcuoglu, Rémi Munos, Michal Valko

arXiv:2006.07733v1Jun 13, 2020cs.LGcs.CVstat.ML

TL;DR

Self-supervised image representation learning commonly relies on contrastive negative pairs and augmentation-sensitive training. BYOL instead bootstraps representations with online and target networks, achieving state-of-the-art results without negative pairs, including 79.6% ImageNet top-1 accuracy with a larger ResNet.

Problem
Contrastive self-supervised methods depend on negative pairs and careful augmentation choices for learning useful image representations.
Method
BYOL trains an online network to predict a target network’s representation of another augmented view, updating the target with a slow-moving average of the online network.
Results
BYOL achieves state-of-the-art results without negative pairs, reaching 79.6% top-1 accuracy on ImageNet with ResNet-200 (2×) and performing on par or better on transfer and semi-supervised benchmarks.
Takeaways & Limitations
BYOL provides a self-supervised representation-learning approach that is more resilient than contrastive methods to batch-size and augmentation changes.
Takeaways & Limitations
BYOL remains dependent on suitable, vision-specific augmentation sets, whose design may require substantial effort to extend the method to other modalities.

Abstract

from arXiv · show

We introduce Bootstrap Your Own Latent (BYOL), a new approach to self-supervised image representation learning. BYOL relies on two neural networks, referred to as online and target networks, that interact and learn from each other. From an augmented view of an image, we train the online network to predict the target network representation of the same image under a different augmented view. At the same time, we update the target network with a slow-moving average of the online network. While state-of-the art methods intrinsically rely on negative pairs, BYOL achieves a new state of the art without them. BYOL reaches $74.3\%$ top-1 classification accuracy on ImageNet using the standard linear evaluation protocol with a ResNet-50 architecture and $79.6\%$ with a larger ResNet. We show that BYOL performs on par or better than the current state of the art on both transfer and semi-supervised benchmarks.

1 Introduction

BYOL addresses limitations of contrastive self-supervised learning by learning representations without negative pairs and achieves strong ImageNet and broader benchmark results.

1 Introduction: Contrastive methods require positive and negative pairs, with performance depending on negative-pair handling and image augmentations.They use large batches, memory banks, or mining strategies to obtain negatives.
1 Introduction: BYOL learns by predicting another augmented view’s target representation without using negative pairs.Its online network predicts the target network representation, while the target is updated using a slow-moving average of the online network.
1 Introduction: BYOL reaches 74.3% top-1 accuracy with ResNet-50 and 79.6% with a larger ResNet under ImageNet linear evaluation.It also performs on par or better than the state of the art in semi-supervised and transfer settings.
1 Introduction: BYOL is reported to be more robust than contrastive methods to changes in batch size and image augmentations.With only random crops, BYOL suffers a much smaller performance drop than SimCLR.

2 Related work

Prior self-supervised approaches include generative, contrastive, bootstrapping, and handcrafted prediction methods, while BYOL builds on bootstrapping without negative pairs or clustering.

2 Related work: Generative methods model data and latent embeddings, often through auto-encoding or adversarial learning, but typically operate directly in computationally expensive pixel space.
2 Related work: Contrastive methods achieve state-of-the-art self-supervised performance by bringing positive pairs together and separating negative pairs.Their success often depends on comparing each example with many others, motivating whether negative pairs are necessary.
2 Related work: DeepCluster avoids negative pairs by clustering representations and using cluster indices as targets, but requires costly clustering and anti-collapse precautions.
2 Related work: BYOL differs from PBL by using a slow-moving representation average for targets and by not requiring a second network.
2 Related work: MoCo uses a moving-average network to stabilize negative pairs from a memory bank, whereas BYOL uses it to stabilize prediction targets.The paper reports that this stabilizing effect can also improve existing contrastive methods.

3 Method

BYOL predicts one augmented view’s representation from another using online and target networks, with a moving-average target intended to avoid collapse.

3 Method: BYOL trains an online network to predict the target network’s representation of another augmented view of the same image.The method uses cross-view prediction in representation space, where direct prediction can otherwise produce collapsed representations.
3 Method: A fixed random target avoids collapse but yields weaker representations, motivating iterative refinement with a slowly moving target network.The fixed-random-target procedure reaches 18.8% top-1 accuracy versus 1.4% for the random network itself.
3 Method: The online network contains an encoder fθ, projector gθ, and predictor qθ, while training ultimately retains only the encoder representation.The final representation y is used for downstream tasks, and the other training components are discarded.
3 Method: The target network matches the online architecture, but its parameters are an exponential moving average of the online parameters.This creates a delayed target representation; the target is not directly updated by the prediction loss.
3 Method: BYOL minimizes mean squared error between normalized online predictions and target projections, then symmetrizes the loss across the two views.The target branch uses stop-gradient so optimization updates the online weights rather than the target weights.
3 Method: BYOL uses image augmentations adapted from SimCLR and evaluates architectures based on ResNet encoders with projected representations.The experiments use random crops, flips, color changes, blur, and solarization, alongside ResNet variants.

4 Experimental evaluation

BYOL’s ImageNet representation performs strongly under linear and semi-supervised evaluation, while transferring effectively across classification and other vision tasks.

Linear evaluation on ImageNet: 74.3% top-1 accuracy with standard ResNet-50 and 79.6% with a larger ResNet under ImageNet linear evaluation, surpassing previous self-supervised approaches.BYOL achieves 74.3% top-1 and 91.6% top-5 accuracy with ResNet-50, improving top-1 accuracy by 1.3% over the previous self-supervised state of the art.
Semi-supervised training on ImageNet: BYOL consistently outperforms previous approaches across ImageNet semi-supervised settings and reaches 77.7% top-1 accuracy when fine-tuned with all labels using ResNet-50.The evaluation uses fixed 1% and 10% labeled-data splits and multiple ResNet architectures.
Transfer to other classification tasks: BYOL’s representation transfers across classification benchmarks using linear evaluation and fine-tuning protocols on held-out test sets.These experiments assess whether ImageNet features remain useful across image domains rather than being ImageNet-specific.
Transfer to other vision tasks: BYOL outperforms Supervised-IN and SimCLR on VOC2012 semantic segmentation by +1.9 and +1.1 mIoU, respectively.The task classifies each pixel in the image.
Transfer to other vision tasks: BYOL is significantly better than Supervised-IN and SimCLR on object detection, improving AP50 by +3.1 and +2.3, respectively.The evaluation fine-tunes Faster R-CNN on trainval2007 and reports test2007 results.
Transfer to other vision tasks: On NYU v2 depth estimation, BYOL is better or on par across reported metrics, improving pct.<1.25 by +3.5 and +1.3 points over supervised and SimCLR baselines.The evaluation measures relative error, RMS error, and the percentage of pixels below the stated depth-ratio threshold.

5 Building intuitions with ablations

The ablations show that BYOL’s robustness depends on its bootstrap target and predictor, while avoiding negative pairs improves resilience to smaller batches and weaker augmentations.

Batch size: BYOL remains stable from batch sizes 256 to 4096, whereas SimCLR rapidly deteriorates as fewer negative examples become available.BYOL drops only at smaller batch sizes because of batch normalization layers.
Image augmentations: Removing color distortions decreases accuracy by 9.1 points for BYOL versus 22.2 points for SimCLR.With random crops alone, BYOL reaches 59.4% while SimCLR reaches 40.3%.
Bootstrapping: Target-network decay rates between 0.9 and 0.999 exceed 68.4% top-1 accuracy at 300 epochs, while decay rates of 0 or 1 destabilize or limit improvement.The target is an exponential moving average of the online network, balancing update frequency against stability.
Ablation to contrastive methods: Only BYOL’s combination of a bootstrap target network and predictor performs well without negative examples.Adding negative pairs back can match BYOL after properly tuning the temperature.
Ablation to contrastive methods: Adding a target network to SimCLR improves performance by 1.6 points, indicating a stabilization benefit beyond supplying additional negative examples.Adding a predictor to SimCLR has only a mild effect.

6 Conclusion

BYOL learns image representations by predicting previous outputs without negative pairs and achieves strong ImageNet performance. Its broader generalization remains limited by reliance on vision-specific augmentations.

6 Conclusion: BYOL reaches 79.6% top-1 accuracy with ResNet-200 (2×), improving over the previous 76.8% state of the art with 30% fewer parameters.With ResNet-50 (1×), it bridges most of the remaining gap between self-supervised and supervised learning.
6 Conclusion: BYOL learns representations by predicting previous versions of its outputs without using negative pairs.The method achieves state-of-the-art results across various benchmarks.
6 Conclusion: BYOL depends on augmentation sets specific to vision, so extending it to audio, video, or text requires suitable augmentations for each modality.Automating augmentation search is identified as an important next step.

Broader impact

The paper’s broader-impact discussion places BYOL within unsupervised learning and notes both broad vision applicability and risks from dual use and biased datasets.

Broader impact: BYOL is categorized as unsupervised learning and can be used for many different vision applications.Specific uses may have both positive and negative impacts, creating a dual-use concern.
Broader impact: Representations learned from biased vision datasets could replicate those biases.This is identified as a potential broader-impact risk of applying BYOL.

C.1 Self-supervised learning evaluation on ImageNet

BYOL is evaluated on ImageNet using linear and semi-supervised protocols, including modified evaluation settings and larger ResNet architectures. Its results approach supervised performance and remain competitive across label fractions.

C.1 Self-supervised learning evaluation on ImageNet: The standard linear evaluation protocol trains a classifier on frozen representations without updating network parameters or batch statistics.Training uses spatial augmentations, SGD with Nesterov momentum, 80 epochs, and a sweep over five learning rates.
C.1 Self-supervised learning evaluation on ImageNet: 74.3% top-1 accuracy is achieved by standard ResNet-50 linear evaluation on ImageNet, while a modified protocol increases it to 74.8%.The modified protocol replaces spatial augmentations with pre-training augmentations and regularizes the linear classifier.
C.1 Self-supervised learning evaluation on ImageNet: Semi-supervised experiments fine-tune BYOL representations using 1%, 2%, 5%, 10%, 20%, 50%, or 100% of ImageNet labels.The experiments use a ResNet-50 (1×) and balanced custom splits for the fractional-label setting.
C.1 Self-supervised learning evaluation on ImageNet: 77.7 top-1 is obtained after full-ImageNet fine-tuning from BYOL initialization, exceeding the vanilla supervised baseline, whereas SimCLR initialization reaches no better than random initialization at 76.5.The comparison is reported alongside supervised baselines in the full-ImageNet setting.
C.2 Linear evaluation on larger architectures and supervised baselines: 79.6% top-1 accuracy is achieved with a larger ResNet under ImageNet linear evaluation, approaching supervised baselines within 0.4 accuracy points for ResNet-50 (2×) and (4×).The larger-architecture comparison uses supervised baselines and reports the narrowest self-supervised gap described here.

D.1 Datasets

The transfer evaluation covers classification, segmentation, detection, and depth datasets using frozen-representation or fine-tuning protocols. Across comparable published baselines, BYOL generally performs better, with one noted NYU Depth exception.

D.1 Datasets: Transfer evaluation spans Food-101, CIFAR-10/100, Birdsnap, SUN397, Stanford Cars, FGVC Aircraft, PASCAL VOC, DTD, Oxford-IIIT Pets, Caltech-101, and Oxford 102 Flowers.The reported classification transfer uses linear evaluation and fine-tuning on the same dataset set as [8].
D.1 Datasets: Dataset-specific validation procedures include a single SUN397 split and newly created validation subsets when official validation data are unavailable.SUN397 uses 40 training and 10 validation images per class from its training images.
D.1 Datasets: Linear transfer evaluation uses a regularized multinomial logistic regression classifier on frozen representations without image augmentation.Images are resized and normalized with ImageNet statistics, and regularization is selected from 45 logarithmically spaced values between 10^-6 and 10^5.
D.1 Datasets: The transfer suite also includes semantic segmentation, object detection, and NYU v2 depth estimation.The associated result tables cover segmentation and detection together, and depth estimation separately.
D.7 Further comparisons on PASCAL and NYU v2 Depth: BYOL outperforms comparable published baselines in almost all transfer settings, except NYU Depth RMS error.The exception is sensitive to outliers, and the authors state that its reason is unclear.

E Pretraining on Places 365

The paper pretrains BYOL and SimCLR on Places365 before transfer evaluation. BYOL trained on Places365 consistently beats SimCLR trained on the same dataset, while ImageNet-trained BYOL is stronger on most tasks.

E Pretraining on Places 365: Places365 pretraining uses 1.80 million training images and 36,500 validation images, with BYOL and SimCLR trained for 1000 epochs using the Section 4 parameters.The dataset scale is described as roughly similar to ImageNet.
E Pretraining on Places 365: BYOL-PL consistently outperforms SimCLR on Places365, but BYOL-IN outperforms BYOL-PL on every task except Places365 and SUN397.All three unsupervised representations perform relatively well on the Places365 task.
E Pretraining on Places 365: Projector and predictor depth affect BYOL performance, with the default depth-2 configuration performing best.The ablation varies both MLP depths and projection size.
E Pretraining on Places 365: BYOL’s performance is reported as robust within a range of learning-rate and weight-decay settings, while batch-size sweeps show faster SimCLR deterioration.BYOL remains mostly stable as batch size decreases from 4096 toward 64, whereas SimCLR degrades faster.
E Pretraining on Places 365: BYOL is more resilient than SimCLR to changes in image augmentations across the tested transformations.The comparison is reported in the image-transformation ablation.

F.4 Details on the relation to contrastive methods

The paper analyzes BYOL’s relation to contrastive learning through loss variants, target-network behavior, normalization, and implementation details. The ablations identify target stabilization and ℓ2 normalization as important design choices.

F.4 Details on the relation to contrastive methods: No tested combination of temperature α and dispersion weight β with a predictor and target network significantly outperforms BYOL, which corresponds to β = 0.Adding negative examples with β = 1 hurts performance in the BYOL setting.
F.4 Details on the relation to contrastive methods: Stable, stale prediction targets—not merely stopping their gradients—are identified as the main source of the target network’s improvement.The target network both blocks gradients and updates targets by averaging the online network, and the ablation separates these effects.
F.4 Details on the relation to contrastive methods: BYOL minimizes squared error between ℓ2-normalized predictions and targets, using online-network predictions against target-network projections from augmented views.The implementation computes gradients with respect to online parameters while stopping gradients through target outputs.

G.4 Training loop

The training loop creates two augmented views per input image, updates the online network by gradient descent, and updates the target network with an exponential moving average.

G.4 Training loop: After completing the loop, the implementation returns the trained online-network parameters.The loop maintains optimizer state and both parameter sets, but the returned representation is the online parameter collection.
G.4 Training loop: Two independently augmented views of each input image are passed to the update function at every training step.The loop generates image_1 and image_2 using separate random-number streams before calling update_fn.
G.4 Training loop: The loss is differentiated with respect to the online parameters only, which are then updated using the optimizer and scheduled learning rate.The update computes gradients with jax.grad, applies optimizer updates, and returns revised online parameters.
G.4 Training loop: The target parameters are updated after the online step through a target-EMA coefficient that moves them toward the online parameters.The coefficient tau is obtained from target_ema and used in the parameter interpolation update.
G.4 Training loop: Initialization creates online and target network parameters from a dummy input and initializes optimizer state for the online parameters.The dummy image is used to compute intermediate output shapes during initialization.