PaperlayerView exact revision on arXiv

Source-linked AI summary

Barlow Twins: Self-Supervised Learning via Redundancy Reduction

Jure Zbontar, Li Jing, Ishan Misra, Yann LeCun, Stéphane Deny

arXiv:2103.03230v3cs.CVcs.AIcs.LGq-bio.NC

TL;DR

Self-supervised learning methods can collapse to trivial constant representations when enforcing similarity across distorted views. Barlow Twins uses cross-correlation redundancy reduction to avoid collapse and outperforms previous methods in low-data ImageNet semi-supervised classification while matching state-of-the-art performance on several other evaluations.

  • Problem

    Similarity-based self-supervised learning faces trivial constant representations, requiring mechanisms to learn useful representations.

  • Method

    Barlow Twins matches the cross-correlation matrix of embeddings from two distorted views to the identity, promoting similarity while reducing component redundancy.

  • Results

    55% top-1 accuracy with 1% labels; Barlow Twins is on par with state of the art for ImageNet linear evaluation and transfer classification and detection.

  • Takeaways & Limitations

    Barlow Twins benefits from very high-dimensional embeddings, whereas other popular self-supervised methods rapidly saturate as dimensionality increases.

  • Takeaways & Limitations

    Exploring embeddings beyond 16,000 dimensions would require new methods or alternative hardware to accommodate their memory requirements.

Abstract

from arXiv · show

Self-supervised learning (SSL) is rapidly closing the gap with supervised methods on large computer vision benchmarks. A successful approach to SSL is to learn embeddings which are invariant to distortions of the input sample. However, a recurring issue with this approach is the existence of trivial constant solutions. Most current methods avoid such solutions by careful implementation details. We propose an objective function that naturally avoids collapse by measuring the cross-correlation matrix between the outputs of two identical networks fed with distorted versions of a sample, and making it as close to the identity matrix as possible. This causes the embedding vectors of distorted versions of a sample to be similar, while minimizing the redundancy between the components of these vectors. The method is called Barlow Twins, owing to neuroscientist H. Barlow's redundancy-reduction principle applied to a pair of identical networks. Barlow Twins does not require large batches nor asymmetry between the network twins such as a predictor network, gradient stopping, or a moving average on the weight updates. Intriguingly it benefits from very high-dimensional output vectors. Barlow Twins outperforms previous methods on ImageNet for semi-supervised classification in the low-data regime, and is on par with current state of the art for ImageNet classification with a linear classifier head, and for transfer tasks of classification and object detection.

1. Introduction

The introduction frames self-supervised vision learning as competitive with supervised representations but vulnerable to trivial constant solutions. It presents Barlow Twins, which applies redundancy reduction to address this problem and reports strong ImageNet and transfer performance.

  • Self-supervised learning seeks useful input representations without human annotations and has produced representations competitive with supervised methods.
  • Similarity-based Siamese methods can admit trivial constant representations, so existing approaches require additional mechanisms to learn useful representations.
  • Barlow Twins applies neuroscience’s redundancy-reduction principle to self-supervised learning.The principle recodes redundant sensory inputs into a factorial code with statistically independent components.
  • Barlow Twins strongly benefits from very high-dimensional embeddings.
  • 55% top-1 accuracy for 1% labels is reported on ImageNet semi-supervised classification in the low-data regime.The method is also described as on par with current state of the art for ImageNet linear evaluation and several classification and object-detection transfer tasks.

2. Method

Barlow Twins learns from two augmented views of each image by matching their cross-correlation matrix to the identity. Its loss combines distortion invariance with redundancy reduction, while the implementation uses a ResNet-50 encoder and projector to produce embeddings for optimization.

  • Method: Barlow Twins generates two augmented views of every image in a batch and processes them with the same function fθ.The views are sampled from a distribution of data augmentations and fed to identical networks.
  • Loss function: The cross-correlation matrix C is computed between outputs of the two identical networks along the batch dimension and has one dimension per output component.Its entries range from -1 for perfect anti-correlation to 1 for perfect correlation.
  • Loss function: The loss drives diagonal cross-correlation values toward 1 for invariance and off-diagonal values toward 0 for redundancy reduction.The invariance term makes embeddings robust to applied distortions, while the redundancy term decorrelates their vector components.
  • Implementation: Each input is transformed twice using cropping, resizing to 224 × 224, flipping, color jittering, grayscale conversion, blurring, and solarization.Cropping and resizing are always applied; the remaining transformations are randomized with specified probabilities.
  • Implementation: The architecture uses a ResNet-50 encoder followed by a three-layer projector, with 2048 encoder output units and 8192 output units in each projector layer.The encoder outputs representations for downstream tasks, while the projector outputs embeddings used by the Barlow Twins loss.

3. Results

Barlow Twins representations are evaluated through ImageNet linear and semi-supervised learning, plus transfer to image classification, object detection, and instance segmentation. The method achieves 73.2% ImageNet top-1 accuracy and competitive or better transfer performance against prior representation-learning methods.

  • Evaluation setup: Representations are pretrained without labels on ImageNet ILSVRC-2012 and evaluated across classification, object detection, and fixed-feature or fine-tuned settings.The evaluation follows standard transfer-learning practice across different datasets and computer-vision tasks.
  • Linear evaluation on ImageNet: 73.2% top-1 accuracy is achieved on ImageNet under linear evaluation using fixed ResNet-50 representations, comparable to state-of-the-art methods.A linear classifier is trained on top of a ResNet-50 pretrained with Barlow Twins.
  • Semi-supervised training on ImageNet: On ImageNet semi-supervised training with 1% and 10% of the data, Barlow Twins is slightly better than competing methods at 1% and on par at 10%.The ResNet-50 is fine-tuned using the same data splits as SimCLR.
  • Image classification with fixed features: For fixed-feature image classification on Places-205, VOC07, and iNaturalist2018, Barlow Twins performs competitively and outperforms SimCLR and MoCo-v2 on most datasets.The evaluation covers scene, multi-label, and fine-grained image classification.
  • Object Detection and Instance Segmentation: For object detection and instance segmentation on VOC07+12 and COCO, Barlow Twins performs comparably or better than state-of-the-art representation-learning methods.The representations are evaluated with fine-tuned ConvNet parameters using Faster R-CNN and Mask R-CNN setups.

4. Ablations

The ablations show that both invariance and redundancy-reduction terms are necessary, while BARLOW TWINS remains robust to small batches but depends on its augmentations. Its performance improves with wider projector outputs and deeper projector networks, without requiring symmetry breaking.

  • Baseline: The 300-epoch BARLOW TWINS baseline reaches 71.4% top-1 and 90.2% top-5 ImageNet accuracy using linear classifiers on 2048-dimensional res5 features.All ablations use this evaluation protocol and shortened training schedule.
  • Loss Function Ablations: Removing either the invariance or redundancy-reduction term substantially reduces performance, while alternative normalization schemes and a cross-entropy loss also perform worse.The loss-function ablations test the necessity of each proposed term and practices used in other SSL objectives.
  • Robustness to Batch Size: BARLOW TWINS performance is almost unaffected by reducing batch size, unlike the batch-size sensitivity reported for SIMCLR.Learning rates are separately tuned with a LARS grid search for each batch size.
  • Effect of Removing Augmentations: Removing augmentation types reduces BARLOW TWINS robustness, unlike BYOL, indicating that its learned invariances are more controlled by the selected distortions.The paper characterizes this dependence as a disadvantage relative to BYOL, while noting that it may provide tighter control over representations.
  • Projector Network Depth & Width: BARLOW TWINS keeps improving as projector output dimensionality increases, whereas other methods rapidly saturate; performance also saturates at three projector layers.The ResNet output remains fixed at 2048 dimensions, making the benefit of larger projector outputs notable.
  • Breaking Symmetry: Because the loss avoids trivial solutions by construction with symmetric networks, adding a predictor network tests whether asymmetry can further improve performance.The predictor consists of two fully connected layers of size 8192 and follows settings used by SIMSIAM and BYOL.

5. Discussion

Barlow Twins learns invariant, redundancy-reduced representations without large batches or asymmetric twin networks. Compared with InfoNCE and asymmetric alternatives, it estimates embedding variability in a way that supports small batches and very high-dimensional outputs while avoiding trivial solutions by construction.

  • Core objective: Barlow Twins maximizes similarity between embeddings of distorted images while reducing redundancy between embedding components, without requiring large batches or asymmetric twin-network structures.Its objective combines invariance and redundancy reduction through batch statistics.
  • Comparison with InfoNCE: Both Barlow Twins and InfoNCE combine invariance with embedding-variability objectives and rely on batch statistics, but InfoNCE maximizes pairwise distances between samples.InfoNCE’s variability term can be interpreted as a non-parametric entropy estimator vulnerable to the curse of dimensionality.
  • Comparison with InfoNCE: Barlow Twins estimates embedding variability under a Gaussian parametrization, enabling reliable estimation from fewer samples and very large-dimensional embeddings.Ablations find robustness to small batches unlike SimCLR and benefits from very large output dimensionality unlike InfoNCE-based methods.
  • Asymmetric twins: Unlike BYOL and SimSiam, which avoid collapse through architectural or optimization asymmetry, Barlow Twins avoids trivial solutions by construction.The paper characterizes this as conceptually simpler and more principled than alternatives relying on implementation choices or non-trivial learning dynamics.
  • Whitening methods: Barlow Twins encourages whitening through a soft redundancy-reduction constraint, whereas W-MSE explicitly whitens each batch using a differentiable Cholesky-based operation.Both methods operate on embeddings from identical twin networks before comparing them.
  • Embedding dimensionality: Performance improves steadily as Barlow Twins increases projector-output dimensionality, contrasting with SimCLR and BYOL, whose gains rapidly saturate.The paper identifies exploration of even higher-dimensional embeddings as a promising direction.

A. Connection between BARLOW TWINS and the Information Bottleneck Principle

BARLOW TWINS can be interpreted as an information-bottleneck objective that preserves information about image samples while discarding information about their distortions. Its practical loss approximates this objective through alignment and redundancy reduction, with several simplifying substitutions.

  • Information Bottleneck Principle: The information bottleneck frames SSL as learning representations informative about samples but invariant to their distortions.Here, distortions correspond to the data augmentations applied to each sample.
  • BARLOW TWINS Loss: BARLOW TWINS’ loss contains an invariance term and a redundancy-reduction term weighted by the positive constant λ.The cross-correlation matrix C is computed between identical networks’ outputs across batch samples and has entries from -1 to 1.
  • Approximations and Limitations: BARLOW TWINS replaces direct covariance-determinant optimization with minimizing the Frobenius norm of the cross-correlation matrix as a proxy for maximizing information about samples.This substitution assumes representations are rescaled to 1 along the batch dimension before entering the loss.
  • Approximations and Limitations: For β <= 1, the IB trade-off favors a constant representation; for β > 1, the coefficient 1−β is replaced by a positive λ with a negative sign.The constant solution is therefore considered uninteresting for representation learning.
  • Approximations and Limitations: The IB formulation would use an auto-correlation matrix, but experiments found no strong performance difference from the cross-correlation matrix used by BARLOW TWINS.The alignment term in BARLOW TWINS also has the same global optimum as minimizing information about distortions in the IB objective.

B. Evaluations on ImageNet · B.1. Linear evaluation on ImageNet

The ImageNet linear evaluation trains a classifier with SGD for 100 epochs, using cosine learning-rate decay and standard image augmentations. The protocol uses 224 × 224 crops for training and center-cropped 224 × 224 inputs at test time.

  • B.1. Linear evaluation on ImageNet: The linear classifier is trained for 100 epochs with a learning rate of 0.3 and cosine scheduling.
  • B.1. Linear evaluation on ImageNet: Training minimizes cross-entropy loss with SGD, momentum, and weight decay of 10^-6.
  • B.1. Linear evaluation on ImageNet: The evaluation uses a batch size of 256.
  • B.1. Linear evaluation on ImageNet: During training, each image receives a random crop resized to 224 × 224.
  • B.1. Linear evaluation on ImageNet: Training augmentation optionally flips images horizontally.
  • B.1. Linear evaluation on ImageNet: At test time, images are resized to 256×256 and center-cropped to 224×224.

B.2. Semi-supervised training on ImageNet · C. Transfer Learning

The semi-supervised ImageNet training uses a 20-epoch SGD schedule with staged learning-rate reductions, fixed batch size, and augmentations matching linear evaluation.

  • B.2. Semi-supervised training on ImageNet: 20 epochs are used for semi-supervised ImageNet training.The ResNet-50 learning rate is 0.002, while the final classification layer uses 0.5.
  • B.2. Semi-supervised training on ImageNet: The learning rate is multiplied by 0.2 after epochs 12 and 16.This schedule applies to the stated training setup.
  • B.2. Semi-supervised training on ImageNet: Training minimizes cross-entropy loss with SGD and momentum.Weight decay is not used.
  • B.2. Semi-supervised training on ImageNet: The batch size is 256.This batch size is used throughout the described semi-supervised training procedure.
  • B.2. Semi-supervised training on ImageNet: Image augmentations are the same as those used in the linear evaluation setting.The augmentation policy is therefore shared between these two settings.
  • B.2. Semi-supervised training on ImageNet: The final classification layer uses a learning rate of 0.5, compared with 0.002 for ResNet-50.Both learning rates are specified for the 20-epoch training procedure.

C.1. Linear evaluation

Linear evaluation follows PIRL’s exact settings across Places-205, VOC07, and iNaturalist2018. Places-205 and iNaturalist2018 use SGD-trained linear classifiers, while VOC07 uses SVM classifiers with cross-validated C values.

  • Evaluation protocol: Evaluation uses PIRL’s exact settings on Places-205, VOC07, and iNaturalist2018.The protocol covers linear-classifier evaluation across all three datasets.
  • Evaluation protocol: Places-205 and iNaturalist2018 use linear classifiers trained with SGD.Training lasts 14 epochs on Places-205 and 84 epochs on iNaturalist2018.
  • Evaluation protocol: The learning rate is 0.01, reduced tenfold at two equally spaced intervals, with weight decay of 5 × 10−4 and momentum of 0.9.These optimization settings apply to the SGD-trained classifiers.

C.2. Object Detection and Instance Segmentation

The evaluation initializes Faster R-CNN and Mask R-CNN ResNet50 backbones with a pretrained Barlow Twins model and follows established Detectron2 evaluation settings. Experiments cover VOC07+12 Faster R-CNN training and COCO 2017 Mask R-CNN evaluation.

  • Model setup: Faster R-CNN and Mask R-CNN ResNet50 backbones are initialized with the pretrained Barlow Twins model using Detectron2 evaluation settings.The detection models closely follow He et al. (2019).
  • VOC07+12: VOC07+12 uses 16K trainval images to train a Faster R-CNN C-4 backbone for 24K iterations with batch size 16 across 8 GPUs.Training uses SyncBatchNorm, a learning rate of 0.1, step reductions after 18K and 22K iterations, and 1000-iteration linear warmup.
  • COCO: COCO experiments train a Mask R-CNN C-4 backbone on the 2017 train split and report results on the validation split.The learning rate is 0.03, with other parameters matching Detectron2’s 1× schedule.
Loading 2103.03230v3…