PaperlayerView exact revision on arXiv

Source-linked AI summary

VICReg: Variance-Invariance-Covariance Regularization for Self-Supervised Learning

Adrien Bardes, Jean Ponce, Yann LeCun

arXiv:2105.04906v3cs.CVcs.AIcs.LG

TL;DR

Self-supervised joint-embedding methods can collapse to constant vectors when they maximize agreement between views. VICReg explicitly combines invariance with variance preservation and covariance-based decorrelation, achieving results on par with the state of the art across many downstream tasks.

  • Problem

    Joint-embedding self-supervised methods risk collapse when both branches ignore inputs and produce identical constant vectors.

  • Method

    VICReg combines view invariance with per-dimension variance preservation and covariance regularization applied independently to both branches.

  • Results

    VICReg achieves results on par with the state of the art on many downstream tasks and performs better than Barlow Twins across classification tasks.

  • Takeaways & Limitations

    The variance term also stabilizes other methods, while VICReg’s separate branch regularization supports architectures with different parameters, structures, or input modalities.

Abstract

from arXiv · show

Recent self-supervised methods for image representation learning are based on maximizing the agreement between embedding vectors from different views of the same image. A trivial solution is obtained when the encoder outputs constant vectors. This collapse problem is often avoided through implicit biases in the learning architecture, that often lack a clear justification or interpretation. In this paper, we introduce VICReg (Variance-Invariance-Covariance Regularization), a method that explicitly avoids the collapse problem with a simple regularization term on the variance of the embeddings along each dimension individually. VICReg combines the variance term with a decorrelation mechanism based on redundancy reduction and covariance regularization, and achieves results on par with the state of the art on several downstream tasks. In addition, we show that incorporating our new variance term into other methods helps stabilize the training and leads to performance improvements.

1 INTRODUCTION

Joint-embedding self-supervised methods learn agreement between views but risk collapse to constant outputs. VICReg addresses this through explicit variance preservation and covariance-based decorrelation rather than relying only on implicit architectural biases.

  • Joint-embedding methods produce similar embeddings for different views of the same image, but can collapse to identical constant vectors.
  • Contrastive collapse prevention often requires large batches or memory banks and explicitly separates embeddings from dissimilar images.
  • Information-maximization methods prevent informational collapse by decorrelating embedding variables.
  • Barlow Twins drives the normalized cross-correlation matrix toward identity, while Whitening-MSE whitens embeddings and spreads them over the unit sphere.

2 VICREG: INTUITION

VICReg combines invariance, variance preservation, and covariance regularization in a joint-embedding objective. Its separate branch regularization preserves information while relaxing requirements on architectures, parameters, and input modalities.

  • VICReg uses three terms: invariance aligns embeddings from different views, variance preserves per-dimension variation, and covariance reduces redundancy.
  • The variance term is a hinge loss that keeps each embedding variable’s batch standard deviation above a threshold, making within-batch embeddings different.
  • The covariance term attracts pairwise centered-variable covariances toward zero, decorrelating embedding dimensions and preventing informational collapse from correlated variables.
  • Variance and covariance are applied separately to both branches, preserving information and preventing informational collapse independently.
  • VICReg does not require shared weights, identical architectures, same-nature inputs, memory banks, contrastive samples, large batches, normalization, vector quantization, or a predictor module.
  • The method’s flexible branches support non-contrastive joint embedding for multimodal signals such as video and audio.

3 RELATED WORK

Related self-supervised approaches prevent collapse through contrastive separation, clustering, distillation, or information maximization. VICReg belongs to the information-preserving family while avoiding shared-branch requirements that constrain many alternatives.

  • Contrastive methods bring embeddings of related views together while pushing embeddings of other samples apart, often using identical, weight-sharing Siamese branches.
  • Clustering methods group samples by similarity and can use previous-iteration k-means assignments as pseudo-labels for new representations.
  • Distillation methods avoid collapse by training a student to predict representations from a teacher updated by running averages or gradient blocking.
  • VICReg opens multimodal joint-embedding applications by removing the requirement for shared weights and identical architecture.
  • Information-maximization methods include W-MSE, which whitens embeddings and spreads them on the unit sphere, and Barlow Twins, which targets cross-correlation structure.

4 VICREG: DETAILED DESCRIPTION

VICReg maps two transformed views through an encoder and expander, then minimizes a weighted sum of invariance, variance, and covariance terms. The variance criterion targets per-dimension standard deviation, while covariance penalizes redundancy between embedding dimensions.

  • Two random transformations create views x and x′, which the encoder and expander map to representations y, y′ and embeddings z, z′.
  • The variance regularizer uses a hinge on each embedding dimension’s batch standard deviation, with target γ fixed to 1 in the experiments.
  • Using standard deviation rather than variance avoids vanishing gradients near equal embeddings, which would otherwise allow collapse.
  • The covariance regularizer penalizes squared off-diagonal covariance coefficients, encouraging dimensions to encode nonredundant information.
  • The loss is a weighted sum of invariance, variance, and covariance terms controlled by λ, µ, and ν.
  • The objective is minimized over encoder parameters θ and expander parameters φ on batches from an unlabeled dataset.
  • ImageNet pretraining uses a ResNet-50 encoder with 2048 output units and an expander containing fully connected layers with batch normalization.

5 RESULTS

VICReg representations are evaluated across ImageNet, transfer, and multimodal retrieval settings, where they generally perform competitively with contemporary self-supervised methods. The paper also compares VICReg with Barlow Twins and examines variance/covariance regularization in other methods.

  • Evaluation scope: VICReg is evaluated after ResNet-50 pretraining on ImageNet and on downstream classification, detection, segmentation, and retrieval tasks.The experiments include frozen-representation linear evaluation, semi-supervised classification, transfer learning, and MS-COCO image-text retrieval.
  • Transfer results: VICReg performs on par with most concurrent methods and better than Barlow Twins across classification tasks, while remaining slightly behind the top three on detection tasks.This comparison covers the transfer tasks reported in Table 2.
  • Multimodal retrieval: VICReg compares against VSE++ and Barlow Twins for MS-COCO 5K image-text retrieval using a GRU text encoder and a ResNet-152 image encoder.The multimodal experiment pretrains on paired image and text data.
  • Regularization comparisons: Table 4 measures Top-1 ImageNet accuracy after 100 pretraining epochs when variance and covariance regularization are incorporated into different methods.The comparisons use the original methods’ architectures, optimization, and augmentation protocols in a reimplementation.

6 ANALYSIS

The analysis studies how variance and covariance regularization interact with asymmetric architectures, predictors, stop-gradient, batch normalization, and branch weight sharing. Variance regularization is associated with collapse prevention and small performance gains in several settings, while VICReg supports independently configured branches.

  • Branch regularization: VICReg’s variance and covariance terms regularize the expander outputs separately for each branch rather than the predictor outputs.The paper reports that regularizing the expander output works better empirically.
  • Asymmetric networks: Without variance regularization, representations collapse and both stop-gradient and predictor mechanisms are necessary in the analyzed setting.With variance regularization, adding a predictor to VICReg does not significantly change performance, indicating redundancy between the predictor and variance regularization.
  • Asymmetric networks: Batch normalization in the expander’s inner layers increases VICReg performance by 1.0%, whereas stop-gradient and predictor without batch normalization perform at 35.1%.The paper characterizes the 1.0% increase as modest relative to the poor 35.1% result.
  • Weight sharing: VICReg permits shared weights, different weights, or different architectures between branches, which are compared using Top-1 accuracy after 100 pretraining epochs.The weight-sharing analysis includes ResNet-50, ResNet-101, and ViT-S encoders.
  • Regularization effects: Adding variance regularization to stop-gradient or momentum-encoder methods improves performance by 0.2% and 0.9%, respectively.The paper suggests these architectural mechanisms may allow very slow collapse because they do not perfectly maintain representation variance.

7 CONCLUSION

The paper presents VICReg as a three-term self-supervised objective that combines invariance, variance preservation, and covariance regularization. It reports state-of-the-art-level downstream performance without requiring identical or similar embedding branches.

  • Conclusion: VICReg combines invariance to different views, variance preservation to avoid representation collapse, and covariance regularization to maximize representation information.The variance term is the paper’s main contribution, while covariance regularization addresses redundancy between embedding variables.
  • Conclusion: VICReg achieves results on par with the state of the art on many downstream tasks without requiring identical or similar embedding branches.The conclusion emphasizes that the method is not subject to limitations imposed by identical branch structure.

B RELATION TO OTHER SELF-SUPERVISED METHODS

VICReg is compared with self-supervised methods that prevent collapse through decorrelation, whitening, asymmetric architectures, or negative-pair comparisons. Its conceptual design combines branch-wise variance and covariance regularization with an invariance loss.

  • Methodological comparison: VICReg compares methods by their mechanisms for avoiding collapse and learning representations, illustrated conceptually in Figure 2.The comparison covers VICReg, Barlow Twins, W-MSE, BYOL, SimSiam, SimCLR, SwAV, and OBoW.
  • Relation to Barlow Twins: VICReg penalizes off-diagonal covariance terms separately within each branch, whereas Barlow Twins uses cross-correlations between branches and requires standardization.VICReg’s variance term removes the need to standardize embeddings and avoids the described numerical edge case without normalization.
  • Relation to W-MSE: W-MSE whitens embeddings using an inverse covariance operator, while VICReg avoids covariance-matrix inversion.The paper identifies inversion as costly and potentially unstable.
  • Relation to BYOL and SimSiam: BYOL and SimSiam use moving-average weights, stop-gradient, or asymmetry as collapse-prevention components, unlike VICReg’s explicit regularization.The paper reports that these components also have a decorrelation effect in experiments.
  • Relation to contrastive and clustering methods: Contrastive and clustering methods compare negative pairs or use batch-derived clusters, whereas VICReg’s conceptual framework uses variance, covariance, and invariance terms.SimCLR requires large batches for its mined negative pairs, while SwAV does not appear to have the same batch-size dependency.

C.1 DATA AUGMENTATION

The evaluation uses the SimCLR image-augmentation protocol, combining two random crops with geometric, photometric, blur, grayscale, solarization, and channel-normalization transforms. Downstream evaluations use specified linear, semi-supervised, classification, detection, segmentation, and implementation protocols.

  • Data augmentation: Two random crops are resized to 224 × 224 and augmented with horizontal flips, color jittering, Gaussian blur, grayscale, and channel normalization.The stated protocol follows the augmentation pipeline introduced in SimCLR.
  • Data augmentation: Random crops use area ratios from 0.08 to 1.0 before resizing, with horizontal flipping applied at probability 0.5.The crop specification lists RandomResizedCrop(224) and the stated flip probability.
  • Data augmentation: Color jittering is applied with probability 0.8, grayscale with probability 0.2, Gaussian blur with probability 0.5, and solarization with probability 0.1.The listed jitter parameters are brightness 0.4, contrast 0.4, saturation 0.2, and hue 0.1; blur uses kernel size 23.
  • Evaluation protocols: Linear and semi-supervised evaluations train classifiers on frozen or fine-tuned representations, while VISSL and detectron2 support classification, detection, and segmentation tasks.The protocols specify optimizer, batch-size, learning-rate, epoch, and augmentation settings for these evaluations.
  • Method-specific setups: The reported BYOL, SimSiam, and VICReg experiments use 100-epoch pretraining settings with method-specific optimizers, learning rates, architectures, and batch-size schedules.The VICReg setup uses base learning rate 0.3, with separate rates specified for the batch-size study.

D.1 OTHER RESNET ARCHITECTURES

VICReg benefits from larger or wider ResNet architectures, with performance depending on representation dimensionality. Narrow ResNet variants improve linear-classification accuracy over the standard ResNet-50, while the largest architecture is competitive under semi-supervised evaluation.

  • Linear classification: 73.2% top-1 accuracy with ResNet-50 rises to 74.7% with Narrow ResNet-50 (x2) and 76.0% with Narrow ResNet-50 (x4).These correspond to improvements of 1.5% and 2.8% over the standard ResNet-50 result.
  • Architecture design: Narrow ResNet widens only bottleneck convolutional layers, unlike standard widening that also increases representation dimensionality.The section compares these two ways of enlarging ResNet architectures.
  • Semi-supervised evaluation: ResNet-200 (x2) performs below BYOL with 1% of labels but is on par with BYOL with 10% of labels.The paper presents these results as evidence that VICReg scales to large architectures.

D.2 PRETRAINING AND EVALUATION ON ESC-50 AUDIO CLASSIFICATION

VICReg is evaluated across audio transfer, loss-term ablations, normalization choices, and architecture settings. The results support complementary roles for invariance, variance, and covariance regularization, while showing that embedding normalization is unnecessary and can reduce performance.

  • ESC-50 audio classification: VICReg jointly embeds raw audio and time-frequency representations on ESC-50, demonstrating use with different architectures and input modalities.The experiment uses the standard ESC-50 split and evaluates frozen representations with a linear classifier.
  • ESC-50 audio classification: VICReg exceeds the supervised baseline by 5.7% and Barlow Twins by 3.0% on ESC-50 validation accuracy.The authors frame the experiment as demonstrating applicability rather than advancing ESC-50 state of the art.
  • Loss-function coefficients: Without variance regularization, representations immediately collapse to one vector; invariance is necessary for learning useful representations, and variance plus covariance have complementary effects.Using only invariance and variance still reaches 57.5% accuracy, while the paper concludes that both variance and covariance are required.
  • Loss-function coefficients: On ImageNet, λ = µ = 25 and ν = 1 provides stable, strong performance, while substantially different coefficient choices can destabilize training.The exact coefficient choice has limited influence on final linear-classification accuracy within the reported stable regime.
  • Normalization: VICReg does not require normalization of embeddings, and standardizing embeddings reduces accuracy by 0.2%.Standardization in expander hidden layers remains useful for training stability, while removing it reduces performance by 1.2%.
  • Normalization: Constraining embeddings to the unit sphere reduces performance by 3.5%.The paper attributes this setting to additional constraints on the network.

D.6 EXPANDER NETWORK ARCHITECTURE

The experiments examine how variance and covariance regularization affect representation statistics, accuracy, convergence, and computational trade-offs in VICReg and related methods.

  • Expander network architecture: 55.9% top-1 accuracy is the reported starting point for the expander-width study, whose performance increases substantially as dimensionality grows.VICReg inherits Barlow Twins’ dependence on expander-network dimensionality.
  • Batch size: 0.7% and 1.2% accuracy drops occur at batch sizes 256 and 128, respectively, indicating limited degradation with smaller batches.The comparison spans batch sizes from 128 to 4096 and uses a learning-rate grid search for each size.
  • Combination with BYOL and SimSiam: 0.9% improvement after 100 epochs and 0.2% after 1000 epochs show that variance regularization accelerates BYOL convergence, with a smaller effect in SimSiam.The comparison uses the original methods’ augmentation and optimization settings in the authors’ implementation.
  • Combination with BYOL and SimSiam: Lower average correlation with covariance regularization in BYOL coincides with a small performance improvement, whereas SimSiam shows neither change.The average correlation coefficient averages off-diagonal correlation-matrix coefficients of the representations.
  • Running time: VICReg offers a running-time and memory trade-off, using less memory than BYOL and multi-crop SwAV while running faster than multi-crop SwAV.The comparison measures 100-epoch training on 32 Tesla V100 GPUs; VICReg has slightly worse performance than the compared alternatives described.
Loading 2105.04906v3…