PaperlayerView exact revision on arXiv

Source-linked AI summary

On the duality between contrastive and non-contrastive self-supervised learning

Quentin Garrido, Yubei Chen, Adrien Bardes, Laurent Najman, Yann Lecun

arXiv:2206.02574v3cs.LGcs.AIcs.CV

TL;DR

The paper asks how closely contrastive and non-contrastive self-supervised learning methods are related, despite their commonly emphasized differences. It compares contrastive and covariance-based criteria, relates popular methods to them, and studies method variations. It reports that careful tuning lets SimCLR match VICReg, while VICReg can gain robustness to embedding dimension.

  • Problem

    Contrastive and non-contrastive self-supervised methods are often presented as different, while their theoretical similarities and relationship to performance gaps remain to be examined.

    Source: §1 · pp. 1–2

  • Method

    The paper defines contrastive and covariance-based criteria, proves equivalence under embedding-normalization assumptions, and studies variations of VICReg and SimCLR.

    Source: §1 · p. 2

  • Results

    Carefully tuned SimCLR matches VICReg, and VICReg is improved to be as robust to embedding dimension as SimCLR.

    Source: §1 · p. 2, §6 · p. 9

  • Takeaways & Limitations

    The results support using a unified view of sample-contrastive and dimension-contrastive methods to understand their behavior and close performance gaps.

    Source: §1 · p. 2, §6 · p. 9

    SimCLR and DCL do not fit the theoretical framework perfectly because their criteria aim for opposite rather than orthogonal negative-pair embeddings.

    Source: §3 · pp. 3–4

Abstract

from arXiv · show

Recent approaches in self-supervised learning of image representations can be categorized into different families of methods and, in particular, can be divided into contrastive and non-contrastive approaches. While differences between the two families have been thoroughly discussed to motivate new approaches, we focus more on the theoretical similarities between them. By designing contrastive and covariance based non-contrastive criteria that can be related algebraically and shown to be equivalent under limited assumptions, we show how close those families can be. We further study popular methods and introduce variations of them, allowing us to relate this theoretical result to current practices and show the influence (or lack thereof) of design choices on downstream performance. Motivated by our equivalence result, we investigate the low performance of SimCLR and show how it can match VICReg's with careful hyperparameter tuning, improving significantly over known baselines. We also challenge the popular assumption that non-contrastive methods need large output dimensions. Our theoretical and quantitative results suggest that the numerical gaps between contrastive and non-contrastive methods in certain regimes can be closed given better network design choices and hyperparameter tuning. The evidence shows that unifying different SOTA methods is an important direction to build a better understanding of self-supervised learning.

1 INTRODUCTION

The introduction frames self-supervised image representation learning around sample-contrastive and non-contrastive methods, then develops their theoretical and empirical connections. The paper relates covariance-based methods to contrastive learning across embedding dimensions and studies whether their performance gaps can be closed.

1 INTRODUCTION

  • Self-supervised image representation methods are commonly grouped into contrastive and non-contrastive approaches within the joint-embedding framework.Both attract embeddings from different views of the same image; contrastive methods push apart embeddings from different images, while non-contrastive methods avoid explicit negative pairs.Source: §1 · p. 1
  • The paper defines dimension-contrastive methods as covariance-based approaches that contrast embedding dimensions, and sample-contrastive methods as approaches that contrast samples.It relates Frobenius-norm criteria for Gram and covariance matrices and proves equivalence under assumptions on embedding normalization.Source: §1 · p. 2
  • The authors introduce VICReg and SimCLR variations to isolate loss-function components and test the theoretical connection empirically.They report that SimCLR can match VICReg with careful hyperparameter tuning, while VICReg can be made as robust to embedding dimension as SimCLR.Source: §1 · p. 2
  • The paper argues that empirical performance gaps between several state-of-the-art sample-contrastive and dimension-contrastive methods can be closed.It presents this unification as a way to consolidate understanding of self-supervised learning methods.Source: §1 · p. 2

2 RELATED WORK

Related work spans sample-contrastive methods, approaches that avoid negative samples, and theoretical efforts to characterize or connect self-supervised learning objectives. The paper situates its contribution among prior links between these objectives and other frameworks.

2 RELATED WORK

  • Many sample-contrastive methods use InfoNCE, while HaoChen et al. (2021) use squared sample similarities; clustering methods can be viewed as contrastive between prototypes.Examples of sample-contrastive work include Chen et al. (2020a), He et al. (2020), and Chen et al. (2021b).Source: §2 · p. 2
  • Non-contrastive methods avoid negative samples through architectural designs such as BYOL (Grill et al., 2020), SimSiam (Chen & He, 2020), and DINO (Caron et al., 2021), or through covariance regularization.The paper focuses on covariance-based information-maximization methods, which regularize empirical embedding covariance to maximize informational content.Source: §2 · p. 2
  • Prior work characterizes contrastive objectives, studies optimization and loss landscapes, and links Barlow Twins to an upper bound of a sample-contrastive criterion; this paper says it quantifies that gap exactly.The passage also notes links between Barlow Twins and HSIC (Tsai et al., 2021).Source: §2 · pp. 2–3
  • Other unifying analyses connect self-supervised methods to spectral methods, gradient behavior, or CCA and sample-contrastive learning error bounds.The cited examples include Balestriero & LeCun (2022), Tao et al. (2021), and Lee et al. (2021a).Source: §2 · p. 3

3 EQUIVALENCE OF THE CONTRASTIVE AND NON-CONTRASTIVE CRITERION

The section connects sample-contrastive and dimension-contrastive criteria, showing they are equivalent when the embedding matrix is normalized across both rows and columns. It relates the criteria to popular methods and argues that practical differences also reflect optimization and implementation choices.

3 EQUIVALENCE OF THE CONTRASTIVE AND NON-CONTRASTIVE CRITERION

  • Lc penalizes off-diagonal similarities between samples, while Lnc penalizes off-diagonal covariance between embedding dimensions; both seek orthogonality.The paper calls methods minimizing these criteria sample-contrastive and dimension-contrastive, respectively.Source: §3 · p. 3
  • Original SimCLR and DCL push negative pairs toward opposites rather than orthogonality, but their negative-pair dot products have null mean; squared- and absolute-cosine variants do not change performance.SimCLR-abs reaches 68.71% ImageNet top-1 accuracy with linear evaluation, versus 68.61% for the improved SimCLR reproduction.Source: §3 · pp. 3–4
  • With both rows and columns of K L2-normalized, Lnc = Lc + N − M, making the criteria equivalent from an optimization perspective.Without this double normalization, the theorem relates the criteria but does not establish that they are interchangeable.Source: §3 · p. 4
  • Normalization strategy affects whether methods normalize embeddings or dimensions, but practical criteria cannot be arbitrarily far apart, and normalization strategy does not affect SimCLR performance.The paper reports that practical norm sums are within a single-digit factor of their lower bounds; the stated bounds use collapsed representations as extrema.Source: §3 · pp. 4–5
  • Applying VICReg’s covariance criterion to transposed embeddings makes it sample-contrastive while retaining its invariance and variance criteria.The rewriting adds a norm regularizer; the paper presents it as evidence that behavioral differences are not mainly due to whether a method is contrastive.Source: §3 · pp. 5–6

4 INTERPOLATING BETWEEN METHODS: IMPACT OF THE LOSS FUNCTION.

The section introduces VICReg variants to isolate the effects of loss-function design choices. In the reported comparisons, the VICReg variants perform similarly, while tuning improves SimCLR to VICReg’s level.

4 INTERPOLATING BETWEEN METHODS: IMPACT OF THE LOSS FUNCTION.

  • Some design choices negatively affect embedding optimization, but the authors report no easily visible differences in the representations used in practice.Source: §4 · p. 6
  • The two VICReg variants are designed to isolate loss components: one adds an InfoNCE-inspired repulsive force, and the other also transposes embeddings to make the criterion contrastive.The variants compare LogSumExp with sum-of-squares repulsion and sample-contrastive with dimension-contrastive criteria.Source: §4 · p. 6
  • VICReg, VICReg-exp, and VICReg-ctr perform similarly after 100 epochs; tuned SimCLR reaches VICReg’s level, unlike the significantly worse original implementation.The figure caption also reports that projector architectures affect performance, while behaviors are similar across methods.Source: Figure 1 · p. 7

5 PRACTICAL DIFFERENCES BETWEEN SAMPLE-CONTRASTIVE AND DIMENSION-CONTRASTIVE METHODS

The experiments test how loss implementation, projector architecture, hyperparameters, and normalization affect practical comparisons between sample-contrastive and dimension-contrastive methods. They find that careful tuning and projector choices can close performance gaps, while normalization changes do not alter SimCLR’s peak performance.

5 PRACTICAL DIFFERENCES BETWEEN SAMPLE-CONTRASTIVE AND DIMENSION-CONTRASTIVE METHODS

  • The experiments compare d −d −d, 2048 −d, and 8192 −8192 −d projectors using an online ImageNet linear-classifier proxy that correlates almost perfectly with offline evaluation.The online classifier is trained alongside the backbone and projector without sending its criterion’s gradients back through the backbone.Source: §5 · p. 7
  • With larger projectors, VICReg and VICReg-exp achieve 68.13% and 68.00%, respectively, while VICReg-exp and VICReg-ctr show no noticeable performance gap.The authors treat this controlled comparison as empirical validation of their theoretical equivalence result.Source: §5 · p. 7
  • Using original hyperparameters, VICReg-ctr achieves 67.92% top-1 accuracy versus SimCLR’s 66.33%.The authors use this gap to motivate investigating SimCLR’s suboptimal performance.Source: §5 · p. 8
  • With a temperature of 0.15 and base learning rate of 0.5, SimCLR reaches 68.6% top-1 accuracy, matching VICReg’s performance.The tuned SimCLR uses the VICReg projector.Source: §5 · p. 8
  • For every studied method, moving from a 2048 −d to an 8192 −8192 −d projector boosts performance; VICReg and VICReg-ctr gain 3.5 −4 points.The d −d −d projector falls between those architectures depending on embedding dimension.Source: §5 · p. 8
  • At 256 dimensions, tuned VICReg reaches 61.36% top-1 accuracy versus the initially reported 55.9%; a bigger projector raises it to 65.01%.Peak performance occurs at 1024 dimensions, below the representation dimension of 2048.Source: §5 · p. 8

5.1 INFLUENCE OF THE NORMALIZATION STRATEGY

  • Figure 2 shows centering and dimension standardization leave SimCLR’s peak unchanged; at 1,000 epochs, standardized SimCLR reaches 72.6% versus VICReg’s 73.3% top-1 accuracy.Dimension standardization normalizes along dimensions rather than embeddings and is the scheme used by VICReg.Source: Figure 2 · p. 9, §5.1 · p. 9

6 CONCLUSION

The paper argues that sample-contrastive and dimension-contrastive methods have closely related objectives, equivalent up to row and column normalization. Its experiments report that tuning can close SimCLR’s performance gap with VICReg, while normalization strategy has little effect on performance.

6 CONCLUSION

  • Sample-contrastive and dimension-contrastive objectives are equivalent up to row and column normalization of the embedding matrix.The paper describes the two method families as two sides of the same coin.Source: §6 · p. 9
  • Careful hyperparameter tuning closes SimCLR’s performance gap with VICReg.The authors report that this significantly improves over known baselines.Source: §6 · p. 9
  • The normalization strategy does not play an important role in performance, and the studied design choices have little impact on it.Variations of VICReg were used to study popular loss-function design choices and improve robustness to embedding dimension.Source: §6 · p. 9
  • The authors argue that unifying state-of-the-art self-supervised learning methods can help consolidate understanding of the field.They also expect the results to support theoretical analyses across a wider family of methods.Source: §6 · p. 9

8 REPRODUCIBILITY STATEMENT

The appendix provides reproducibility materials, recalls the methods’ loss functions, and discusses links between popular methods and the paper’s sample- and dimension-contrastive framework.

8 REPRODUCIBILITY STATEMENT

  • Complete hyperparameter values, pseudocode for VICReg-ctr and VICReg-exp, and numerical performance for the main figure are provided to support reproduction.The authors report that pretraining took around a day on eight V100 GPUs.Source: §8 · p. 10

A BACKGROUND

  • The background section recalls the loss functions considered in the theoretical analysis.The section includes TCR, DCL, SimCLR, Spectral Contrastive Loss, Barlow Twins, and VICReg.Source: A BACKGROUND · p. 13, C PROOFS · p. 17

B LINKS BETWEEN METHODS AND OUR CRITERIA

  • Analyses link BYOL and SimSiam training dynamics to variance and covariance regularization, but rely on assumptions such as encoder linearity.These links help relate their behavior to the paper’s framework despite the methods not explicitly avoiding collapse through their criteria.Source: B LINKS BETWEEN METHODS AND OUR CRITERIA · p. 14
  • MoCo remains related to sample-contrastive methods through its InfoNCE criterion, while clustering methods use prototypes as negative-pair references.The authors characterize links to MoCo and clustering methods as less rigorous; most methods can still be related to one of the two families.Source: B LINKS BETWEEN METHODS AND OUR CRITERIA · p. 14

C PROOFS

The proofs classify several popular criteria as sample- or dimension-contrastive and establish an equivalence between the two prototype criteria under normalization. They also identify specific conditions and mechanisms behind these relationships.

C PROOFS

  • With infinitely many negative samples, the proof uses the result that InfoNCE’s repulsive force yields uniformly distributed embeddings on the hypersphere.This distribution lets the authors apply their random-sphere dot-product result to SimCLR and DCL.Source: C PROOFS · p. 15
  • SimCLR-abs/sq and DCL-sq/abs, Spectral Contrastive Loss, Barlow Twins, VICReg, and TCR are classified as sample- or dimension-contrastive methods.The proposition assigns the first three to sample-contrastive methods and the latter three to dimension-contrastive methods.Source: C PROOFS · p. 15
  • For DCL-sq/abs, either repulsive-force choice leads to a diagonal Gram matrix, the goal shared with the sample-contrastive criterion.Penalizing one Gram-matrix value per row or column is less effective than penalizing all values at once, but converges to the same solution given sufficient training iterations.Source: C PROOFS · p. 16
  • SimCLR-sq/abs minimizes negative-pair similarities, yielding a diagonal Gram matrix and thereby minimizing the sample-contrastive criterion.The proof notes that the positive pair in SimCLR’s repulsive force prevents using the same max-operator reasoning as for DCL.Source: C PROOFS · p. 16
  • TCR’s criterion leads to a diagonal covariance matrix, placing it in the dimension-contrastive category.The text relates this outcome to null off-diagonal covariance terms.Source: C PROOFS · p. 17
  • The sample-contrastive and dimension-contrastive criteria are equivalent up to row and column normalization of the embedding matrix.The theorem states this result for batch size N and embedding dimension M.Source: C PROOFS · p. 17

D TRAINING PROCEDURE

The training procedure uses a ResNet-50 with LARS and a 100-epoch schedule, adapting the base protocol for each method. The appendix identifies the augmentation parameters and reports the computational cost.

D TRAINING PROCEDURE

  • Experiments use ResNet-50 with LARS, a base learning rate of 0.3 scaled by batch size, momentum 0.9, weight decay 10^-6, and 100 training epochs.The learning rate follows cosine annealing after a 10-epoch linear warmup.Source: D TRAINING PROCEDURE · p. 18
  • Table S1 lists the image-augmentation parameters taken from Grill et al. (2020).Source: Table S1 · p. 18
  • Each experiment ran on eight Nvidia V100 GPUs with 32 GB of memory each and took around 24 hours.Source: D TRAINING PROCEDURE · p. 18
  • The base protocol was adapted for each method, mainly through method-specific hyperparameters and learning rates.The authors point to supplementary hyperparameter values and pseudocode for VICReg-exp and VICReg-ctr.Source: D TRAINING PROCEDURE · p. 18

E ONLINE LINEAR PROBE

The section tests whether a jointly trained online linear classifier can stand in for offline linear evaluation. For VICReg and VICReg-exp, the evaluations are reported to be closely aligned, while the online approach costs far less compute.

E ONLINE LINEAR PROBE

  • The offline evaluation trained a linear classifier on frozen VICReg and VICReg-exp representations, using projectors with output dimensions from 256 to 8192.The online classifier was trained jointly with the main network, on suboptimal representations and stronger augmentations than typical linear evaluation.Source: E ONLINE LINEAR PROBE · pp. 18–19
  • Offline linear evaluation was almost as long as pretraining, whereas the online classifier had negligible compute cost and showed very correlated performance.The offline evaluation also required significant learning-rate tuning.Source: E ONLINE LINEAR PROBE · p. 19
  • A linear fit between online and offline evaluations had slope 0.97, intercept 2.1, and R2 of 1.0.The authors note that most values are close to 68, making the fit sensitive to noise.Source: E ONLINE LINEAR PROBE · p. 19

F ADDITIONAL EVIDENCE OF THE SIMILARITY OF LEARNED

The section compares sample- and dimension-contrastive methods using k-NN performance and CKA. Across these analyses, the authors report preserved performance in the transition between method variants and highly similar learned representations.

k-nn evaluation

  • Tuned SimCLR recovered VICReg-level k-NN performance with a 5.5-point increase, while the VICReg-exp-to-VICReg-ctr transition preserved peak performance.The original SimCLR implementation performed significantly worse than other methods.Source: F ADDITIONAL EVIDENCE OF THE… · pp. 19–20
  • k-NN results retained the pattern seen previously: moving from dimension-contrastive to sample-contrastive did not cause a significant performance drop.The best performance across tested k values and classifier temperatures was used for comparison.Source: F ADDITIONAL EVIDENCE OF THE… · p. 19, Figure S1 · p. 20

CKA

  • CKA was similarly high within and between methods, indicating that the studied methods learned highly correlated representations across runs and across method families.The authors suggest that differing projectors and augmentations may explain lower cross-method CKA reported by Gwilliam & Shrivastava (2022).Source: F ADDITIONAL EVIDENCE OF THE… · p. 20, Figure S2 · p. 20
  • Taken together, linear classification, k-NN classification, and CKA show that the studied self-supervised methods produce extremely similar representations.Source: F ADDITIONAL EVIDENCE OF THE… · p. 21

G IMPACT OF THE SIMILARITY MEASURE ON SIMCLR

Changing SimCLR’s cosine-similarity measure altered dimensional collapse and similarity-distribution spread without materially changing classification performance. In practical embedding matrices, row- or column-normalization bounds were much closer than their worst-case theoretical gaps suggest.

G IMPACT OF THE SIMILARITY MEASURE ON SIMCLR

H ROW AND COLUMN NORMS INTERPLAY

  • For tested methods and projector settings, empirical row- or column-normalized norms stayed within a factor of three of the lower bound, making practical bounds closer than the worst-case factors N or M.The authors infer that practical embedding matrices are closer to a constant matrix with uniformly spread information, though they retain some sparsity.Source: H ROW AND COLUMN NORMS INTERPLAY · p. 22, Table S4 · p. 22

I IMPACT OF THE PROJECTOR CAPACITY

Projector size and architecture both relate to downstream performance, while optimization differences among loss variants in embedding space become less distinct in representation space. The appendices provide complete performance and hyperparameter tables alongside implementation pseudocode.

I IMPACT OF THE PROJECTOR CAPACITY

  • An 8192−8192−d projector behaved similarly to 2048−2048−d but performed somewhat better, with the drop especially noticeable for dimension-contrastive methods.Parameter count alone was not sufficient: VICReg with 2048−2048−1024 reached 66.68% top-1 using 10 million parameters, while 8192−8192−256 reached 65.01% using 86 million.Source: I IMPACT OF THE PROJECTOR CAPACITY · p. 23
  • Increasing projector parameter count generally increased performance across methods, though the 2048−256 VICReg and VICReg-exp projectors were clear low-performing outliers.Source: I IMPACT OF THE PROJECTOR CAPACITY · p. 23, Figure S7 · p. 23

J INFLUENCE OF LOSS FUNCTION DESIGN ON OPTIMIZATION QUALITY

  • In embedding space, VICReg, VICReg-exp, VICReg-ctr, and SimCLR all produced Gram and covariance matrices dominated by their diagonals, but with different off-diagonal noise.The authors associate noise mainly with LogSumExp and also with the sample-contrastive nature of VICReg-ctr and SimCLR; representation-space differences were less distinct.Source: J INFLUENCE OF LOSS FUNCTION DESIGN ON… · p. 24, Figure S8 · p. 25

K COMPLETE PERFORMANCE AND HYPERPARAMETER TABLES

  • The appendix tables provide ImageNet online-classifier performance for figures 1 and S6 and the corresponding hyperparameters for those results.Source: Table S5 · p. 26, Table S6 · p. 26

L VICREG VARIATIONS PSEUDOCODE

  • The VICReg-ctr pseudocode processes two randomly augmented views through an encoder and projector, then computes covariance losses from the embeddings.The covariance penalty uses a LogSumExp over off-diagonal covariance terms.Source: L VICREG VARIATIONS PSEUDOCODE · p. 28

Related papers

Loading 2206.02574v3…