PaperlayerView exact revision on arXiv

Source-linked AI summary

Contrastive and Non-Contrastive Self-Supervised Learning Recover Global and Local Spectral Embedding Methods

Randall Balestriero, Yann LeCun

arXiv:2205.11508v3cs.LGcs.AIcs.CVmath.SPstat.ML

TL;DR

SSL lacks a unified, principled account of how its methods relate to representations and downstream performance. The paper uses spectral manifold learning to analyze VICReg, SimCLR, and BarlowTwins, showing shared spectral structure and method-dependent behavior under relation-matrix alignment or misspecification.

  • Problem

    Recent empirical advances in SSL have outpaced principled theoretical understanding, leaving limited guidance on how methods and pairwise relations affect representations and downstream tasks.

  • Method

    The paper unifies VICReg, SimCLR, and BarlowTwins with spectral embedding methods and derives closed-form optimal representations and linear-network parameters from the relation matrix G.

  • Results

    All SSL methods align optimal representations’ top left singular vectors with G; downstream optimality holds when G and Y have aligned left spectra, while VICReg can remain full-rank when G is misspecified.

  • Takeaways & Limitations

    When G matches the downstream task, any analyzed SSL method can produce an ideal representation; when G may be misaligned, VICReg with lower invariance is preferred over SimCLR or BarlowTwins.

  • Takeaways & Limitations

    The BarlowTwins formulation is only well-defined when K < min(Da, Db), motivating a ridge-type CCA formulation.

Abstract

from arXiv · show

Self-Supervised Learning (SSL) surmises that inputs and pairwise positive relationships are enough to learn meaningful representations. Although SSL has recently reached a milestone: outperforming supervised methods in many modalities\dots the theoretical foundations are limited, method-specific, and fail to provide principled design guidelines to practitioners. In this paper, we propose a unifying framework under the helm of spectral manifold learning to address those limitations. Through the course of this study, we will rigorously demonstrate that VICReg, SimCLR, BarlowTwins et al. correspond to eponymous spectral methods such as Laplacian Eigenmaps, Multidimensional Scaling et al. This unification will then allow us to obtain (i) the closed-form optimal representation for each method, (ii) the closed-form optimal network parameters in the linear regime for each method, (iii) the impact of the pairwise relations used during training on each of those quantities and on downstream task performances, and most importantly, (iv) the first theoretical bridge between contrastive and non-contrastive methods towards global and local spectral embedding methods respectively, hinting at the benefits and limitations of each. For example, (i) if the pairwise relation is aligned with the downstream task, any SSL method can be employed successfully and will recover the supervised method, but in the low data regime, VICReg's invariance hyper-parameter should be high; (ii) if the pairwise relation is misaligned with the downstream task, VICReg with small invariance hyper-parameter should be preferred over SimCLR or BarlowTwins.

1 Introduction

The paper develops a spectral framework to unify SSL methods and address limited, method-specific theoretical foundations. It derives shared representation properties, connects contrastive and non-contrastive learning to global and local spectral embeddings, and studies downstream-task alignment.

  • SSL uses inputs and inter-sample relations rather than labels, motivating theory about how learned representations transfer to downstream tasks.
  • The study unifies VICReg, SimCLR, and BarlowTwins with known spectral embedding methods and identifies settings where methods become provably identical.
  • All SSL methods’ optimal representations align their top left singular vectors with those of G, while leaving Z’s right-singular vectors unconstrained.
  • Any SSL loss is guaranteed to solve a downstream task when the left spectrum of the training relation matrix G aligns with that of the task labels Y.
  • VICReg can be full-rank through loss-hyperparameter selection, whereas SimCLR and BarlowTwins enforce rank(Z) = rank(G), suggesting an advantage for VICReg when G is misspecified.
  • Contrastive methods target metric preservation and suit linear manifolds, whereas non-contrastive methods suit manifolds that are locally linear but globally nonlinear.
  • The framework provides closed-form optimal representations and linear-network parameters, alongside practical guidance for choosing SSL methods.

2 Notations and Background on Self-Supervised Learning

SSL represents observations together with a pairwise positive-relation matrix and treats model outputs as signals on a relation graph. The paper formalizes VICReg, SimCLR, and BarlowTwins through their embeddings, relation matrices, and spectral representations.

  • SSL operates on input samples X and a known pairwise positive-relation matrix G, commonly represented as a symmetric binary matrix with zero diagonal.
  • Augmented views are stacked into X, with corresponding samples across views treated as similar and encoded by positive entries in G.
  • The embedding matrix Z contains the model feature maps for all samples, with rows given by fθ(x_n).
  • VICReg: VICReg combines variance, covariance, and invariance losses, with computational cost dominated by O(NK^2) when the number of positives is small.
  • SimCLR: SimCLR estimates relations from embedding cosine similarities and matches them to G using InfoNCE, requiring O(N^2) pairwise-similarity computation.
  • BarlowTwins: BarlowTwins compares two views through their K × K cross-correlation matrix, penalizing deviations of diagonal entries from one and nonzero off-diagonal entries.
  • Spectral viewpoint: The analysis relies on singular-value decompositions and shows that SSL optimal representations share spectral structure with the relation matrix G.

3 VICReg

VICReg admits closed-form optimal representations whose spectral structure is determined by the relation graph and loss hyperparameters. Its constrained and linear forms recover Laplacian Eigenmaps, Locality Preserving Projections, and Linear Discriminant Analysis, while hyperparameter choice controls rank and task-relevant information.

  • Closed-form optimum: VICReg's closed-form optimum depends only on the relation matrix G and the loss hyperparameters.The optimal representation is characterized through the spectral decomposition of a matrix combining Laplacian terms.
  • Spectral interpretation: The invariance term minimizes graph-signal energy, encouraging representation rows connected by G to remain close and smooth on the graph.This links non-contrastive learning to spectral graph embeddings through the left singular vectors of the representation.
  • Hyperparameter effects: The ratio γ/α controls VICReg's representation, enabling a full-rank solution that still incorporates enough information from G for the downstream task.The representation can range from ignoring G to incorporating G while remaining full-rank or collapsing to G; experiments use N = 256, K = 32, and rank(G) = 8.
  • Optimization and computation: VICReg's optimal representation can be obtained through efficient spectral computation and its optimal linear parameters admit gradient-based convergence toward the analytical solution.The alternative sparse formulation supports efficient top-eigenvector routines, while experiments depict convergence toward the optimal representation and weights.
  • Spectral-method recovery: VICReg recovers Laplacian Eigenmaps under a constrained formulation, with the relation matrix G providing the graph structure.The equivalence holds for a given relation matrix, whose construction is a crucial design choice in both spectral learning and SSL.
  • Linear networks: With a linear network, VICReg recovers Locality Preserving Projections for arbitrary G and Linear Discriminant Analysis for supervised class relations.The optimal weights are obtained analytically from the corresponding generalized eigenvalue problem.

4 SimCLR/NNCLR/MeanShift Solve a Generalized Multidimensional Scaling Problem à la ISOMAP

The paper interprets SimCLR, NNCLR, and MeanShift as estimating and matching graphs, yielding generalized multidimensional scaling solutions akin to ISOMAP. This framework explains their spectral alignment and rank behavior.

  • 4.1 Step 1: SimCLR variants first estimate a similarity graph from representation distances, with regularization preventing the estimate from collapsing to the zero matrix.The estimated graph is optimized so the representation is smooth on it; the regularizer pushes weight entries toward 1 according to temperature.
  • 4.1 Step 1: With cosine distance, the graph-estimation solution recovers SimCLR similarities, while ℓ2 distance recovers the Lifted Structured Loss formulation.
  • 4.2 Step 2: SimCLR matches the estimated graph to the known relation matrix, forcing representation singular vectors to align with G and its rank to match G.
  • 4.2 Step 2: Across distances and regularizers, SimCLR exhibits dimensional collapse, with learned representations inheriting the rank of the relation matrix.Figure 8 reports this behavior across varying rank(G), hyper-parameters, and graph-estimation configurations.
  • 4.3 SimCLR is Akin to ISOMAP in Feature Space and Kernel ISOMAP in Input Space: SimCLR recovers generalized multidimensional scaling, while NNCLR builds a nearest-neighbor graph and thereby connects to ISOMAP; MDS is recovered when G = XXT.

5 BarlowTwins

BarlowTwins is connected to canonical correlation analysis in embedding and data spaces, including kernel CCA in the nonlinear setting. Its representation rank follows the relation matrix, and its linear supervised case recovers LDA.

  • 5.1 BarlowTwins Recovers Kernel Canonical Correlation Analysis: BarlowTwins uses cross-correlation between positive-pair embeddings and is tied to Kernel Canonical Correlation Analysis.
  • 5.1 BarlowTwins Recovers Kernel Canonical Correlation Analysis: The BarlowTwins objective equals the CCA objective up to rescaling of the weight matrices.
  • 5.1 BarlowTwins Recovers Kernel Canonical Correlation Analysis: In the nonlinear regime, BarlowTwins corresponds to nonlinear canonical correlation analysis, particularly Kernel CCA, in data space.
  • 5.1 BarlowTwins Recovers Kernel Canonical Correlation Analysis: The CCA formulation is well-defined only when K < min(Da, Db), motivating ridge-type regularization for numerical stability.
  • 5.1 BarlowTwins Recovers Kernel Canonical Correlation Analysis: BarlowTwins enforces rank(Z) = rank(G), regardless of the initialized representation rank, producing dimensional collapse during optimization.
  • 5.2 With a Linear Network BarlowTwins Recovers Canonical Correlation Analysis and Linear Discriminant Analysis: In the linear regime, BarlowTwins recovers CCA with optimal weights; with supervised G and K = C, it recovers LDA and thus VICReg.

6 Optimality of Self-Supervised Methods to Solve Downstream Tasks

The section characterizes when SSL representations support zero-error downstream prediction through the alignment of the left singular spaces of the learned representation, relation matrix, and task target. It also shows that method choice matters when the relation matrix is misaligned with the downstream task.

  • Representation optimality: A representation achieves zero minimum linear loss exactly when its nonzero left-singular vectors span those of the downstream target matrix.This gives the necessary and sufficient spectral condition for zero training error.
  • Representation optimality: The downstream task is optimally solvable by VICReg, SimCLR, or BarlowTwins when the target matrix's left-singular subspace lies in the span of the relation matrix's left-singular vectors.The result applies to converged embeddings produced from the dataset and relation matrix.
  • Aligned relations: When the relation matrix is correctly aligned with the task target, all three SSL methods can produce an ideal representation, so no method has an advantage.The same conclusion also applies to methods using the SimCLR loss, such as NNCLR and MeanShift, with a changed relation design.
  • Misaligned relations: The analysis bounds downstream performance over relation matrices of fixed rank, comparing the best geometry-aligned case with the worst geometry-orthogonal case.The bounds assume rank(Y) ≤ K and K < N.
  • Misaligned relations: VICReg should be preferred over BarlowTwins and SimCLR when the relation matrix is misaligned, because VICReg preserves full rank while the other methods match the rank of the relation matrix.The rank distinction is irrelevant under correct relations but becomes important in practical misalignment settings.

7 Conclusions

The paper unifies major SSL methods through spectral embedding theory and connects contrastive versus non-contrastive learning with global versus local spectral methods. Its conclusions distinguish method choice under aligned and misaligned similarity relations and motivate transferring results between the fields.

  • Unified analysis: The study provides a unifying analysis of VICReg, SimCLR, and BarlowTwins and ties them to common spectral embedding techniques.The analysis also identifies commonalities across these methods.
  • Aligned similarity: When the similarity matrix is correctly defined for a downstream task, any of the analyzed methods produces an ideal representation that perfectly solves that task.In this regime, the paper finds no benefit of one method over another.
  • Misaligned similarity: With a similarity matrix misaligned to the downstream task, the conclusion favors VICReg with lower invariance regularization because it preserves full rank and information beyond the matrix.SimCLR and BarlowTwins instead collapse the representation to encode the similarity matrix and nothing else.
  • Global and local methods: Contrastive SSL parallels global spectral methods that preserve metric structure, whereas non-contrastive SSL parallels local methods suited to manifolds with locally Euclidean but globally nonlinear geometry.The comparison frames complementary strengths and weaknesses for different manifold structures.
  • Future directions: The spectral connections motivate future work that transfers existing results and techniques between spectral embedding and SSL methods.The paper notes that spectral methods have struggled on high-dimensional datasets where SSL has achieved state-of-the-art performance.

Supplementary Materials

The supplementary materials contain proofs of the paper's main formal results and additional background derivations and references intended to make the analysis self-contained.

  • Supplementary scope: The supplementary materials provide proofs for the main formal results and background results and references throughout the derivations.Some derivations are included to give readers additional insight into current SSL methods.

A Formal Statements Proofs

The appendix supplies algebraic foundations for low-rank least squares, VICReg's graph-Laplacian connection, representation non-uniqueness, and the equivalence between SimCLR and ISOMAP/MDS spectral objectives.

  • A.1 VICReg Variance+Covariance Versus Representation’s Singular Values: VICReg's variance and covariance terms can be related to constraints on the singular values of the representation.The derivation shows an upper-bound formulation that keeps singular values close to 1, while the general criterion requires variance at least 1.
  • A.2 Non-Unique Solution to Least-Square: For a rank-deficient representation, least-squares solutions remain non-unique because parameters can vary within the kernel space of the representation matrix.The appendix uses inverse notation only on nonzero singular values.
  • A.3 Any Linear Weight from Appendix A.2 Has Zero Least-Square Gradient: The appendix verifies that every linear weight in the identified solution subspace has zero least-squares gradient at the global optimum.The construction handles zero singular values through the corresponding diagonal terms.
  • A.4 Achievable Loss with Low-Rank Representation: With a low-rank representation, the minimum least-squares loss is generally nonzero unless the relevant singular-value projection equals the identity on the represented subspace.The condition is not guaranteed when the representation omits task-relevant directions.
  • A.5 Equivalence Between VICReg Invariance Term and Trace with Graph Laplacian: VICReg's invariance loss is connected to graph Dirichlet energy through the graph Laplacian L = D − G.The derivation uses the relationship between pairwise distances and Dirichlet energy.
  • A.6 Non-Uniqueness of a Representation to a Given VICReg Loss Value: A fixed VICReg loss does not uniquely determine the representation, because adding a constant vector to every row leaves the covariance and pairwise invariance terms unchanged.This establishes representation non-uniqueness independently of the achievable loss value.
  • A.12 SimCLR/NNCLR recovers ISOMAP and MDS (Proposition 2): SimCLR and ISOMAP optimization recover equivalent embeddings up to shift and rescaling because G and G + I share eigenvectors.The eigenvalues shift by 1 while the eigenvectors remain unchanged.

A.7 Optimal Representation and Loss for VICReg (Theorem 1)

This section derives the closed-form optimum for VICReg’s least-square variance loss and establishes properties of the resulting representation.

  • Optimal representation: The analysis targets VICReg’s closed-form optimal representation and its associated minimum loss.The derivation uses the least-square variance loss rather than the hinge-loss formulation.
  • Derivation: The proof uses trace-term simplifications to characterize the optimum.Intermediate steps plug derived quantities into the loss and simplify the trace expression.
  • Representation properties: Nonzero-eigenvalue directions of the solution are centered because their eigenvectors have zero mean.The proof derives this from 1^T v = 0 for eigenvectors with positive eigenvalues.
  • Derivation: The derivation concludes after applying these centering simplifications to the optimal representation.The final passage marks completion of the proof.

A.8 Proof of VICReg Recovering Laplacian Eigenmaps (Theorem 2)

VICReg recovers Laplacian Eigenmaps when its invariance term supplies the trace objective and its variance-covariance constraints impose the corresponding normalization.

  • Objective equivalence: VICReg’s invariance term corresponds to the trace term minimized by Laplacian Eigenmaps.The equivalence is established by matching the invariance objective with the Laplacian Eigenmaps trace formulation.
  • Eigenvector solution: Laplacian Eigenmaps selects nonzero-eigenvalue eigenvectors of its Laplacian matrix, which are centered.The selected eigenvectors correspond to those of D − G when D is isotropic.
  • Constraint equivalence: The variance-covariance constraint becomes Z^T Z = N when the degree matrix D is isotropic.Under D = cI, enforcing zero variance and covariance is equivalent, up to rescaling, to this normalization.

A.9 Proof of Linear VICReg Optimal Parameters (Eq. (12))

The linear VICReg analysis rewrites the quadratic loss in trace form, differentiates it with respect to network parameters, and characterizes its minimizers under rank assumptions.

  • Loss formulation: The quadratic variance-term formulation is used to obtain the linear VICReg loss.This replaces the hinge-term formulation at 1.
  • Optimization: The loss is expressed through trace terms before differentiation with respect to W.The derivation expands quadratic matrix expressions into trace form to simplify optimization.
  • Optimality condition: Differentiating the trace-form loss yields a stationarity condition for W.The resulting derivative is set to zero and simplified using α > 0.
  • Assumptions: When X^T H X is invertible and K = D, the data are assumed to lie on a D-dimensional affine space.If invertibility fails, the input can be projected onto its subspace before applying VICReg.
  • Multiplicity of solutions: When K < D, multiple local minima share the same loss value.They are obtained by selecting any K columns from the stated solution.

A.10 Proof of Laplacian estimation with contrastive learning Theorem 4

The proof connects contrastive learning to graph Laplacian estimation and spectral methods, showing how similarity choices recover SimCLR, NNCLR, and related embedding formulations.

  • Graph estimation: The proof begins by recovering the softmax step for an arbitrary pairwise metric d.The metric computes distances between input pairs before the graph-based optimization.
  • SimCLR and NNCLR: Using cosine distance produces the features used by SimCLR or NNCLR.Applying cross-entropy to known positive-pair or nearest-neighbor graphs recovers their losses.
  • Graph estimation: Graph Laplacian estimation recovers the first step of contrastive methods using row-stochastic or related graph constraints.The Rlog derivation obtains an optimum satisfying symmetry even without explicitly enforcing W_i,j = W_j,i.
  • Global spectral embedding: SimCLR matches the representation’s outer product to the relation matrix through a multidimensional-scaling-style objective.The optimal representation is proportional to eigenvectors of G + I, rescaled by square roots of eigenvalues divided by 2τ.
  • BarlowTwins: BarlowTwins corresponds to canonical correlation analysis, with nonlinear representations yielding Kernel CCA.The formulation works directly on the final representations rather than only on linear input weights.
  • Supervised spectral connections: With supervised relation structure, the generalized eigenvalue formulation links LPP, LDA, and linear VICReg.The relation matrix encodes total, within-cluster, and between-cluster variances, yielding the same solution as LPP under the stated assumptions.
Loading 2205.11508v3…