Contrastive and Non-Contrastive Self-Supervised Learning Recover Global and Local Spectral Embedding Methods

• SSL learns from inputs and inter-sample semantic relations rather than labels, and rapid empirical progress has made a principled theoretical account increasingly urgent.The paper notes that prior theoretical work studied training dynamics, individual components, or new SSL criteria, while fundamental questions remained open.Source: §1 · pp. 1–2
• Contrastive and non-contrastive SSL correspond to global and local spectral embeddings, respectively, with contrastive methods suiting metric-preserving manifolds and non-contrastive methods suiting locally linear, globally nonlinear manifolds.The paper summarizes this unification in Figure 1.Source: Figure 1 · p. 2, §1 · p. 3
• Across SSL methods, optimal Z aligns its top left singular vectors and singular values with G, while its right singular vectors remain unconstrained.The paper also derives closed-form optimal representations for the methods it studies.Source: §1 · pp. 2–3
• Any SSL loss yields an optimal downstream representation when the left singular vectors of G with nonzero singular values span those of Y; it matches the supervised representation up to right-singular-vector rotation.This is the stated condition for perfect downstream-task performance.Source: §1 · pp. 2–3
• With carefully selected loss hyperparameters, VICReg can learn a full-rank representation, whereas SimCLR and BarlowTwins enforce rank(Z) = rank(G).The paper identifies VICReg as a possible preference when G is misspecified.Source: §1 · p. 2
• The spectral unification supports practical design guidance, including analytical linear-regime network parameters and novel interpretable SimCLR variations derived from first principles.The paper also uses it to identify situations in which different SSL methods should be preferred.Source: §1 · p. 3
• The framework derives closed-form optimal representations and linear-network parameters, and links VICReg to Laplacian Eigenmaps, SimCLR/NNCLR to ISOMAP, and BarlowTwins to Canonical Correlation Analysis.For linear networks, it also connects VICReg and BarlowTwins to Locality Preserving Projection and Linear Discriminant Analysis.Source: §1 · p. 3

Dataset, Embedding and Relation Matrix Notations

• SSL uses observations X and a positive-relation matrix G, often constructed by treating different augmentations of the same sample as positive views; a model maps samples to embeddings Z.The common setup typically uses two views, while the number of views can vary.Source: §2 · pp. 3–4

VICReg

• VICReg combines invariance, variance, and covariance loss terms; its computational complexity is O(NK^2 + PNK), dominated by covariance when P ≪ K.Here, P is the average number of positive samples per row of G.Source: §2 · p. 4

SimCLR

• SimCLR estimates a right-stochastic relation matrix from embedding similarities, then uses infoNCE to match it to G; its pairwise-similarity computation has complexity O(N^2).SimCLR variants differ in how they construct G when only unaugmented inputs are given.Source: §2 · p. 4

BarlowTwins

• BarlowTwins uses two views and a cross-correlation matrix, bringing its diagonal toward one while penalizing off-diagonal entries.For data with multiple positive samples, the section describes constructing paired left and right matrices from G.Source: §2 · pp. 4–5

Linear Algebra Notations

• The paper specifies SVD notation for singular vectors and values, distinguishing vectors associated with zero singular values from those associated with positive values.Singular values are taken in descending order unless otherwise stated.Source: §2 · p. 5

Dataset, Embedding and Relation Matrix Notations

• The analysis aims to connect existing SSL methods to spectral methods, characterize optimal representations in closed form, and derive necessary and sufficient conditions bounding downstream task error.The paper identifies these as three results that will emerge in the following sections.Source: §2 · pp. 5–6

3.1 Closed-Form Optimal Representation for VICReg

• VICReg’s global optimum is a closed-form representation determined by the relation graph and hyperparameters, with the ratio γ/α controlling their spectral balance.Its invariance term is the Dirichlet energy of the representation on the graph, encouraging non-degenerate signals that are smooth on that graph. A suitable hyperparameter range can preserve full rank while retaining enough graph information to solve the downstream task.Source: §3.1 · pp. 6–7
• A sparse reformulation of the optimal VICReg eigensystem enables efficient storage and computation of the top eigenvectors, including for large datasets.It replaces the dense connection to all graph nodes with an added node connected to all others, allowing matrix-vector-product eigensolvers such as LOBPCG.Source: §3.1 · p. 8

3.2 VICReg Recovers Laplacian Eigenmaps in Feature Space and Kernel Locality Preserving Projection in Data Space

• With variance and covariance constrained to zero, VICReg is equivalent to Laplacian Eigenmaps (LE) for a given relation matrix G and to kernel Locality Preserving Projection in input space.The choice of G is consequential: LE commonly builds it from a k-NN graph, while SSL constructs it from augmentations or supplies it directly; when D − G is idempotent, LE and VICReg also recover Locally Linear Embedding.Source: §3.2 · p. 9

3.3 With a Linear Network VICReg Recovers Locality Preserving Projections and Linear Discriminant Analysis with Analytical Optimal Network Parameters

• With a linear network, variance-covariance-constrained VICReg recovers LPP for arbitrary G and LDA for the supervised class-relation graph, with optimal weights given by top-K eigenvectors.For LDA, the result uses K = C; the eigenvalues associated with the optimal weights recover MANOVA sufficient statistics.Source: §3.3 · p. 10

4.1 Step 1: SimCLR Pairwise Similarities Solve a Graph Laplacian Estimation Problem

• SimCLR estimates a graph whose signals Z are smooth: regularized graph-weight optimization recovers SimCLR similarities with cosine distance and the Lifted Structured Loss with ℓ2 distance.The regularizer prevents the estimated graph from collapsing to zero, and its temperature-dependent strength pushes weight entries toward 1.Source: §4.1 · p. 11
• Graph-estimation constraints yield novel SimCLR variants, including forms akin to Hadsell et al. [2006] and Weinberger and Saul [2009] using ℓ2 distance without exponentiation.Different similarity-estimation mappings correspond to different optimization problems, extending the framework to NNCLR [Dwibedi et al., 2021] and MeanShift [Koohpayegani et al., 2021].Source: §4 · p. 10, §4.1 · p. 11

4.2 Step 2:

• SimCLR’s optimal representation has the same nonzero left singular vectors as G’s eigenvectors, and its rank matches G’s, producing dimensional collapse.The result applies across the specified loss and graph-estimation choices, with τ ≥ max_i,j(D)_i,j.Source: §4.2 · p. 12

4.3 SimCLR is Akin to ISOMAP in Feature Space and Kernel ISOMAP in Input Space

• Under the settings of Theorem 5, SimCLR recovers a Generalized MDS akin to ISOMAP, and ordinary MDS iff G = XXT.Source: §4.3 · p. 13
• When SimCLR recovers ISOMAP or MDS in feature space, its input-space formulation recovers kernel ISOMAP or kernel PCA.The input-space formulation replaces ZZT with ΦW^T WΦT.Source: §4.3 · p. 13

5.1 BarlowTwins Recovers Kernel Canonical Correlation Analysis

• BarlowTwins recovers Kernel CCA with a representation rank of min(K, D, rank(G)).This follows by using neural-network feature embeddings in the nonlinear CCA formulation.Source: §5.1 · p. 13
• The BarlowTwins loss’s epsilon term corresponds to canonical ridge regularization, which adds numerical stability to CCA.The stated CCA objective is well-defined only when K < min(Da, Db).Source: §5.1 · pp. 13–14
• Figure 9 shows rank(Z) matches rank(G) regardless of initialization; BarlowTwins and SimCLR enforce that rank, unlike VICReg, whose rank can vary with loss hyperparameters.BarlowTwins’ performance is correspondingly more sensitive to the design of G.Source: Figure 9 · p. 14, §5.1 · p. 14

5.2 With a Linear Network BarlowTwins Recovers Canonical Correlation Analysis and Linear Discriminant Analysis

• BarlowTwins’ linear-regime optimal weights recover CCA; weight sharing follows for symmetric G and same-dimensional inputs, while supervised G with K = C recovers LDA and thus VICReg.The connection also suggests a sparse BarlowTwins variant based on penalized matrix decomposition for CCA.Source: §5.2 · pp. 14–15

6 Optimality of Self-Supervised Methods to Solve Downstream Tasks

• The section asks what target-matrix information Z must preserve for a linear predictor to reach zero error, then develops optimal parameters, representation conditions, and SSL criteria.Source: §6 · p. 15

6.1 Characterizing a Representation Usefulness by its Ability to Linearly Solve a Task

• The section evaluates representation usefulness by the minimum linear MSE for mapping Z to target Y, using squared loss and noting ℓ2 can match cross-entropy with proper tuning.The analysis seeks the possibly non-unique linear parameters that minimize this loss.Source: §6.1 · p. 15

6.2 Necessary and Sufficient Conditions for Optimality of a Representation

• Zero linear task loss is achievable exactly when Z’s nonzero left-singular vectors span Y’s nonzero left-singular-vector subspace.The stated alignment of the top-K left singular vectors is necessary but not sufficient for optimality.Source: §6.2 · p. 16

6.3 Contrastive and Non-Contrastive Learning can all be Optimal

• VICReg, SimCLR, and BarlowTwins yield an optimal representation exactly when the target’s relevant left-singular vectors lie in the span of those of relation matrix G.The same conclusion applies to NNCLR and MeanShift when they use the SimCLR loss; this aligns with empirical findings that SSL and supervised models can converge to nearly the same thing, as observed by Geirhos et al. [2020].Source: §6.3 · pp. 16–17

6.4 Non-Contrastive Methods Should be Preferred: Best and Worst Downstream Task Error Bounds

• For rank-R relation matrices G, tight downstream-error bounds imply preferring VICReg, then BarlowTwins, then SimCLR.This distinction matters when G is not aligned with the task: SimCLR forces representation rank to match G, whereas VICReg enforces full rank.Source: §6.4 · p. 17

7 Conclusions

• When G matches the downstream task, VICReg, SimCLR, and BarlowTwins all yield an ideal solution; when misaligned, low-invariance VICReg preserves full rank, unlike SimCLR and BarlowTwins.In the misaligned regime, VICReg retains all information from G while allowing use on other tasks; SimCLR and BarlowTwins collapse the representation to information about G alone.Source: §7 · p. 17
• Contrastive SSL parallels global spectral methods that aim to preserve metric structure, while non-contrastive SSL parallels local methods that suit manifolds with near-Euclidean local but potentially non-Euclidean global geometry.Source: §7 · p. 17
• The authors propose transferring results and techniques between SSL and spectral embedding, noting that their main difference is how G is constructed.They observe that spectral methods have fallen short on high-dimensional datasets such as ImageNet, while SSL methods have become state of the art there.Source: §7 · p. 17

• The supplement proves the paper’s formal results, supplies background and references for self-contained derivations, and includes additional insights into current SSL methods.Source: Supplementary Materials · p. 23

A.1 VICReg Variance+Covariance Versus Representation’s Singular Values

• Constraining ΣZ’s singular values near 1 upper-bounds VICReg’s variance-plus-covariance objective, although its variance term only requires values of at least 1.Here, σZ denotes the diagonal part of ΣZ.Source: §A.1 · p. 23

A.2 Non-Unique Solution to Least-Square

• For rank-deficient Z, least-squares solutions W predicting Y = ZW are non-unique because they can vary within ker(Z).The inverse notation for ΣZ applies only to its nonzero diagonal elements; the zeros correspond to dimensions of ker(Z).Source: §A.2 · p. 23

A.3 Any Linear Weight from Appendix A.2 Has Zero Least-Square Gradient

• Every optimal W from the rank-deficient least-squares solution space has zero loss gradient and is a global optimum.The derivation handles zero singular values separately when establishing the equality.Source: §A.3 · p. 23

A.4 Achievable Loss with Low-Rank Representation

• With a low-rank representation, the minimum least-squares loss is generally nonzero unless ΣZΣZ^-1 = I, where the inverse covers only nonzero singular values.The identity condition is not guaranteed for a low-rank Z.Source: §A.4 · p. 24

A.5 Equivalence Between VICReg Invariance Term and Trace with Graph Laplacian

• VICReg’s invariance loss equals a graph-Laplacian trace, linking pairwise distances to the graph’s Dirichlet energy.The derivation uses L = D − G, with D the diagonal row-sum degree matrix [Von Luxburg, 2007].Source: §A.5 · p. 24

A.6 Non-Uniqueness of a Representation to a Given VICReg Loss Value

• A VICReg representation’s loss value is non-unique: adding a constant vector to every row leaves covariance unchanged and cancels in pairwise invariance comparisons.Source: §A.6 · p. 25

A.12 SimCLR/NNCLR recovers ISOMAP and MDS (Proposition 2)

• G and G + I have identical eigenvectors with eigenvalues shifted by 1, making SimCLR/NNCLR and ISOMAP optimization equivalent up to representation shift and rescaling.The proof also characterizes the equivalence as up to rescaling.Source: §A.12 · p. 29

A.7 Optimal Representation and Loss for VICReg (Theorem 1)

• The section seeks the closed-form optimal VICReg representation under the least-square variance loss, rather than the hinge loss, and its associated minimum loss.The stated optimum is non-unique; the supplied passages do not include its explicit form or loss value.Source: §A.7 · p. 25
• The derivation simplifies the trace term using the first K indices or columns of Eq. (8), then substitutes the resulting value into the loss.The supplied passages do not show the resulting expression.Source: §A.7 · p. 25
• Eigenvectors with positive eigenvalues have zero mean, a property used to simplify the derivation.The passage gives the implication 1^T v = 0 for any eigenvector v with λ > 0.Source: §A.7 · p. 25

A.8 Proof of VICReg Recovering Laplacian Eigenmaps (Theorem 2)

• VICReg’s invariance term equals twice Laplacian Eigenmaps’ trace term, with the graph supplied by SSL rather than built from k-NN and geodesic estimates.The proof identifies matching objectives while noting this difference in how the graph is obtained.Source: §A.8 · p. 26
• When D is isotropic (D = cI), exact variance and covariance minimization is equivalent, up to rescaling, to the constraint Z^T Z = N.The LE eigenvectors with nonzero eigenvalues are centered; under isotropic D, this supports the equivalence between VICReg’s variance and covariance terms and the LE constraint.Source: §A.8 · p. 26

A.9 Proof of Linear VICReg Optimal Parameters (Eq. (12))

• Replacing the variance hinge at 1 with a quadratic variance term reduces the linear VICReg loss to a trace expression that can be differentiated with respect to W.The derivation sets the resulting derivative to zero to obtain the parameter condition.Source: §A.9 · p. 26
• With X^T H X invertible and K = D, the derivation obtains the optimal parameters; if the data lack this property, project X onto its subspace before applying VICReg.The invertibility assumption corresponds to data lying on a D-dimensional affine space.Source: §A.9 · p. 26
• For K < D, multiple local minima share the same loss, obtained by selecting any K columns of the K = D solution.The first K columns are chosen arbitrarily for Eq. (12).Source: §A.9 · p. 26

• Graph Laplacian estimation recovers the first contrastive step using both Rlog and RF; for G, the row-sum-to-one constraint can be removed.Source: §A.10 · pp. 27–28
• For Rlog, the optimum has zero diagonal and is symmetric even without an enforced symmetry constraint, yielding an undirected graph Laplacian estimate.Source: §A.10 · p. 27
• With cosine distance, the recovered features are those used by SimCLR or NNCLR, and applying cross-entropy to known positive pairs recovers their loss exactly.Source: §A.10 · pp. 27–28

A.11 Proof of SimCLR Theorem 5

• SimCLR’s representation is proportional to the first K eigenvectors of G+I, rescaled by the square roots of their eigenvalues divided by 2τ.For RBF kernels with distinct samples, the kernel is nonsingular and minimizing the loss requires the representation to collapse so its rank aligns with G.Source: §A.11 · p. 29

A.13 Barlow Twins optimal Representation (Theorem 6)

• BarlowTwins with a nonlinear DN recovers Kernel CCA, using regularized CCA for the representation-rank analysis.The proof works directly with the representation fed into the loss; the regularization adds a small constant to the cosine-similarity denominator.Source: §A.13 · pp. 29–30

A.14 Linear VICReg is Locality Preserving Projection (Theorem 3)

• With an invertible within-component variance matrix, linear VICReg corresponds to Locality Preserving Projection (LPP).The stated invertibility assumption is for clarity, not required for a solution to exist.Source: §A.14 · p. 30

A.15 Linear VICReg with supervised relation matrix is LDA (Theorem 3)

• Linear VICReg with a supervised relation matrix recovers the simplified LDA solution, which is equivalent to LPP using that relation matrix.The proof identifies the solution with Fisher’s criterion and its generalized eigenvalue problem when the within-class covariance is invertible.Source: §A.15 · pp. 30–31

A.16 Proof of BarlowTwins in the Linear Regime (Theorem 7)

• In the linear regime, BarlowTwins’ CCA solution and LDA solution are recoverable from each other, differing only in singular values.For the supervised CCA construction, the two views are permutations of one another and represent within-class sample pairs.Source: §A.16 · pp. 31–32

A.17 Proof of VICReg Optimality (Theorem 9)

• VICReg admits a linear probe that solves the downstream task when the left singular vectors of G span those of Y.The proof combines the optimal representation’s left singular vectors of G, up to rotations for repeated singular values, with Theorem 9.Source: §A.17 · p. 32