PaperlayerView exact revision on arXiv

Source-linked AI summary

Understanding self-supervised Learning Dynamics without Contrastive Pairs

Yuandong Tian, Xinlei Chen, Surya Ganguli

arXiv:2102.06810v4cs.LGcs.AIcs.CV

TL;DR

Non-contrastive SSL can learn useful representations without negative pairs, but why it avoids trivial collapse remains an open theoretical question. The paper analyzes these methods through simple linear-network dynamics and proposes DirectPred, which directly sets predictor weights from input statistics. DirectPred matches more complex predictors on ImageNet and exceeds a linear predictor by 2.5% after 300 epochs and 5% after 60 epochs.

  • Problem

    Non-contrastive SSL methods learn powerful representations without negative pairs, raising the unresolved question of how they avoid collapsed representations and how their design choices interact.

  • Method

    The paper analyzes nonlinear learning dynamics in simple linear non-contrastive SSL networks and proposes DirectPred, which directly sets a linear predictor from principal-component statistics of its inputs.

  • Results

    DirectPred achieves 72.4%/91.0% Top-1/Top-5 on ImageNet after 300 epochs, 2.5% higher than BYOL with a linear predictor and comparable with its two-layer predictor.

  • Takeaways & Limitations

    The theory provides conceptual insights into collapse avoidance and predicts performance patterns across ablations involving predictors, stop-gradients, EMA, learning rates, and weight decay.

  • Takeaways & Limitations

    The theoretical analysis relies on proportional-EMA and isotropic-data-and-augmentation assumptions.

Abstract

from arXiv · show

While contrastive approaches of self-supervised learning (SSL) learn representations by minimizing the distance between two augmented views of the same data point (positive pairs) and maximizing views from different data points (negative pairs), recent \emph{non-contrastive} SSL (e.g., BYOL and SimSiam) show remarkable performance {\it without} negative pairs, with an extra learnable predictor and a stop-gradient operation. A fundamental question arises: why do these methods not collapse into trivial representations? We answer this question via a simple theoretical study and propose a novel approach, DirectPred, that \emph{directly} sets the linear predictor based on the statistics of its inputs, without gradient training. On ImageNet, it performs comparably with more complex two-layer non-linear predictors that employ BatchNorm and outperforms a linear predictor by $2.5\%$ in 300-epoch training (and $5\%$ in 60-epoch). DirectPred is motivated by our theoretical study of the nonlinear learning dynamics of non-contrastive SSL in simple linear networks. Our study yields conceptual insights into how non-contrastive SSL methods learn, how they avoid representational collapse, and how multiple factors, like predictor networks, stop-gradients, exponential moving averages, and weight decay all come into play. Our simple theory recapitulates the results of real-world ablation studies in both STL-10 and ImageNet. Code is released https://github.com/facebookresearch/luckmatters/tree/master/ssl.

1. Introduction

Non-contrastive SSL learns powerful representations without negative pairs, but its mechanisms for avoiding collapse and the interactions among its design choices remain theoretically unresolved. The paper studies these dynamics and introduces DirectPred, which achieves competitive ImageNet performance with a directly set linear predictor.

  • Motivation: BYOL and SimSiam learn powerful representations using only positive pairs, with predictor and stop-gradient components, rather than contrasting negative pairs.Their online-plus-predictor representations are trained to match target-network representations.
  • Open questions: Non-contrastive SSL raises unresolved questions about avoiding collapsed representations and how predictors, EMA, learning rates, and weight decay interact.The paper frames these as questions about the nonlinear learning dynamics of the overall scheme.
  • Key design choices: The predictor and stop-gradient are essential: removing either causes representational collapse in BYOL and SimSiam.
  • Key design choices: EMA is not necessary when the predictor has a higher relative learning rate or is updated more often, although performance is slightly lower.
  • Contribution: DirectPred sets predictor weights from principal-component statistics and reaches 72.4%/91.0% Top-1/Top-5 on ImageNet after 300 epochs.It is 2.5% higher than BYOL with a linear predictor and comparable with the default two-layer predictor.

2. Two-layer linear model

The paper models non-contrastive SSL with a simple bias-free linear BYOL system whose online, predictor, and target matrices follow coupled gradient-flow and EMA dynamics. These equations provide a tractable basis for analyzing predictors, augmentation statistics, learning rates, EMA, and weight decay.

  • Model: The model uses online, predictor, and target matrices W, Wp, and Wa in a two-layer, bias-free linear BYOL setting.
  • Dynamics: The online and predictor dynamics minimize prediction error, while the target matrix evolves through exponential moving average of the online matrix.
  • Dynamics: The analysis represents augmented views through their averaged covariance X and conditional augmentation covariance X′, with αp and β encoding relative learning-rate ratios.Weight-decay effects enter the dynamics through η.
  • Dynamics: SimSiam follows from the same dynamics by setting Wa = W and removing the EMA equation while retaining the stop-gradient on the target branch.
  • Analysis: Weight decay promotes balancing of the predictor and online networks.
  • Analysis: With no predictor, the dynamics can drive W(t) toward zero, corresponding to collapse.

3. How multiple factors affect learning dynamics

The paper reduces non-contrastive SSL to tractable linear-network dynamics and uses this analysis to explain how EMA, predictor learning rate, weight decay, and initialization affect collapse and representation growth.

  • Analytical setup: The analysis uses simplifying assumptions about proportional EMA, isotropic data and augmentation, and a symmetric predictor to obtain tractable dynamics.These assumptions reduce the original high-dimensional nonlinear system while retaining experimentally testable predictions.
  • Eigenspace alignment: Under the reduced dynamics, the online-output correlation matrix F and predictor Wp evolve jointly, with their eigenspaces gradually aligning.F is the covariance matrix of online-network outputs under the isotropic-data assumption.
  • Scope: The symmetric-predictor analysis does not extend cleanly to non-symmetric Wp, whose dynamics are left for future work.This is an explicit scope limitation of the theoretical treatment.
  • Predictor learning rate: A fast predictor reduces the basin of attraction of collapse, but an excessively large αp prevents the online network from growing with the predictor.The theory therefore predicts that predictor speed helps only within a suitable range.
  • Weight decay: Weight decay forgets initial conditions and drives the dynamics toward an invariant relation in which the learned representation can reflect augmentation variance through the predictor.With sufficiently strong weight decay, however, collapse becomes unavoidable.
  • EMA and non-EMA dynamics: EMA acts as an automatic curriculum by gradually increasing the predictor’s target, whereas very slow EMA produces slower training.Without EMA, satisfying the growth condition can require alternative choices such as a larger predictor learning rate or selective predictor weight decay.

4. Optimization-free Predictor Wp

DirectPred replaces gradient-trained predictor dynamics with predictor weights set directly from the input correlation matrix, using eigendecomposition to align predictor and input eigenspaces. It matches BYOL’s nonlinear predictor on ImageNet and outperforms learned linear predictors across reported settings.

  • Optimization-free predictor: DirectPred estimates the predictor-input correlation matrix and directly sets Wp from its eigendecomposition, avoiding predictor gradient dynamics and collapse-sensitive initialization.The method is motivated by eigenspace alignment and convergence to an invariant relationship between eigenvalues and predictor weights.
  • STL-10 and CIFAR-10: 76.77% STL-10 Top-1 accuracy exceeds 74.51% from gradient training of the regular predictor with EMA.The reported DirectPred experiment uses BYOL training for 100 epochs with EMA.
  • STL-10 and CIFAR-10: 78.20±0.16 STL-10 Top-1 accuracy after 100 epochs closes the gap to two-layer predictors at 78.85%.This variant uses six random partitions so DirectPred can estimate different correlation matrices across input regions.
  • ImageNet: 72.4%/91.0% ImageNet Top-1/Top-5 accuracy matches BYOL’s two-layer predictor at 72.5%/90.8% after 300 epochs.DirectPred sets a 256×256 linear predictor every minibatch without gradient-based predictor training.
  • ImageNet: 2.5% higher ImageNet Top-1 accuracy than a learned linear predictor is achieved after 300 epochs, while the 60-epoch gap reaches 5.0%.At 300 epochs, the learned linear predictor reaches 69.9% Top-1; at 60 epochs, the comparison is 59.4% versus 64.4%.

5. Discussion

The paper’s simple nonlinear dynamical theory explains how non-contrastive SSL ingredients interact and motivates DirectPred. A formal analysis of two-layer predictors remains out of scope.

  • 5. Discussion: A three-dimensional nonlinear dynamical analysis explains the roles and interactions of EMA, stop-gradients, predictors, symmetry, learning rates, and weight decay.The theory also predicts patterns observed in ablation studies and motivates DirectPred’s real-world performance.
  • 5. Discussion: The paper does not provide a formal analysis of two-layer predictors because their dynamics and mathematics can be complicated.The discussion instead offers intuition based on the large pool of initial weight directions supplied by wide hidden layers and ReLU.

Supplementary Materials for “Understanding Self-Supervised Learning

The supplementary material is titled “Dynamics without Contrastive Pairs.”

  • Supplementary Materials: The supplementary materials extend the paper on dynamics without contrastive pairs.

A. Section 2

The appendix derives gradient-flow dynamics for BYOL and SimSiam, then analyzes how stop-gradients and predictors affect collapse. Without stop-gradients, the resulting dynamics drive the online weights to zero under positive-definiteness conditions.

  • A. Section 2: Lemma 1 derives BYOL/SimSiam dynamics for the online weights, predictor weights, and EMA target weight.The analysis uses large batches and small learning rates so discrete training is approximated by gradient flow.
  • A. Section 2: With weight decay η, the predictor update gains an additional −ηWp term.This modifies the gradient flow while retaining the core update involving online, target, and predictor weights.
  • A. Section 2: The gradient updates possess an initialization-dependent invariant involving a symmetric matrix C.This invariance is stated for the gradient update rules in the analyzed system.
  • A. Section 2: Without stop-gradient in the SimSiam case Wa = W, the dynamics are governed by a positive-semidefinite matrix H(t).The analysis represents the dynamics using vectorized weights and a Kronecker-product matrix.
  • A. Section 2: If H(t) remains uniformly positive definite, the online weights converge exponentially to zero, preventing meaningful feature learning.The same collapse conclusion applies when Wa = W and no predictor is used, with or without stop-gradient.
  • A. Section 2: The EMA assumption is largely correct: even with γa close to 1 and large η, normalized correlation between Wa and W remains about 0.9.Figure 7 reports this behavior across η = 0, 0.01, and 0.1.

B. Section 3

The analysis models non-contrastive SSL with simple linear-network dynamics, incorporating EMA, predictor updates, and weight decay to characterize fixed points and symmetry.

  • Isometric assumptions: The theory studies nonlinear learning dynamics of simple linear networks under isotropic assumptions X = I and X′ = σ2I.These assumptions simplify the data correlations used in the dynamics.
  • EMA assumption: EMA is modeled as Wa = τW, where τ < 1 attenuates W and varies according to how rapidly W changes.If W is stable, τ is approximately 1; rapid growth makes τ smaller.
  • Learning dynamics: Under the EMA and weight-decay assumptions, the coupled dynamics are expressed through Wp and F, with F = WW⊺.The predictor and representation dynamics are analyzed jointly.
  • Fixed points: When η = 0 and F has zero eigenvalues, the predictor can have infinitely many fixed points, including nonsymmetric solutions.This contrasts with the symmetric fixed point obtained under positive weight decay.

B.1. Section 3.1

The symmetrized dynamics show that predictor and representation eigenspaces can align, while stability and convergence depend on the spectrum of the governing matrix and on EMA and weight decay.

  • Alignment of eigenspaces: If λmin[K(t)] remains positive, ||[F(t), Wp(t)]||F decays exponentially to zero and the eigenspaces gradually align.The bound is ||[F(t), Wp(t)]||F ≤ e^−2λ0t||[F(0), Wp(0)]||F.
  • Alignment of eigenspaces: The commutator L = [F, Wp] evolves as L̇ = −KL − LK, linking eigenspace alignment to the spectrum of K.The Kronecker-sum operator K(t) ⊕ K(t) is positive semidefinite when K is positive semidefinite.
  • Numerical behavior: Large weight decay or small τ can drive the symmetric component of Wp toward zero, while the asymmetric component tends to vanish in simulation.The simulations use different τ values and weight-decay settings to examine these effects.
  • Stability: When K has negative eigenvalues, the dynamics may diverge despite approximate eigenspace alignment because of noise or numerical instability.Thus alignment alone does not guarantee stable dynamics.
  • Aligned dynamics: After eigenspace alignment, Wp and F decouple into independent eigenmodes, so their common eigenvectors remain fixed over time.The dynamics of each aligned eigenmode can then be analyzed separately.
  • Trivial solutions: The trivial attractive region expands with weight decay, trapping more eigenvalues at the trivial solution.The boundary of this region depends on η and τ.

D. Analysis of BYOL and SimSiam learning dynamics without isotropic assumptions on data

Without isotropic data assumptions, the analysis derives fixed-point structure for BYOL and SimSiam from view correlations and studies how data and augmentation distributions shape non-collapsed solutions.

  • Setup: The alternate analysis replaces isotropic assumptions with decoupled initial conditions to study BYOL and SimSiam under non-isotropic data.It defines single-view and cross-view correlation matrices from the augmentation process.
  • Data correlations: The matrices satisfy Σs = X + X′ and Σd = X, where X captures augmentation-averaged data correlations and X′ captures averaged conditional augmentation covariance.These quantities separate shared structure from augmentation variability.
  • Fixed-point analysis: The analysis derives exact fixed points for BYOL and SimSiam and examines their dependence on data and augmentation distributions.It also develops a reduced three-dimensional description under decoupled initial conditions.
  • Fixed-point structure: Sufficient fixed-point conditions are WpWΣs = WaΣd and W = Wa, applying to both BYOL and SimSiam.When Σs is invertible, these conditions yield a nonlinear manifold of fixed-point solutions.
  • Fixed-point structure: The fixed-point equations generically form a nonlinear manifold with dimensionality n2 × n2, corresponding to the number of predictor parameters.A special solution class uses eigenvectors of Σd[Σs]−1 and a diagonal predictor.

D.2. Illustrative models for data and data augmentation

Illustrative augmentation models identify the eigenmodes governing non-collapsed solutions and show how preserved or noise-attenuated dimensions shape learned representations.

  • Eigenmode selection: The top eigenmodes of Σd[Σs]−1 control the non-collapsed solutions in BYOL and SimSiam.The examples make this dependence concrete for different augmentation procedures.
  • Multiplicative scrambling: Multiplicative scrambling randomizes a k-dimensional subspace while preserving its orthogonal complement, which can represent semantically important features.The preserved subspace is orthogonal to the scrambled subspace.
  • Multiplicative scrambling: When the conserved-subspace projector commutes with the data correlation matrix, Σd[Σs]−1 reduces to that projector independently of the data correlation matrix.In this setting, non-collapsed solutions can learn features identically conserved by augmentation.
  • Additive scrambling: With additive Gaussian noise, BYOL selects principal eigenmodes of Σx[Σx + Σn]−1, attenuating dimensions with larger noise variance.Data correlations not attenuated by noise are preferentially retained.

D.3. The importance of the predictor in BYOL and SimSiam.

The theory shows that an adjustable predictor creates non-collapsed fixed points for BYOL and SimSiam, whereas removing it imposes restrictive conditions that often leave collapse stable.

  • Without a predictor, non-collapsed solutions require Σd[Σs]^-1 to have eigenvectors with eigenvalue 1.This condition is not generically satisfied for additive scrambling, where all eigenvalues are less than 1 and the collapsed solution is stable.
  • For multiplicative scrambling, the eigenvalue-1 condition can be satisfied when an eigenvector of Σx lies in the projection operator’s column space.
  • An adjustable predictor creates new non-collapsed solutions for BYOL and SimSiam without adding expressive capacity to the online network.Its role is to modify the learning dynamics rather than improve target-matching expressiveness.

D.4. Reduction of BYOL learning dynamics to low dimensions

The paper reduces BYOL dynamics to tractable low-dimensional systems by using decoupled eigenmode initial conditions and analyzes how predictor, online, and target weights escape collapse.

  • Reduction of BYOL learning dynamics to low dimensions: Under decoupled initial conditions, each eigenmode evolves through independent scalar dynamics driven by the corresponding eigenvalues of Σs and Σd.The matrix dynamics preserve the eigenmode structure while the scalar weights change over time.
  • Reduction of BYOL learning dynamics to low dimensions: The reduced three-dimensional system is also obtained directly when all matrices are 1 × 1, replacing W, Wp, Wa, Σs, and Σd with scalars.This system is equivalent to the paper’s analyzed three-dimensional dynamics under a change of variables.
  • Reduction of BYOL learning dynamics to low dimensions: The fixed-point conditions are wa = w and wpw = waλdλs^-1, yielding both the collapsed origin and a family of non-collapsed solutions.The non-collapsed family has wp = λdλs^-1 with w and wa taking the same value.
  • Reduction of BYOL learning dynamics to low dimensions: When β is small, the online and predictor weights evolve rapidly while the target weight changes slowly, producing fast-slow dynamics.The rapid variables approach a moving zero-error contour as the slowly changing target network adjusts.
  • Reduction of BYOL learning dynamics to low dimensions: The online and predictor weights cooperatively amplify from the unstable origin and approach the contour wpw = waλdλs^-1.The larger λdλs^-1, determined by data and augmentation, the larger the final online and predictor weights tend to be.
  • Reduction of BYOL learning dynamics to low dimensions: With an adjustable predictor, the dynamics yield stable non-collapsed convergence; removing it generally leaves only the stable collapsed origin.In the illustrated setting, the predictor’s removal prevents non-collapsed solutions unless λd = λs.
Loading 2102.06810v4…