PaperlayerView exact revision on arXiv

Source-linked AI summary

$\lambda$-JEPA Spectral Anti-Collapse Regularization for Self-Supervised Learning

Berker Demirel, Clémentine Dominé, Valentino Maiorca, Marco Fumero, Marco Mondelli, Francesco Locatello

arXiv:2609.35288v2cs.LGcs.CV

TL;DR

In JE-SSL, anti-collapse constraints on projected representations do not necessarily prevent low-rank backbone representations, even though downstream tasks use the backbone. This paper derives SACReg from λ-balance and applies it in λ-JEPA, which improves image transfer and classification performance and reports gains on video benchmarks.

  • Problem

    Anti-collapse constraints can preserve rank in projected JE-SSL representations without preventing low-rank backbone representations, which downstream tasks use.

    Source: §1 · p. 1, §2 · p. 2

  • Method

    SACReg uses a two-layer linear λ-balance analysis to motivate covariance regularization of view-averaged backbone features, which λ-JEPA applies alongside projected-representation regularization.

    Source: §1 · p. 2, §3 · p. 3

  • Results

    λ-JEPA improves frozen transfer while remaining competitive on ImageNet-1k classification, and reports large gains in video temporal recognition.

    Source: §1 · p. 2, §4.2 · p. 7

  • Takeaways & Limitations

    In controlled experiments, SACReg consistently increases backbone rank and improves downstream performance.

    Source: §5 · p. 9

    SACReg is limited to CLS-token and global-view regularization; extending it to patch-level features and local crops led to unstable training.

    Source: §5 · p. 9

Abstract

from arXiv · show

Joint-embedding self-supervised learning typically combines an invariance objective across augmented views with additional mechanisms to prevent representational collapse. These objectives are often applied after a projection head, while downstream tasks use the backbone representation before the projector. We find that this mismatch does not necessarily prevent dimensional collapse in the backbone, which can retain low effective rank and potentially limit downstream transfer. To address this, we introduce SACReg, a spectral anti-collapse regularizer motivated by an analysis of $\lambda$-balance, which captures the relative scale of weight matrices across layers. In a two-layer linear network, we show that (i) $\lambda$-balance prevents collapse, and (ii) our regularizer applied to the backbone induces $\lambda$-balance. In the nonlinear case, this regularizer leads to anti-collapse as well and, in realistic architectures on ImageNet100, it empirically increases the representations' ranks. We apply SACReg to JEPA and propose $\lambda$-JEPA, which improves over LeJEPA and VISReg on ImageNet-1k classification and in average linear-probe transfer performance across eight downstream image datasets. On video self-supervised learning, $\lambda$-JEPA improves over LeVJEPA and V-JEPA 2 on the Something-Something-v2 and Kinetics-400 benchmarks. Code is available at https://github.com/berkerdemirel/lambda-jepa.

1 INTRODUCTION

Projected-space anti-collapse can leave JE-SSL backbones low rank, potentially limiting transfer; the paper introduces SACReg and λ-JEPA to regularize backbone representations and reports improvements across image and video benchmarks.

1 INTRODUCTION

  • High-rank projected representations do not guarantee high-rank backbones, potentially restricting downstream feature directions and transfer.The introduction notes that JE-SSL methods optimize projected representations, while downstream tasks use the backbone instead.Source: §1 · pp. 1–2
  • On ImageNet-1K from scratch, λ-JEPA outperforms LeJEPA and VISReg, approaching DINO in linear probing and transfer learning.The introduction also describes λ-JEPA as improving frozen transfer while remaining competitive on ImageNet classification.Source: §1 · p. 2
  • On Something-Something-v2 and Kinetics-400, λ-JEPA trained from scratch outperforms V-JEPA 2 and LeVJEPA.The introduction characterizes its temporal-recognition gains as large.Source: §1 · p. 2
  • SACReg derives from λ-balance in a two-layer linear model and extends to nonlinear encoders through covariance regularization of view-averaged backbone representations.The introduction says λ-balance captures the relative scale of adjacent layers and prescribes both the regularizer’s form and where it is applied.Source: §1 · p. 2

2 PROBLEM AND RELATED WORKS

Explicitly regularized JE-SSL methods constrain collapse in projected representations, but this does not guarantee that backbone representations avoid dimensional collapse. Prior feature-learning work motivates examining how adjacent-layer scale balance affects backbone rank, a question less characterized in nonlinear, larger networks.

Problem and related works

  • With nonlinear projectors, preventing collapse in projected representations z does not in general prevent dimensional collapse in backbone representations h.The paper identifies this mismatch as a motivation for backbone-level anti-collapse regularization.Source: §2 · p. 2
  • A nonlinear projector can map a rank-one backbone representation to a full-rank projected representation, illustrating why projected-space rank can mask backbone collapse.For h = (X, 0)^T with X uniform on {−1, 0, 1}, the example gives rank-one Cov(h) and full-rank Cov(p(h)).Source: §2 · p. 2
  • Backbones can remain low rank, especially for VISReg and LeJEPA, while RankMe is typically higher after projection; low-rank backbones may restrict task-agnostic transfer.RankMe measures how evenly variance is distributed across feature directions, and prior work reports that it correlates with JE-SSL downstream performance (Garrido et al., 2023).Source: §2 · p. 3
  • Feature-learning research studies initialization and adjacent-layer λ-balance, but their effects on representation rank remain less characterized, particularly in nonlinear and larger networks.The paper addresses this gap by investigating how layer balance shapes rank and using those insights to derive a backbone anti-collapse regularizer.Source: §2 · p. 3

3 METHOD

The method develops SACReg from a two-layer linear analysis, extends it to nonlinear representation covariance, and applies it to JE-SSL as λ-JEPA to protect backbone representations from collapse.

3.1 DERIVING A SPECTRAL ANTI-COLLAPSE MECHANISM IN LINEAR NETWORKS

  • Negative layer balance guarantees full-row-rank encoders and non-collapsed hidden representations when input covariance is positive definite.The linear analysis motivates a regularizer that dynamically drives the network toward a selected negative balance.Source: §3.1 · pp. 3–4
  • Encoder-side SACReg combines weight decay with a log-determinant penalty, and its gradient flow converges exponentially to a negative balance independent of initial imbalance.The ratio γ/λreg sets the selected balance and encoder-spectrum lower bound, while λreg controls convergence speed.Source: §3.1 · p. 4

3.2 SPECTRAL REGULARIZATION OF NONLINEAR REPRESENTATIONS

  • Nonlinear SACReg penalizes mean displacement, covariance scale, and small covariance eigenvalues, favoring zero-mean isotropic representations and increasing rank across initialization scales.For ε = 0, the preferred variance is γ/λcov; the task loss, architecture, and minibatch rank constraints may prevent attaining the preferred moments.Source: §3.2 · p. 5

3.3 SPECTRAL ANTI-COLLAPSE FOR JE-SSL

  • λ-JEPA applies SACReg to view-averaged backbone and projected representations, encouraging variation across images while preserving backbone features for downstream transfer.For existing SSL methods with a projected-space anti-collapse mechanism, the method adds only the backbone term; orthogonal slices handle feature dimensions larger than the number of image centers.Source: §3 · p. 3, §3.3 · pp. 5–6

4 EXPERIMENTS

Experiments evaluate λ-JEPA on large-scale image and video benchmarks and use controlled ImageNet-100 studies to examine SACReg’s effects across SSL objectives. λ-JEPA improves over explicitly regularized JEPA methods, while SACReg increases backbone rank and downstream performance without requiring greater augmentation invariance.

4.1 SETUP

  • The evaluation trains ViT-S and ViT-B from scratch on ImageNet-1k for 100 or 400 epochs and on a class-balanced 20% Kinetics-710 subset for up to 1,085 epochs.Controlled SACReg experiments use ViT-S on ImageNet-100; the longer image and video runs were not retuned and were used primarily to study scaling with extended training.Source: §4.1 · p. 6

4.2 PERFORMANCE ON IMAGE SELF-SUPERVISED LEARNING

  • At 100 epochs, λ-JEPA improves average transfer over LeJEPA by 8.9 points with ViT-S and over VISReg by 4.5 points with ViT-B; both sizes achieve the highest average transfer.Its ImageNet-1k performance approaches DINO and iBOT. The comparison is on ImageNet-1k linear probing and mean linear-probe transfer across eight datasets.Source: §4.2 · pp. 6–7, Table 1 · p. 7

4.3 PERFORMANCE ON VIDEO SELF-SUPERVISED LEARNING

  • On video, λ-JEPA’s ViT-B improves over LeVJEPA on SSv2 by 13.5 points at 240 epochs and 7.9 points at 1,085 epochs.At 240 epochs, it also improves over LeVJEPA and V-JEPA 2 on ImageNet-1k for both ViT-S and ViT-B. At 1,085 epochs, the CLS probe reaches 45.7 versus 44.6 for LeVJEPA’s mean-pooled-feature probe; these use different probes.Source: Table 2 · p. 7, §4.3 · pp. 7–8

4.4 CONTROLLED EXPERIMENTS ON IMAGENET-100

  • Across six SSL objectives on ImageNet-100, backbone SACReg substantially increases RankMe and improves kNN accuracy for every method, while linear-probe accuracy also improves.Cosine geometry becomes less concentrated, while projection/loss-space statistics change much less; the downstream gains do not come from making positive pairs more invariant.Source: Figure 2 · p. 8, §4.4 · pp. 8–9
  • On ImageNet-100, SACReg increases both class-center separability and view sensitivity on average across all six methods; for λ-JEPA, separability improves for 96 of 100 classes.View sensitivity moves in both directions for λ-JEPA, so the gains are not explained by uniformly suppressing augmentation-dependent variation.Source: Figure 2 · p. 8, §4.4 · p. 9

5 CONCLUSION … 2 GW⊤ 1 −W1G⊤W2. (10)

The paper introduces SACReg to address low-rank backbone representations that can persist despite anti-collapse regularization of projected representations. It applies this regularizer in λ-JEPA, reporting increased backbone rank and improved downstream performance while preserving augmentation-dependent variation.

5 CONCLUSION

  • At scale, λ-JEPA narrows the gap between explicitly regularized JEPA methods and DINO while improving frozen transfer across image and video SSL benchmarks.In controlled experiments, SACReg consistently increases backbone rank and improves downstream performance while preserving augmentation-dependent variation.Source: §5 · p. 9
  • Projected-representation anti-collapse does not guarantee a high-rank backbone, which can reduce representational capacity and harm downstream performance.The paper motivates SACReg through a linear analysis of negative layer balance, also known as λ-balance in feature-learning literature.Source: §5 · p. 9
  • Extending SACReg from CLS tokens and global views to patch-level features and local crops led to unstable training in the authors’ experiments.Local views increase augmentation-induced variation, making per-image view centers noisier and covariance across centers harder to regularize reliably.Source: §5 · p. 9

A.1.1 PROBLEM SETUP

  • The theoretical setup considers a two-layer linear network, with encoder W1, decoder W2, and end-to-end mapping M = W2W1.It frames supervised learning using training pairs (x_n, y_n).Source: §A.1.1 · p. 14

A.1.2 BALANCE CONSERVATION WITHOUT EXPLICIT REGULARIZATION

  • Under gradient flow on a differentiable task loss without weight decay or a log-determinant regularizer, layer imbalance is conserved, so initial λ_bal-balancedness persists during training.This balance-conservation property is also derived in Dominé et al. (2025).Source: §A.1.2 · p. 14

2 GW⊤ 1 −W1G⊤W2. (10)

  • The balance-conservation law is exact for continuous-time gradient flow, but finite-step gradient descent generally adds terms of order η^2, so exact conservation need not hold.The derivation equates the two gradient-flow expressions and subtracts them to establish equation 8.Source: §2 · p. 15

A.1.3 NON-COLLAPSE PROPERTY OF BALANCED INITIALIZATION

Negative layer balance prevents encoder collapse and, under positive-definite input covariance, rules out hidden-representation collapse. This non-collapse guarantee persists through unregularized gradient-flow training when imposed at initialization.

A.1.3 NON-COLLAPSE PROPERTY OF BALANCED INITIALIZATION

  • With positive-definite input covariance, negative layer balance yields hidden variance at least -κxλbal > 0 in every unit direction, ruling out complete and dimensional collapse.Negative layer balance is also sufficient to prevent encoder collapse.Source: §A.1.3 · p. 15
  • A network initialized with negative layer balance retains the corresponding non-collapse bounds throughout unregularized gradient-flow training; the guarantee depends on initialization balance.Source: §A.1.3 · p. 15

A.1.4 CONSTRAINED OPTIMIZATION CHARACTERIZATION … A.1.7 RICH AND LAZY REGIME

The appendix characterizes how the encoder-side regularizer selects a negative layer balance that prevents collapse, and how this balance shapes singular-mode allocation and learning regimes. It also establishes convergence to the selected balance and distinguishes linear from nonlinear regime behavior.

A.1.4 CONSTRAINED OPTIMIZATION CHARACTERIZATION

  • At constrained regularizer minima, negative layer balance makes the hidden representation covariance positive definite under positive definite input covariance, preventing complete and dimensional collapse.The selected balance is λbal = −γ/λreg.Source: §A.1.4 · pp. 15–16

A.1.5 BALANCE SELECTED BY THE ENCODER-SIDE REGULARIZER

  • From any admissible initialization, gradient flow preserves encoder full row rank and drives layer imbalance exponentially to λenc bal = −γ/λreg at rate 2λreg/τ.The result assumes λreg > 0, γ > 0, and an initially full-row-rank encoder.Source: §A.1.5 · p. 17
  • When initialized at the selected balance, the network preserves that balance throughout regularized gradient flow.The stated initialization is ∆(0) = −γ/λreg I_Nh.Source: §A.1.5 · p. 18

A.1.6 SINGULAR-MODE CHARACTERIZATION

  • For a fixed end-to-end mapping, the minimizing factors align their Gram matrices in a shared eigenbasis, with B² − A² = λbal I_Nh and BA = S.This characterizes how each singular mode is allocated across the encoder and decoder, up to hidden-layer orthogonal transformations and rotations within degenerate singular subspaces.Source: §A.1.6 · p. 19

A.1.7 RICH AND LAZY REGIME

  • In the linear setting, sufficiently strong negative imbalance can favor lazy dynamics while retaining the anti-collapse guarantee; zero-balanced initialization can support rich dynamics at sufficiently small initialization scales.The effect depends on architecture, initialization scale, and the sign of the balance.Source: §A.1.7 · p. 19
  • In nonlinear networks, full matrix imbalance is not conserved and need not induce lazy learning; asymmetric parameter scalings can instead accelerate rich feature learning while preserving non-collapse.The appendix distinguishes this nonlinear behavior from the linear regime discussion.Source: §A.1.7 · p. 19

A.1.8 EXPERIMENTAL VERIFICATION … A.2.2 LOG-DETERMINANT COVARIANCE REGULARIZER WITH THE MEAN

The experimental verification supports the predicted collapse-preventing balance in a two-layer linear network. The nonlinear extension regularizes representation mean and covariance, with its strict non-collapse guarantee depending on the numerical stabilizer being zero.

A.1.8 EXPERIMENTAL VERIFICATION

  • Figure 3’s verification found imbalance eigenvalues in [−2.005, −1.984] around the predicted −2 fixed point, with final MSE ≈5.47 × 10−4.The encoder-side log-determinant penalty drove the network toward the predicted balanced fixed point without impeding task fit.Source: Figure 3 · p. 20, §A.1.8 · p. 20

A.2.1 FROM LINEAR BALANCE TO NONLINEAR COVARIANCE REGULARIZATION

  • For centered, whitened inputs and ε = 0, the covariance penalty exactly recovers the linear encoder regularizer when λcov = λreg.In nonlinear or biased encoders, centered inputs may yield nonzero-mean representations, which covariance terms alone cannot control; a mean penalty favors zero-mean representations.Source: §A.2.1 · pp. 20–21

A.2.2 LOG-DETERMINANT COVARIANCE REGULARIZER WITH THE MEAN

  • Nonlinear SACReg controls representation location with a mean penalty, scale with a trace term, and small covariance eigenvalues with a negative log-determinant, but does not enforce Gaussianity.It regularizes the mean and covariance, not higher-order moments.Source: §A.2.2 · p. 21
  • The unique covariance-level minimizer is full-rank iff γ > λcovε; each eigenvalue’s optimum is max(γ/λcov − ε, 0).The mean and covariance minimize separately, with the mean optimum at zero.Source: §A.2.2 · p. 22
  • At ε = 0, the centered optimum centers representations, prevents vanishing covariance eigenvalues, equalizes the spectrum, controls scale, and has condition number one.Source: §A.2.2 · p. 22
  • When ε = 0, any finite upper bound on SACReg bounds the mean and keeps all covariance eigenvalues bounded above and away from zero.For ε > 0, SACReg only discourages covariance collapse, though its covariance-level minimizer remains full-rank when γ > λcovε.Source: §A.2.2 · pp. 22–23
  • The nonlinear extension does not preserve the linear balance identity W1⊤ = −(γ/λreg)I_Nh or generally conserve a difference between decoder Gram matrix and representation covariance.Source: §A.2.2 · p. 23

A.2.3 FEATURE LEARNING: TWO-LAYER RELU EXPERIMENT · A.2.4 FEATURE LEARNING IN THE VIT

In the two-layer ReLU experiment, negative λ and positive regularization favor higher rank, while SACReg also promotes feature learning across initialization scales. In the ViT experiment, representations and the empirical NTK drift substantially during self-supervised training.

A.2.3 FEATURE LEARNING: TWO-LAYER RELU EXPERIMENT

  • Negative λ and positive wreg favor higher rank in the Rich regime, but do not correspond exactly; negative-λ runs rise from a shared initial effrank frac of 0.467.The rank increase develops monotonically during training rather than reflecting a decaying initialization offset.Source: Figure 4 · p. 24, §A.2.3 · p. 25
  • Increasing wreg increases NTK kernel distance and extends the initialization-scale range with feature learning, although evolution remains limited at sufficiently large scales.The regularizer was applied directly to the hidden representation after the ReLU and before the readout.Source: Figure 4 · p. 24, §A.2.3 · p. 25
  • Following Kunin et al. (2024), the study tests nonlinear teacher–student regression with a two-layer ReLU student and applies the covariance regularizer to its hidden representation.The student has hidden width 50, and the regularizer is applied after the nonlinearity and before the readout.Source: §A.2.3 · pp. 24–25
  • At small initialization scales, varying λ alone yields kernel distances of 0.16–0.58, comparable to wreg; at large scales, both have little effect on kernel evolution.The authors interpret λ as modulating feature learning within an already rich regime, while wreg can also promote departures from lazy dynamics.Source: §A.2.3 · p. 25

A.2.4 FEATURE LEARNING IN THE VIT

  • In ImageNet-100 ViT-S/16 training with SACReg, feature drift increases with transformer depth and is largest after the final block.Drift is measured as one minus linear CKA of CLS representations from 4,096 held-out images at blocks 3, 6, 9, and 12.Source: §A.2.4 · pp. 25–26, Figure 5 · p. 26
  • The empirical NTK moves substantially from initialization during the first 25 epochs, then changes only slightly between checkpoints, consistent with rich feature learning.The drift estimate uses 256 validation images and eight fixed unit-norm output directions.Source: Figure 5 · p. 26, §A.2.4 · pp. 26–27

B IMPLEMENTATION OF THE REGULARIZER … C.1 DATASETS

The implementation pairs a multi-view invariance objective with SACReg computed on random feature slices. The expected sliced penalty retains the full regularizer’s preferred representation while allowing covariance estimation from more centers than dimensions.

B IMPLEMENTATION OF THE REGULARIZER

  • The image objective averages squared distances across projected views, has no negatives or anti-collapse term, and uses V=4 on ImageNet-100 and V=6 on ImageNet-1k and video.Source: B IMPLEMENTATION OF THE REGULARIZER · p. 27
  • When d ≥ B, the centered covariance has rank at most B−1, so SACReg uses random orthonormal projections, shared across GPUs and resampled each step, with ε=10^-4 and division by d′.The regularizer is evaluated on projected centers; dividing by d′ makes its scale approximately independent of slice width.Source: B IMPLEMENTATION OF THE REGULARIZER · p. 27
  • The slice width is d′=128 after the projector; at the backbone it is 128 for ViT-S and 256 for ViT-B, with independent random slices for backbone and projector.The corresponding backbone widths are d=384 and d=768, respectively.Source: B IMPLEMENTATION OF THE REGULARIZER · p. 27

B.1 RELATION TO THE FULL-COVARIANCE REGULARIZER

  • The expected sliced objective is nonnegative and uniquely minimized at zero mean and identity covariance, while preserving the mean and total variance in expectation.Averaging over random subspaces therefore retains the full regularizer’s preferred representation.Source: §B.1 · p. 28
  • Wider slices make the expected sliced objective increase monotonically toward the full objective, recovering it when d′=d.In practice, d′<d allows covariance estimation from substantially more centers than dimensions.Source: §B.1 · p. 28
  • A FIFO buffer concatenates detached view centers from previous steps with current centers for covariance estimation; only current centers carry gradients.Under data parallelism, the buffer contains centers from the full global batch.Source: §B.1 · p. 28
  • For ImageNet-1k and video, the invariance target uses an exponential moving average of the backbone and projector; ImageNet-100 uses the pairwise objective, while SACReg uses current-network centers.Source: §B.1 · p. 29

C.1 DATASETS

  • ImageNet-1k uses the standard split (Russakovsky et al., 2015), ImageNet-100 is the CMC subset (Tian et al., 2020a), and video training uses a deduplicated 20% union subset of 120,100 clips.Transfer evaluation covers DTD, Aircraft, Cars, CIFAR10/100, Flowers, Food, and Pets using the VISReg protocol (Wu et al., 2026); video encoders are evaluated on ImageNet-1k, Something-Something-v2, and Kinetics-400.Source: §C.1 · p. 29

C.2 MODELS AND TRAINING … C.6 DETAILED RESULTS

The supplied sections specify image and video training setups, loss-weight calibration, evaluation protocols, and baseline sources for assessing λ-JEPA and related methods.

C.2 MODELS AND TRAINING

  • ImageNet-1k runs train ViT-S/16 and ViT-B/16 for 100 and 400 epochs using six global views, AdamW, and shared per-size loss weights.Both use a two-layer projector, batch size 128, and bf16; no local crops are used.Source: §C.2 · p. 29
  • Controlled ImageNet-100 experiments compare each method’s baseline with a matched variant adding βh SACReg to its projector-input representation.Experiments use ViT-S/16, 100 epochs, three seeds, and otherwise preserve each method’s released loss, projector, and augmentation settings.Source: §C.2 · p. 29
  • Video training follows LeVJEPA’s pipeline and configuration, using six global views and evaluating the exponential-moving-average encoder.The setup uses ViT-S/16 and ViT-B/16 with 16 frames, stride 2, tubelet size 1, and 95% token drop; batch sizes are 768 and 512, respectively.Source: §C.2 · pp. 29–30

C.3 SETTING THE LOSS WEIGHTS

  • For existing methods, βh is calibrated so SACReg contributes a target 6% of the weighted backbone-gradient norm sum, rather than being separately tuned per method.The calibration uses the original loss terms at epoch 25 and a fixed SACReg gradient-norm reference of 0.30, chosen to avoid an artificially small weight at a collapsed backbone.Source: §C.3 · p. 30
  • For its standalone objective, λ-JEPA uses target loss-term pull shares of 0.44, 0.54, and 0.02, with total magnitude 57, calibrated from an earlier ViT-S run.The same calibration rule is used for ViT-S and ViT-B without further tuning.Source: §C.3 · p. 30

C.4 EVALUATION

  • Evaluation covers ImageNet-1k and ImageNet-100 linear probes and kNN, eight-dataset average linear-probe transfer, and frozen-encoder video probes.ImageNet-1k and Something-Something-v2 use attentive video probes; Kinetics-400 uses a linear classifier with specified L-BFGS settings because LeVJEPA does not release its classifier settings.Source: §C.4 · pp. 30–31
  • Frozen-feature statistics include RankMe and same-image versus different-image view cosines, while augmentation-thickness analysis estimates class-center separability and view sensitivity from held-out views.For that analysis, eight augmented views are stored for 10,000 training images; class-center separability can be negative because held-out squared residual error can exceed 1.Source: §C.4 · p. 31

C.5 BASELINES

  • Baseline evaluation includes released image checkpoints, a newly trained VISReg ViT-B/16, and video results taken from the LeVJEPA paper.The OK-AI model card lists 1.45M training images, more than the standard ImageNet-1k training split, and this difference was not corrected.Source: §C.5 · p. 31

D A CLOSER LOOK AT LEJEPA · E AUGMENTATION THICKNESS: DEFINITION AND INTERPRETATION · E.1 DEFINITION

The LeJEPA test shows that applying SIGReg at the encoder improves Gaussianity diagnostics without correcting the concentrated backbone spectrum. The augmentation-thickness analysis formalizes a trade-off: too little within-image variation can limit image-center organization, while too much can hinder single-view prediction.

D A CLOSER LOOK AT LEJEPA

  • Adding backbone SIGReg lowers negative-pair cosine (0.58 to 0.01) and diagonal Gaussian KL (0.96 to 0.07), but worsens linear-probe by 1.4 and kNN by 4.3 points.RankMe/d changes only from 0.07 to 0.09, while full-covariance KL remains high (3.89 → 3.03); unlike SIGReg, RSAC directly controls the covariance spectrum through its log-determinant term.Source: D A CLOSER LOOK AT LEJEPA · p. 32, Table 5 · p. 33

E AUGMENTATION THICKNESS: DEFINITION AND INTERPRETATION

  • Augmentation thickness has two opposing effects: too little can suppress meaningful augmentation-dependent variation, while too much can let within-image variation dominate separation between images.Source: E AUGMENTATION THICKNESS: DEFINITION… · p. 33

E.1 DEFINITION

  • The representation covariance decomposes into between-image center covariance Bh and within-image view covariance Ah; Bh measures variation across image centers, while Ah measures variation across views of one image.The paper defines augmentation thickness by normalizing Ah by Bh.Source: §E.1 · p. 33
  • Along a direction with nonzero between-image variance, augmentation thickness Θh is the ratio of within-image to between-image variation.Source: §E.1 · p. 34
  • The analysis distinguishes irreducible single-view prediction error from dh(y)2, the best affine prediction error of an image-level target from image centers.The target y is centered and has L2 norm at most 1.Source: §E.1 · p. 34
  • Theorem E.1 shows that when a target is not perfectly recoverable from an augmented view, very low augmentation thickness limits how well image centers can organize for that target.For fixed image-center organization and task direction, the augmentation-dependent contribution to optimal single-view prediction error increases with Θh.Source: §E.1 · p. 34
  • The view-sensitivity quantity used in Figure 8 isolates the augmentation-variation contribution to optimal single-view linear prediction error from the image-center organization term.It is used instead of raw directional thickness because it captures augmentation variation after optimizing the linear prediction.Source: §E.1 · p. 34

E.2 PROOF OF THEOREM E.1

The proof shows that the residual component contributes exactly dh(y)^2 to the optimal affine prediction error from h, while increasing augmentation thickness can only increase the view-sensitivity term.

E.2 PROOF OF THEOREM E.1

  • In the too-little-thickness case, the proof concludes by establishing Eq. 47.Source: §E.2 · p. 35
  • The residual component ξh contributes exactly dh(y)^2 to the optimal affine prediction error from h.The proof writes h = mh(Q) + ξh.Source: §E.2 · p. 35
  • Increasing augmentation thickness can only increase the view-sensitivity term.The proof states this conclusion for fixed a.Source: §E.2 · p. 36

Related papers

Loading 2609.35288v2…