LeJEPA: Provable and Scalable Self-Supervised Learning Without the Heuristics

Randall Balestriero; Yann LeCun

Source-linked AI summary

LeJEPA: Provable and Scalable Self-Supervised Learning Without the Heuristics

Randall Balestriero, Yann LeCun

arXiv:2511.08544v3Nov 11, 2025cs.LGcs.AIcs.CVstat.ML

TL;DR

JEPAs provide a promising route to self-supervised representations, but their training lacks reliable theory and practical guidance and is vulnerable to collapse. The paper identifies isotropic Gaussian embeddings as optimal, introduces SIGReg to enforce them, and combines SIGReg with prediction in LeJEPA, which matches state-of-the-art performance across diverse settings with a simple training recipe.

Problem
JEPA training lacks reliable theoretical and practical guidance and depends on brittle anti-collapse heuristics.
Method
LeJEPA combines the JEPA predictive objective with SIGReg, which matches embeddings to an isotropic Gaussian using projections and characteristic-function matching.
Results
LeJEPA matches state-of-the-art performance across domains and more than 60 architectures, including models with 1.8B parameters.
Takeaways & Limitations
The framework provides a theoretically grounded JEPA that eliminates collapse by design without teacher-student networks or stop-gradients.
Takeaways & Limitations
CDF-based statistical tests require sorting and synchronization, while Shapiro-Wilk testing was unstable in practice.

Abstract

from arXiv · show

Learning manipulable representations of the world and its dynamics is central to AI. Joint-Embedding Predictive Architectures (JEPAs) offer a promising blueprint, but lack of practical guidance and theory has led to ad-hoc R&D. We present a comprehensive theory of JEPAs and instantiate it in {\bf LeJEPA}, a lean, scalable, and theoretically grounded training objective. First, we identify the isotropic Gaussian as the optimal distribution that JEPAs' embeddings should follow to minimize downstream prediction risk. Second, we introduce a novel objective--{\bf Sketched Isotropic Gaussian Regularization} (SIGReg)--to constrain embeddings to reach that ideal distribution. Combining the JEPA predictive loss with SIGReg yields LeJEPA with numerous theoretical and practical benefits: (i) single trade-off hyperparameter, (ii) linear time and memory complexity, (iii) stability across hyper-parameters, architectures (ResNets, ViTs, ConvNets) and domains, (iv) heuristics-free, e.g., no stop-gradient, no teacher-student, no hyper-parameter schedulers, and (v) distributed training-friendly implementation requiring only $\approx$50 lines of code. Our empirical validation covers 10+ datasets, 60+ architectures, all with varying scales and domains. As an example, using imagenet-1k for pretraining and linear evaluation with frozen backbone, LeJEPA reaches 79\% with a ViT-H/14. We hope that the simplicity and theory-friendly ecosystem offered by LeJEPA will reestablish self-supervised pre-training as a core pillar of AI research (\href{https://github.com/rbalestr-lab/lejepa}{GitHub repo}).

1 Introduction

LeJEPA derives a lean JEPA from two principles: predictive agreement and isotropic-Gaussian embeddings. SIGReg enforces the distributional condition, removing common heuristics while supporting scalable, robust training across architectures and domains.

1 Introduction: JEPAs learn representations by maximizing predictive agreement between semantically related views, but their training remains vulnerable to representation collapse and brittle safeguards.Existing recipes use stop-gradients, asymmetric views, teacher-student networks, normalization, whitening, and delicate hyperparameter choices.
1 Introduction: LeJEPA is built from two axioms: solve the prediction task and enforce an isotropic Gaussian distribution for embeddings.The first follows standard JEPA practice; SIGReg is introduced to enforce the second.
1 Introduction: SIGReg matches embedding and target distributions through random projections and characteristic-function-based statistical testing.Its memory and computational complexity are linear in embedding dimension and sample size, with statistical guarantees and bounded gradients.
1 Introduction: LeJEPA eliminates collapse without stop-gradients or teacher-student architectures, using one trade-off parameter across datasets, architectures, and scales.The paper describes this unified design as working out of the box without hyperparameter tuning.
1 Introduction: Experiments span diverse architectures and domain-specific datasets, with LeJEPA matching or exceeding state-of-the-art methods and outperforming DINOv2-based transfer on Galaxy10 and Food101.The reported architectures include ViTs, ConvNeXts, ResNets, MaxViTs, and Swin Transformers at scales approaching 1 billion parameters.

2 Background and Notations

The paper frames JEPAs as predictive representation learners whose central unresolved challenge is preventing degenerate embeddings. It develops LeJEPA from first principles to connect theoretical conditions, scalable distribution matching, and practical pretraining.

2.1 Notations and Definitions: The dataset is represented as an (N, V, D) tensor for samples, views, and dimensions, with independent and identically distributed samples required by the theory.Views may represent augmentations, video frames, or other observations, and their ordering can be irrelevant.
2.1 Notations and Definitions: A deep network f_θ maps R^D observations to a K-dimensional embedding space, with θ collecting its learnable parameters.The architecture is treated as researcher-designed, while the study claims LeJEPA works across network choices.
2.2 The Need for Reliable Pretraining: JEPAs predict one view’s embedding from another view’s embedding while ensuring embeddings remain non-degenerate for downstream tasks.The intended outcome is solving numerous downstream tasks in zero or few shots.
2.2 The Need for Reliable Pretraining: JEPA anti-collapse mechanisms include whitening, negative samples, asymmetric views, and teacher-student networks, but their design resembles trial-and-error.The paper describes predictive-task design as relatively intuitive and anti-collapse design as under-specified.
2.2 The Need for Reliable Pretraining: These safeguards can suffer from degenerate minima, quadratic complexity, sensitivity to data and hyperparameters, and limited theoretical guarantees.The listed limitations motivate a more principled JEPA design.
2.3 Existing Literature: The paper’s approach derives a new JEPA from proved necessary conditions for optimality rather than post-hoc explanations of existing methods.Its stated goal is to reconcile exploratory research, scalability, and state-of-the-art performance.

3 Latent Euclidean: Embeddings Should be Isotropic Gaussian

The section establishes the isotropic Gaussian as the optimal embedding distribution for minimizing downstream prediction risk across linear and nonlinear probes. Anisotropy increases estimator bias and variance, motivating LeJEPA’s distributional design principle.

3 Latent Euclidean: Embeddings Should be Isotropic Gaussian: The isotropic Gaussian uniquely minimizes downstream prediction risk across linear and nonlinear probing under fixed variance constraints.The result is derived for linear probes and nonlinear methods including radius-based k-NN and kernel methods.
3 Latent Euclidean: Embeddings Should be Isotropic Gaussian: The analysis compares embeddings with identical column spans and total energy but different covariance geometries.The anisotropic covariance has unequal eigenvalues, whereas the isotropic covariance eigenvalues are all 1.
3 Latent Euclidean: Embeddings Should be Isotropic Gaussian: Anisotropy amplifies bias when Tikhonov regularization is used, with some downstream task exhibiting higher estimator bias than isotropic embeddings.This holds when the anisotropic covariance has unequal eigenvalues and λ > 0.
3 Latent Euclidean: Embeddings Should be Isotropic Gaussian: Anisotropic embeddings produce higher estimator variance than isotropic embeddings in linear and logistic-regression evaluations.Repeated training samples yield greater variation in learned parameters under anisotropic covariance, as illustrated in Figure 3.
3 Latent Euclidean: Embeddings Should be Isotropic Gaussian: These theoretical and empirical findings establish that LeJEPA embeddings should follow an isotropic Gaussian distribution to minimize worst-case downstream risk.The subsequent section introduces a regularizer intended to achieve this distribution.
3 Latent Euclidean: Embeddings Should be Isotropic Gaussian: For nonlinear probing, the isotropic Gaussian is the unique optimum for minimizing integrated square bias among distributions satisfying a scalar-based covariance constraint.The optimum’s distributional shape does not depend on the specific fixed total-variance value.

4 SIGReg: Reliable Isotropic Gaussian Regularization in High-Dimension

SIGReg turns isotropic-Gaussian matching into a directional statistical test that is differentiable, scalable, and theoretically consistent. Its sketching, test choice, and direction resampling address high-dimensional collapse while preserving practical training efficiency.

4.1 Hypothesis Testing as a Judge: SIGReg frames embedding matching as testing whether the encoder distribution Pθ equals a target Q, using empirical evidence against the null hypothesis.The framework uses a test statistic, critical value, and rejection rule before constructing a high-dimensional sketch.
4.1 Hypothesis Testing as a Judge: Directional tests project embeddings onto unit vectors, apply univariate statistics, and aggregate results over M directions into a global statistic.The directional distributions are push-forwards of Pθ and Q under projection by a unit-norm vector.
4.1 Hypothesis Testing as a Judge: Theorem 2 establishes that the aggregated directional test is consistent for the original multivariate distribution-matching hypothesis.The result leverages the union-intersection principle and a modified Cramér-Wold theorem; convergence in distribution also holds.
4.1 Hypothesis Testing as a Judge: SIGReg applies such tests toward the isotropic Gaussian, averaging directional statistics rather than taking their maximum to avoid sparse gradients.The regularizer can use different univariate tests, with the paper recommending the Epps-Pulley test.
4.2.1 Moments are Unstable and Insufficient: Moment-based tests face gradient and variance growth, while CDF-based tests require sorting, synchronization, and nondifferentiable operations; SIGReg instead benefits from smooth embeddings and resampling.The directional approximation bound decays as |A|^-2α/(K−1), and resampling can make M = 16 outperform fixed sets with thousands of directions.
4.2.3 Characteristic Functions are Stable, Scalable and Identifiable: Epps-Pulley is selected because its empirical characteristic function is differentiable and distributed-friendly, with bounded gradients and curvature and O(N) memory and computation.The characteristic function is computed as an average of complex exponentials and supports efficient all_reduce operations.

5 LeJEPA: Stable and Scalable Implementation

LeJEPA combines JEPA prediction with SIGReg, using characteristic-function tests to enforce isotropic Gaussian embeddings efficiently and without collapse-prevention heuristics.

SIGReg: SIGReg uses the Epps–Pulley test because it offers provable boundedness and scalable implementation for gradient-based training.The integral is approximated with trapezoidal quadrature; 17 knots are sufficient in the reported ablation.
SIGReg: SIGReg’s minibatch loss and derivative have bias of order O(1/n), with the reported bias remaining minimal for minibatches as small as 16.Unbiased alternatives include U-statistic debiasing or sample splitting, but the study does not explore them.
LeJEPA objective: The final LeJEPA loss combines the prediction loss with SIGReg applied to each view.The prediction setup uses global and local views, with all views predicting the global views.
Implementation: LeJEPA requires one trade-off hyperparameter and omits prototypes, stop-gradients, and teacher-student networks.The implementation besides model definitions, optimizers, and data loaders takes only a few dozen lines of PyTorch.
Connections: SIGReg’s slicing and distribution matching relate LeJEPA to sliced methods and recover VICReg in the limit of infinitely many slices under a degenerate statistical test.In that limit, SIGReg enforces zero mean and identity covariance in expectation.

6 LeJEPA: Empirical Validation

Experiments report reliable LeJEPA training across hyperparameters, architectures, datasets, and scales, alongside informative loss-based model selection and strong in-domain transfer results.

Empirical overview: LeJEPA trains reliably across diverse architectures and datasets, provides an informative training loss, scales to nearly 1 billion parameters, and learns semantic features without explicit supervision.The experiments cover reliability, model selection, in-domain pretraining, large-scale ImageNet-1k training, and unsupervised feature structure.
Stability: Performance remains stable across the number of views and λ on ImageNet-100 with a ResNet-50, motivating λ = 0.05 as a robust default.The commonly used configuration Vg = 2 and Vl = 8 transfers well to LeJEPA on ImageNet-1K with ViT-Large/14.
Stability: Integration range and quadrature-point count have negligible impact, while 512 SIGReg slices still yield competitive performance.The reported starting points are 17 integration points, domain [−5, 5], and 1024 slices.
Architectures and heuristics: LeJEPA works across approximately 50 architectures from 8 families without modification and removes several heuristics traditionally used to prevent collapse or instability.The reported removals include teacher-student structures, stop-gradients, and register tokens.
Model selection: LeJEPA’s combined training loss reaches about 85% Spearman correlation with downstream accuracy and nearly 99% correlation with α ≈ 0.4 across datasets and models.The alignment enables label-free self-supervised model selection and cross-validation.
In-domain pretraining: In-domain LeJEPA pretraining on Galaxy10 substantially outperforms DINOv2 and DINOv3 on frozen-backbone linear probing and full finetuning.The study uses default hyperparameters and reports additional in-domain results on datasets including flowers102 with 1,000 samples.
Scaling: LeJEPA reaches 77.1% online linear-probe accuracy with ViT-Large/14 and 78.5% with ConvNeXtV2-Huge on ImageNet-1K.The study also reports consistent transfer-learning gains over JEPA with a smaller model and shorter training schedule.
Feature structure: PCA of learned embeddings reveals correspondence to object boundaries and salient regions, and the features are explored for unsupervised video segmentation.The visualization follows the DINO procedure of applying PCA to embeddings.

7 Conclusion

The conclusion presents LeJEPA as a theoretically grounded and practically simple JEPA that eliminates collapse by design and remains effective across domains and model scales.

Theory and method: The paper proves isotropic Gaussian embeddings uniquely minimize worst-case downstream risk and introduces SIGReg to enforce that distribution tractably.LeJEPA combines this regularizer with the JEPA predictive objective to eliminate representational collapse by design.
Empirical scope: LeJEPA matches state-of-the-art performance across domains and more than 60 architectures, including models with 1.8B parameters, using fewer than 50 lines of core implementation.The conclusion frames this combination as a mathematically rigorous foundation for practical self-supervised algorithm design.

A.1 kNN Probing

The probing analysis establishes isotropic Gaussian embeddings as optimal for downstream prediction risk under the paper’s stated regularity and covariance assumptions, including k-NN and kernel probes.

k-NN probing: Radial k-NN probing uses a sample-dependent neighborhood size to improve performance for non-uniform sample distributions.The analysis defines the training embeddings and target function under smoothness and covariance regularity conditions.
k-NN optimality: The isotropic Gaussian uniquely minimizes integrated squared bias for k-NN probes under the stated assumptions.This yields a unique optimal embedding density for both linear and k-NN probes.
Kernel probing: Kernel probing performance depends crucially on the distribution of training points, with Nadaraya–Watson estimation analyzed under additional tail conditions.The analysis specifies conditional target variance and light-tail assumptions for the kernel estimator.
Kernel optimality: Under fixed mean and total-covariance constraints, the isotropic Gaussian uniquely minimizes kernel-regression bias and variance.The result is established for the paper’s restricted smoothness class of target functions.

B.1 Proof of lemma. 1

The proof compares ridge-regression bias under isotropic and anisotropic input covariance with equal total variance. It shows that anisotropy can increase downstream estimator bias when the true parameter aligns unfavorably with the covariance structure.

The argument derives bias from the ridge estimator under the linear model with centered label noise.The estimator is analyzed through its closed-form expression and bias decomposition.
The comparison fixes equal total variance while varying whether the covariance is isotropic or anisotropic.This isolates covariance shape as the source of the bias difference.
Anisotropy produces a strictly smaller minimum eigenvalue than the corresponding isotropic covariance, yielding the strict bias comparison.The strict inequality is used in the final downstream-task construction.
Anisotropic covariance increases ridge-regression bias for downstream tasks whose true parameter aligns with the smallest-eigenvalue direction.The proof constructs an unfavorable parameter direction using the eigenvector associated with the smallest covariance eigenvalue.

B.4 Proof of thm. 7

The proof establishes that the isotropic Gaussian uniquely minimizes the Fisher-information and integrated squared-bias objectives under scalar covariance constraints. This provides the distributional basis for targeting isotropic Gaussian embeddings.

Without an orthogonality assumption, the cross term remains finite, but the leading p-dependent bias term is still τ^2 gJ(p).This preserves the Fisher-information dependence under the more general derivative law.
The isotropic Gaussian uniquely minimizes integrated squared bias among distributions satisfying a scalar covariance constraint.The result is stated for the downstream prediction analysis and follows from the Fisher-information characterization.
Under trace, determinant, Frobenius, or spectral-radius constraints, tr(Σ^-1) is minimized when Σ is isotropic.The minimizing covariance has the form Σ = sI_d, with s determined by the constraint.
For fixed covariance Σ, Fisher information is minimized by the Gaussian N(0, Σ), attaining tr(Σ^-1).Equality holds only for the Gaussian with that covariance.
Theorem 9 concludes that p_G = N(0, sI_d) is the unique minimizer up to null sets for each stated scalar constraint.The proof combines the fixed-covariance Gaussian result with the covariance optimization.

B.8 Proof of lemma. 3

The proof uses characteristic functions and dense directional sampling to establish projection-based distributional equivalence and consistency. Sobolev smoothness then controls the approximation error from finitely many sampled directions.

Equality of all one-dimensional projections implies equality of the corresponding multivariate distributions through characteristic-function uniqueness.The proof decomposes any t into a magnitude and unit direction, then transfers projection equality to characteristic functions.
Dense direction sets eventually include a separating direction when two distributions differ, establishing consistency of the projection test.The rejection probability is controlled under the null and converges to one under the alternative.
The finite-direction approximation error is bounded using Sobolev regularity and spherical interpolation with M sampled directions.Quasi-uniform points on the sphere support reconstruction bounds derived from spherical harmonics and Marcinkiewicz–Zygmund inequalities.
Higher-order moment tests require convergence weights because higher moments may grow rapidly.The proof gives w_k = 1/k! as an example of a suitable weighting choice.

B.11 Proof of thm. 3

The proof contrasts moment-based and characteristic-function-based distribution matching. It shows that moment gradients can be unbounded, whereas empirical characteristic-function gradients and curvature remain uniformly controlled, with deterministic discretization adding only approximation error.

Empirical characteristic-function gradients are uniformly bounded and Lipschitz, with scale controlled by minibatch size and integration parameters.This follows from the unit modulus of complex exponentials and bounded characteristic functions.
Moment-test gradients are nonconstant polynomials in each sample and therefore unbounded for orders k ≥ 2.When the top weighted deviation is nonzero, gradient magnitude grows like |X_i|^(k-1).
Deterministic quadrature replaces the integral without affecting sampling unbiasedness, introducing only deterministic approximation error.A finite grid and Gaussian window implement the practical characteristic-function calculation.
The isotropic covariance proof establishes zero mean and identity covariance from constraints on all unit-direction projections.It separately verifies diagonal and off-diagonal covariance entries.