PaperlayerView exact revision on arXiv

Source-linked AI summary

Proving and Improving the Platonic Representation Hypothesis with JEPAs

Randall Balestriero, Yann LeCun

arXiv:2610.39302v1cs.AIcs.CVstat.ML

TL;DR

The Platonic Representation Hypothesis proposes that models trained on different modalities converge, but practical JEPA objectives lacked a proof, a prediction of when it holds, and a recipe for improving alignment. The paper proves a condition for JEPA alignment and reports that good views align better and more efficiently than next-token prediction.

  • Problem

    For practical self-supervised objectives, including JEPAs, the Platonic Representation Hypothesis lacked a proof, a prediction of when it holds, and a recipe for improving alignment.

    Source: §1 · p. 2

  • Method

    The paper proves that separately trained JEPAs align up to rotation when modalities use positive pairs representing the same world states, and characterizes next-token prediction as a bad-view JEPA.

    Source: §1 · p. 2

  • Results

    Good views align best across data, encoders, and modalities; they need 42 times fewer samples than next-token prediction for equal alignment, and a 109M text JEPA exceeds evaluated language models through 72B.

    Source: §1 · p. 2

  • Takeaways & Limitations

    Positive views, rather than scale alone, are the lever the paper identifies for building platonic representations.

    Source: §7 · p. 15

    Good views depend on identifying texts that describe the same scene; without an image captioner, text-only paraphrases and alt-text reach .060 on WIT versus .107 with recaptions.

    Source: §7 · p. 15

Abstract

from arXiv · show

Whether models trained on different modalities converge to the same representation of the world is central to AI. The Platonic Representation Hypothesis (PRH) conjectures that they do as they scale, but without a proof it offers neither a guarantee nor a recipe. We turn the hypothesis into a theorem. First, we prove a \emph{Platonic Representation Theorem}: Joint-Embedding Predictive Architectures (JEPAs) trained separately on any number of modalities learn the same representation up to one rotation whenever their positive pairs are two observations of one scene--\emph{good views}--and their misalignment is bounded otherwise. Second, we show that next-token prediction is a JEPA with \emph{bad views}, parts of a single document, which explains why language models align with vision only through scale. Third, we validate the theory from 145k captions to the texts of 940M web images, and uncover a scaling law of alignment that predicts unseen runs within seed noise. Good views need 42 times fewer training samples than next-token prediction on the same corpus to reach the same alignment, and a 109M-parameter text JEPA aligns with DINOv2 and CLIP's image encoders better than every language model we evaluated, up to 72B parameters, and than CLIP's text encoder: next-token prediction is a remarkably inefficient route to platonic representations. As a showcase, one rotation turns separately trained encoders into an image captioner (CIDEr .838 on COCO). Even that rotation needs no pair: on the latest pair-free benchmark, our text JEPA beats every zero-shot accuracy reported with language models up to 8B parameters, e.g., 64\% against at most 56\% top-5 on ImageNet-100. Positive views, and not scale alone, are thus the lever to build platonic representations.

1 Introduction

The paper addresses the lack of a proof or training recipe for the Platonic Representation Hypothesis and proves that separately trained JEPAs align up to rotation under shared positive world-state pairs. It argues next-token prediction uses bad views and reports stronger alignment from good views, alongside captioning and pair-free zero-shot applications.

1 Introduction

  • Under matching positive world-state pairs and unique-up-to-rotation optima, separately trained JEPAs across any number of modalities learn the same representation up to rotation; otherwise, mismatch bounds alignment.The theorem applies to modalities that render one world; the misalignment bound depends on the mismatch between positive pairs.Source: §1 · p. 2
  • Next-token prediction is a bad-view JEPA: its positives pair a document prefix with its continuation, offering an account of why language models align with vision only partly and mostly through scale.Its anti-collapse term is the softmax log-partition.Source: §1 · p. 2
  • Across captions, web-image texts, and video, good views align best in nearly all comparisons, and an alignment scaling law predicted 85 new runs within seed noise.The evaluation spans 145k Flickr captions to texts of 940M web images, with image encoders trained from scratch or frozen.Source: §1 · p. 2
  • Good views need 42 times fewer samples than next-token prediction for equal alignment, while a 109M text JEPA beats evaluated language models through 72B and CLIP’s text encoder.On the same corpus, the text JEPA reaches next-token prediction’s alignment with 21.6M samples versus 900M.Source: §1 · p. 2
  • A single rotation connects frozen DINOv2 and DINOv3 image encoders to a language model fine-tuned on text alone for captioning, reaching CIDEr .838 on COCO without training a network on image-caption pairs.The rotation is the only part fitted on image-caption pairs.Source: §1 · p. 2
  • With unpaired images and captions, the 109M text JEPA beats all reported zero-shot accuracies across 21 vision-language pairs and five settings, topping identical-condition Qwen3-8B by 17 ImageNet-100 points.The comparison uses zero-shot classification.Source: §1 · p. 3

2 Shared Positives Provably Make Representations Platonic

The theory makes shared positive pairs the basis for cross-modal agreement: under stated JEPA assumptions, separately trained encoders learn one world embedding up to rotations, while differing positives bound alignment. Good views connect observations of the same scene; bad views and next-token prediction instead induce kernels shaped by data and nuisance.

2.1 The Platonic Representation Theorem

  • With shared positive-pair distributions and minimizers unique up to rotation, separately trained modalities learn one world embedding up to a rotation each.The theorem applies to injective renderings of one finite world at population optima; for whitened JEPAs, a spectral gap ensures uniqueness.Source: §2.1 · p. 3
  • For whitened JEPAs with second-order anti-collapse, the learned representation is the graph positives draw’s top eigenvectors, unique up to rotation when the spectrum has a gap.The spectral result assumes a gap after the k-th eigenvalue; the paper leaves uniqueness under LeJEPA’s full-distribution constraint as an assumption.Source: §2.1 · p. 3

2.2 When Positives Differ

  • Theorem 2 bounds post-rotation misalignment by positive-kernel mismatch and spectral separation; in 300 generated worlds, error stayed below the bound, reaching at most 18% of it.The tested kernels shared a marginal, and half the worlds were clustered with small spectral gaps.Source: §2.2 · p. 4, Figure 2 · p. 4

2.3 Good-View and Bad-View JEPAs

  • Good views pair observations of related world states and avoid nuisance, whereas bad-view corruptions can make encoders spend embedding directions on modality-specific nuisance that rotations cannot align.Under the stated correlation and spectral-gap conditions, a bad-view JEPA uses at least one direction for nuisance exactly when nuisance correlation exceeds the state’s k-th-direction correlation.Source: §2.3 · p. 5
  • Views and data jointly determine the induced world kernel: good views recover the world’s kernel, while bad-view agreement depends on whether data make their positives connect the same states.Weak corruptions need denser samples to link observations within a scene; positives that cross scenes cap alignment, while good views align exactly from one sample per state.Source: §2.3 · pp. 5–6

2.4 Next-Token Prediction is a Bad-View JEPA

  • Next-token prediction is a JEPA with prefix-continuation positives and a log-partition anti-collapse term; shared continuations create weak implicit positives that may connect scenes only through dense data.The paper presents nuisance capture and the resulting explanation for scale-dependent vision alignment as heuristic, not a corollary, because next-token prediction is not a spectral JEPA and is identified only up to GL(k).Source: §2.4 · p. 7

3 Good Views Improve Alignment

Across modalities, JEPAs trained with views drawn from different observations of the same scene align better than JEPAs trained on corruptions of one observation or with next-token prediction. Trimming views is the most consequential recipe choice, and good views help on both sides of image–text alignment.

3.1 Good Views Align Across Modalities

  • 4.4% of test images retrieve the true caption first through one rotation with caption views, 44 times chance, while masked spans and next-token prediction remain well below.Adding two captions of the same image to next-token prediction barely changes mutual 10-NN (.042 to .043) or R@1 (.021 to .023).Source: Table 1 · p. 7, §3.1 · p. 8, Figure 5 · p. 8

3.2 Trimming Makes Good Views, the Rest Barely Matters

  • Trimming matters most: whole captions as views lower R@1 from .246 to .185, while other tested settings move the mean by at most .016.Gradient clipping prevents most observed collapses; without it, 3 of 36 default-weight runs collapse abruptly.Source: §3.2 · pp. 8–9, Figure 7 · p. 9

3.3 Good Views Help on Both Sides

  • On MSR-VTT, mutual 10-NN is .067 with good views on both sides, versus .039 with neither; DINOv2 aligns better with image JEPA crops than frames (.144 vs. .132).Good text views alone reach .059 and good image views alone .043; the ordering holds in every seed.Source: §3.3 · p. 9, Figure 8 · p. 9

3.4 Good-View JEPAs Beat a Pretrained Language Model and Approach Pair-Trained Encoders

  • 24.6% of test images retrieve the true caption first with the good-view JEPA, versus .133 for the bad-view JEPA and .118 for next-token prediction.With the rotation fitted on all 30k Flickr pairs, good-view R@1 reaches .346; on web text alone it reaches .464.Source: §3.4 · p. 10, Figure 9 · p. 10

4 Next-Token Prediction Scales Poorly Compared to Good-View JEPAs

Good-view JEPAs scale alignment more efficiently than next-token prediction and outperform the evaluated language models across the reported vision comparisons. Alignment also follows an empirical scaling law within a tested range, with accuracy limits beyond that range.

4 Next-Token Prediction Scales Poorly Compared to Good-View JEPAs

  • The 109M good-view JEPA trained on web text beats the evaluated language models despite reading 116B tokens, while those models were trained on 15–18T tokens.The paper reports that good views need 42 times fewer samples to reach the same alignment.Source: §4 · p. 10

4.1 Good Views Scale on Web Text

  • With 50M web-image texts, the 33M good-view JEPA reaches .108 on WIT, versus .053 for bad-view JEPA and .071 for next-token prediction trained with the same setup.Its WIT alignment rises from .034 on Flickr captions to .125 with texts of 50M images; the 50M and 100M models are level with Qwen2.5-3B and -7B (.125 and .118).Source: §4.1 · p. 10, Figure 10 · p. 11

4.2 Good Views Need 42 Times Fewer Samples

  • 42 times fewer samples let a good-view JEPA reach the WIT alignment that next-token prediction reaches after 900M samples: .109 after 21.6M versus .111 after 900M.The comparison uses the same 109M encoder and corpus; good views also reach higher Flickr R@1 (.388 against .335).Source: §4 · p. 10, §4.2 · p. 11

4.3 A 109M Text JEPA Beats Language Models up to 72B Parameters

  • A 109M good-view JEPA reaches .184 on the PRH benchmark with DINOv2 ViT-B, ahead of every evaluated language model, including models up to 72B parameters.The lead also holds across DINOv2 sizes, other reported vision families, and the larger COCO benchmark; same-text next-token and bad-view controls fall far behind.Source: §4.3 · p. 11, Table 3 · p. 12

4.4 A Scaling Law for the PRH

  • The scaling law predicts 85 unseen runs with Flickr R@1 RMSE .0093, close to the .0096 difference between two seeds of one run.At 320M samples it overestimates runs with good views and has no term for repeated data; its exponents are poorly identified and it summarizes one recipe empirically.Source: Figure 11 · p. 12, §4.4 · p. 13

5 Platonic Representations as a Tool

A rotation can connect separately trained image and text encoders for captioning, and the paper also tests alignment without paired data. The results show captioning transfer across datasets and strong pair-free alignment on the reported benchmarks.

5.1 Captioning Without Training a Network on Image–Caption Pairs

  • The caption decoder reconstructs captions from frozen good-view text-JEPA embeddings, with embedding noise matched to the residual image–caption alignment error.The rotation is the only component fitted on image–caption pairs; the paper reports transfer peaking at noise level σ = 3.Source: §5.1 · p. 13
  • CIDEr .838 on COCO is reached with a single rotation fitted on 30k Flickr pairs and a text-only decoder fine-tuned on text.The setup uses frozen DINOv2 and DINOv3 image encoders and consensus decoding of 32 captions.Source: §5.1 · p. 13, §5 · p. 13
  • The Flickr-fitted rotation carries over to video: on MSR-VTT, the captioner reaches CIDEr .246 without video training, while averaging eight frames raises fit-free mutual 10-NN from .148 to .167.The captioning score is below video–language systems and decoders trained on MSR-VTT captions.Source: §5.1 · pp. 13–14

5.2 Alignment Without Any Pair

  • 64.4% ImageNet-100 top-5 accuracy is achieved by the 109M good-view JEPA with no pairs, above every reported zero-shot accuracy in the comparison, whose maximum is 56%.On the same pair-free COCO protocol, it reaches FOSCTTM .018; its Flickr retrieval is 16.4% ± 10.4% against 0.1% chance.Source: §5.2 · p. 14, Table 4 · p. 14

6 Related Work

The paper situates its contribution among theories of representation alignment, identifiability, and methods for matching embedding spaces. Its distinction is to characterize which separately trained objectives make encoders alignable, rather than only how to align given encoders.

6 Related Work

  • Earlier PRH theory relies on idealized contrastive settings, while the cited deep-linear-network result and multilingual-translation theory address different model or alignment settings.The paper frames lossy observations as an open issue for the prior PRH argument and distinguishes its own focus on mismatched positive pairs.Source: §6 · p. 14
  • Prior identifiability work recovers latent variables up to simple transformations, including across modalities when trained jointly; this paper instead studies separately trained modalities.Its spectral JEPA analysis builds on prior work on self-supervised learning and positive-pair structure.Source: §6 · p. 14
  • Methods for aligning existing embedding spaces range from orthogonal Procrustes to shared-geometry matching without pairs; the paper asks which training objectives make encoders alignable.The authors describe their Procrustes map as the simplest alignment tool and use pairs only to fit one orthonormal map for captioning.Source: §6 · p. 15
  • Unlike prior text-only captioners built on jointly trained encoders, this approach uses separately trained encoders and fits only one map on image–caption pairs.The paper distinguishes this from paired linear-map approaches and unpaired captioning methods based on adversarial training or shared detector concepts.Source: §6 · p. 15

7 Conclusion

The paper states conditions under which separately trained JEPAs recover aligned representations and argues that positive views, rather than scale alone, are the lever for doing so. It reports empirical alignment gains while noting that the theorems and experiments have important scope limits.

Conclusion

  • The PRH becomes a theorem for population optima on injective renderings when modalities agree on positive world-state pairs and minimizers are unique up to rotation.When positive pairs are mismatched, misalignment is bounded; corrupting one observation can allocate dimensions to modality-specific nuisance.Source: §7 · p. 15
  • Good views align best across data, encoders, and modalities, need 42 times fewer samples than next-token prediction for the same alignment, and yield a 109M text JEPA that beats evaluated language models up to 72B parameters on the paper’s benchmark.The paper also reports a scaling law that predicts unseen runs within seed noise.Source: §7 · p. 15
  • The theorems assume a finite world with injective renderings and matched marginals, while real modalities are lossy and finite-sample training relies on network inductive bias.Uniqueness is proved for second-order anti-collapse, not LeJEPA’s full-distribution constraint.Source: §7 · p. 15
  • The main empirical caveat is that good views require knowing which texts describe the same scene, using human caption groups or captioner-generated texts for Recap-DataComp.Without an image captioner, alt-text and text-only paraphrase reach .060 on WIT versus .107 with recaptions; comparisons between views on identical texts do not rely on that difference.Source: §7 · p. 15
  • The paper argues that positive views, rather than scale alone, are a lever for building platonic representations and points to audio, video, and actions as future modalities.It characterizes next-token prediction as an inefficient route to platonic representations.Source: §7 · p. 15

A Proofs

The appendix proofs follow the main text’s subsection order, with one proof subsection corresponding to each main-text subsection.

A Proofs

  • The appendix presents proofs in the same order as Section 2, with one subsection per main-text subsection.Proposition 3 and Theorem 4 are stated informally in the main text, and their formal statements precede their proofs.Source: A Proofs · p. 20
  • Proposition 3 and Theorem 4 receive formal statements before their proofs, although the main text states them informally.Source: A Proofs · p. 20
  • The appendix organizes proof material by the corresponding main-text subsections.Source: A Proofs · p. 20

A.1 Notation and Objectives

This section defines a Markov operator from the positive-pair distribution and expresses the JEPA objective on world states. It also identifies objectives that fit the stated objective form.

A.1 Notation and Objectives

  • The positive-pair distribution defines a Markov operator whose eigenfunctions yield the spectral embedding used in Proposition 1.The operator is self-adjoint in L2(p), and its eigenfunctions are orthonormal there.Source: §A.1 · p. 20
  • On world states, the objective is an expectation over positive pairs of a loss plus a term depending on the embedding distribution.Source: §A.1 · p. 20
  • Spectral contrastive learning and whitened MSE fit the objective form; VICReg and Barlow Twins do so in a whitened hard-constraint limit.LeJEPA’s SIGReg term is rotation invariant in expectation over its projection directions.Source: §A.1 · p. 20

A.2 The Platonic Representation Theorem (Section 2.1)

The proof reduces modality-specific encoders to functions on the shared world by composing with injective renderings. It then uses the spectral characterization of minimizers to establish agreement up to rotation.

A.2 The Platonic Representation Theorem (Section 2.1)

  • Injective renderings make composition with the rendering a bijection between encoders on a modality and functions on world states, preserving the objective.This coordinate reduction lets the proof compare independently trained modalities through their shared world-state objective.Source: §A.2 · p. 20
  • The rendering lemma’s equivalence relies on injectivity, which ensures an inverse mapping between modality encoders and world-state functions.Source: §A.2 · p. 20
  • If the shared world-state objective has minimizers in one rotation orbit, each modality’s optimal encoder composed with its rendering belongs to that orbit.Thus any two modalities agree on world states up to their respective orthogonal rotations.Source: §A.2 · p. 20
  • Under whitened constraints and a spectral gap, the minimizing embedding spans the eigenfunctions for the k largest nonconstant eigenvalues.Ky Fan’s maximum principle gives the maximizing subspace; two orthonormal bases of that subspace differ by an orthogonal matrix.Source: §A.2 · p. 20

A.3 When Positives Differ (Section 2.2)

When positive-pair kernels differ, JEPA representations are aligned only up to a bound determined by kernel mismatch. With a shared world embedding, paired observations can identify the rotation exactly when their embedding matrix has full rank.

A.3 When Positives Differ (Section 2.2)

  • The representation mismatch is bounded using the spectral gap of the second kernel and the difference between the two kernels on the state subspace.The proof applies a Davis–Kahan variant to the top-k eigenspaces after removing the common constant eigenvector.Source: §A.3 · pp. 20–21
  • A rank-k matrix of paired embeddings uniquely determines the rotation between representations.For Y = QX, the objective ∥Y − RX∥² is minimized at R = Q; if X has rank k, no other R fits exactly.Source: §A.3 · p. 21

A.4 Good-View and Bad-View JEPAs (Section 2.3)

Good-view JEPAs learn representations of world state while ignoring independent nuisance variation, whereas bad-view positives can devote dimensions to nuisance. More generally, alignment depends on the world kernel induced by the views and on how much nuisance they preserve or mix.

A.4 Good-View and Bad-View JEPAs (Section 2.3)

  • For two independently corrupted observations, the alignment error is bounded below by 2k(1 − min_m ρ(f_m)); perfect retention of world information requires an encoder to be a function of state.The proof relates cross-view covariance to conditional covariance given the state.Source: §A.4 · p. 21
  • Good-view positives make the JEPA representation a function of world state alone, allowing modalities with the same world kernel to agree up to rotation.This holds under the stated spectral separation condition.Source: §A.4 · p. 21
  • Bad-view positives retain shared nuisance, and when its leading eigenvalue exceeds α_k, at least one top-k direction captures nuisance and alignment error is at least 2r.The representation has r zero covariance eigenvalues and ρ(f) = 1 − r/k.Source: §A.4 · p. 21
  • The general view-and-data theorem measures mismatch through nuisance preservation and state–nuisance mixing, requiring a positive spectral separation δ_m.Its scope includes good views, invariant corruptions, and mixtures; restrictions that merge observations of one state can lower nuisance and mixing terms, while biases merging different states or acting softly are outside the model.Source: §A.4 · p. 22
  • When two modalities share an induced world kernel, their representation mismatch is bounded by the kernel mismatch plus the approximation errors τ_1 and τ_2.The theorem applies to paired observations under any coupling with the same state marginal.Source: §A.4 · pp. 22–23

A.5 Next-Token Prediction is a Bad-View JEPA (Section 2.4)

Next-token prediction's softmax objective has a log-partition term that serves as anti-collapse regularization. Its predictions depend on the context representation and the vocabulary table, with an invariance under invertible changes of representation coordinates.

A.6 Next-Token Prediction is a Bad-View JEPA

  • The softmax log-partition A(h) = log ∑v e^⟨h,wv⟩ involves the whole output table but not the target token y.The log-partition is bounded in relation to the maximum logit, with a gap at most t log |V| after temperature scaling.Source: §A.5 · p. 23
  • For a full-support conditional distribution, expected next-token loss is minimized when the softmax predictions equal that distribution, if the context dimension can represent its log probabilities modulo constants.The required dimension is at least the rank of the conditional log-probability matrix modulo constants.Source: §A.5 · p. 23
  • Applying any invertible transform to context vectors while applying its inverse transpose to output vectors leaves the logits unchanged.This is a GL(k) invariance of the factorization.Source: §A.5 · p. 23

A.6 Numerical Checks

Numerical checks on finite worlds support the paper's spectral solutions, alignment bounds, and good-view versus bad-view predictions. In the reported random-world checks, informative bounds were substantially looser than observed errors.

A.6 Numerical Checks

  • Across random finite worlds, the numerical checks confirmed the spectral solution, rotation agreement for good-view modalities, and predicted nuisance losses for bad views.The checks also covered kernel-mismatch bounds and recovery of rotations from generic paired embeddings.Source: §A.6 · p. 23
  • On random worlds with one to four observations per state, bounds held for positives mixing good views, self-pairs, and random pairs.The checks covered both stated bounds and reported them as informative only in some cases.Source: §A.6 · p. 23
  • For Theorem 4 bounds, the error was at most 32% of the bound in (i) and 10% in (ii), with median error-to-bound ratios of 11% and 0.8%, respectively.The reported cases were 2k–2,848 for (i) and 307 for (ii).Source: §A.6 · p. 23

B Experimental details

Experiments use a shared image–text JEPA recipe on Flickr30k, with additional video and large-scale web-text runs. Evaluation follows the WIT-1024 alignment protocol and includes retrieval and video tests.

Image encoder

  • Image views comprise two global 224² crops and six local 96² crops, with random flips and photometric augmentations.Augmentations include color jitter, grayscale, Gaussian blur, and solarization.Source: B Experimental details · p. 23

Text encoders

  • Text JEPA views retain contiguous spans of 50–100% of a caption globally and 15–50% locally, with 15% of kept tokens masked.The next-token model uses causal attention and mean-pooled final hidden states; on Flickr, mean pooling scores R@1 .119 against .061 for last-token pooling with DINOv2 ViT-B.Source: B Experimental details · p. 23

Optimization

  • Flickr runs use AdamW at learning rate 5 · 10−4 for 200 epochs, with batch 256, bf16, cosine decay, and three seeds.Longer or larger text runs generally use learning rate 10−4, gradient clipping at 1.0, and one seed unless stated otherwise.Source: B Experimental details · p. 23

MSR-VTT

  • Video experiments use 7,010 MSR-VTT training clips and 1,000 test clips, sampling eight evenly spaced frames per clip.Good image views draw crops from separate frames, whereas bad views crop one frame per clip and epoch; evaluation uses the middle frame and JSFUSION test caption.Source: B Experimental details · p. 24

Evaluation

  • WIT-1024 alignment uses released-code features, clipping, L2 normalization, mutual 10-NN, and the maximum score over layer pairs.The protocol uses the first caption from each pair and per-layer vision CLS and language mask-weighted mean features.Source: B Experimental details · p. 24

C Additional results

The additional results examine how good-view training, its positive views, and its text data affect alignment in text-only and pair-free settings, including scaling-law behavior and its limits at very long training. They also assess alignment across datasets, vision-encoder families, and metrics, and test the effects of training views, data scale, and identified web-image overlap.

Next-token overfitting and recipe robustness

  • Next-token validation loss bottoms at 3.04 near epoch 29 and rises to 4.49 by epoch 189; good-view runs can collapse abruptly, though clipping prevents most default-recipe collapses.Early-stopping next-token prediction gives R@1 .134 against DINOv2, versus .246 for the good-view JEPA; without clipping, 3 of 36 default-view runs collapse.Source: C Additional results · pp. 24, 26, 27, Table 8 · p. 27

Nuisance on real encoders

  • Theorem 3’s lower bound holds for every text encoder, and its scene-explained fraction ranks nine encoders almost identically to their alignment (Spearman .98).On Flickr captions, the ranking has one adjacent-pair difference.Source: C Additional results · p. 25

Which texts give the gain

  • On WIT, good views of alt-text and its paraphrase score .060 against .055 for bad views of alt-text, while LLaVA recaptions with the same bad views score .092 against .056 for alt-text.On Flickr caption retrieval, LLaVA recaptions score R@1 .348 and .386 at the two scales; the results show gains from both text choice and view construction.Source: C Additional results · pp. 24–25, Table 7 · p. 25

The scaling law in detail

  • A registered scaling law predicts 85 unseen runs with RMSE .0093, matching the .0096 difference between reruns with different seeds, but overestimates long good-view training at 320M samples.At 320M samples, good-view runs reach R@1 .418 against .455 predicted at q = 1 and .348 against .389 at q = .1; bad views reach .338 against .337.Source: C Additional results · p. 26

Few pairs and no pairs

  • The captioner improves COCO CIDEr from .668 to .838 as stronger frozen encoders, a pretrained language model, and consensus decoding are added.Only the rotation is fitted on image-caption pairs; the decoder does not see images.Source: Table 10 · p. 29

Alignment without any pair, in detail

  • With no pairs, the 109M good-view JEPA reaches Flickr R@1 .164 ± .104, while every encoder that is not a good-view JEPA reaches at most .003.With 10 known pairs, the same method reaches R@1 .32, against about .01 for Procrustes on those 10 pairs and .067 for CLIP’s text encoder.Source: C Additional results · p. 30

Text-only quality

  • The 50M-image good-view JEPA scores .632 on STS-B, above BERT-base (.551) and Qwen2.5-0.5B (.352), but below all-mpnet-base-v2 (.834).On caption-to-caption retrieval, the good-view JEPA goes beyond TF-IDF, while the bad-view JEPA falls below it.Source: Table 5 · p. 24, C Additional results · p. 31

Scaling to the texts of 940M images

  • Across four good-view seeds trained on texts of 940M images, WIT is .168 with DINOv2 ViT-B, above every language model; CLIP ViT-B, -L, and -H scores are .198, .202, and .212, level with the best language model.These runs vary unique texts, samples, and batch size together, so their effects are not separated; WIT is sensitive to the SIGReg weight.Source: C Additional results · p. 32

Overlap checks

  • Removing 43 matched WIT captions changes none of nine checked text models by more than .002, and removing 50 matched COCO test photos changes checked scores by at most .001.At least 43 of 1,024 WIT captions and 50 of 5,000 COCO test photos appear among the 940M web images; the latter overlap includes their web texts, not COCO captions.Source: C Additional results · p. 32

Scaling with training texts

  • WIT alignment jumps between the texts of 200M and 940M images and rises again by 850M samples, while Flickr retrieval rises more slowly.These trends are reported for checkpoints of the largest good-view JEPAs.Source: Table 14 · p. 33

WIT, PRH protocol

  • Flickr-trained text encoders score .029–.037 on Wikipedia captions, while the same recipe trained on texts of 2.16M Recap-DataComp images reaches .099–.114, level with Qwen2.5-0.5B.The 145k-caption Flickr training does not transfer across every objective and DINOv2 size; with 2.16M-image training, the last layer is best.Source: C Additional results · p. 33
  • On WIT-1024, the 109M good-view JEPA surpasses every evaluated language model across the vision encoders it was scored with.The comparison follows Huh et al.’s PRH protocol; the 109M model’s three-seed mean exceeds every language-model result, while removing 43 overlapping captions changes checked models by at most .002.Source: C Additional results · p. 32, Table 15 · p. 33

A second alignment metric

  • With DINOv2 ViT-B, the 109M good-view JEPA scores .507 on linear CKA, versus .390 for all-mpnet-base-v2, and the two alignment metrics rank 16 text models alike (Spearman .88).The good-view JEPA also leads bad-view and next-token JEPAs on the texts of 50M images (.385, .192, and .261, respectively); at 2.16M images, scores are .301, .185, and .225.Source: C Additional results · p. 33

Other vision families

  • With each of six vision encoders, good-view text JEPAs beat same-text next-token and bad-view JEPAs, and the 109M model beats every evaluated language model.The encoders span pixel reconstruction, latent prediction, crops, and image–text contrast; the authors draw no conclusion about the vision objective from MAE and I-JEPA’s similar alignment.Source: C Additional results · p. 34, Table 18 · p. 35

A second benchmark

  • On 5,000 COCO test pairs, the 109M good-view JEPA leads every text encoder with both DINOv2 sizes and both metrics.It also leads next-token and bad-view JEPAs trained on the same texts; removing 50 overlapping photos changes every checked text encoder by at most .001.Source: C Additional results · p. 34, Table 17 · p. 35

Related papers

Loading 2610.39302v1…