PaperlayerView exact revision on arXiv

Source-linked AI summary

Shared Is Not Enough: Predictor Width and Factor Readability Under Masked Latent Prediction

Nilotpal Kakati

arXiv:2610.32562v1cs.LGcs.CVphysics.data-an

TL;DR

Masked latent prediction is intended to learn predictable features, but it is unclear which input factors remain readable. The authors use a simulator with known factors to compare objectives and predictor sizes, finding that a wider I-JEPA predictor reverses the default comparison with SimCLR and BYOL.

  • Problem

    Downstream performance indicates task usefulness but does not reveal which input factors remain readable in learned representations.

    Source: §1 · p. 1

  • Method

    The authors train I-JEPA, SimCLR, and BYOL on views from a simulator with known factors and probe factor readability in frozen representations.

    Source: §2.2 · p. 3, §2.3 · p. 4

  • Results

    With the default predictor, I-JEPA∗’s median score was about 40% of SimCLR∗’s; matching predictor width to encoder width reversed the ordering against both methods at every seed.

    Source: §1 · pp. 1–2

  • Takeaways & Limitations

    In these experiments, predictor size and training data were associated with differences in factor readability, though the reason for those changes remains unestablished.

    Source: §5 · p. 7

    The experiments cover one factor in one simulator with small encoders, and do not establish why predictor or training-data changes affect readability.

    Source: §5 · p. 7

Abstract

from arXiv · show

Masked latent prediction methods such as I-JEPA aim to learn predictable features, but what determines which information they retain is less clear. We built a simulator in which every factor behind an image is known, so a probe can measure how readable each factor remains after training. We trained small encoders (0.7M parameters) with I-JEPA, SimCLR and BYOL on the same pairs of views and probed a factor that both views share. I-JEPA had a predictor with half the encoder's width and depth, the proportions it uses for ViT-B. Its median score was about 40% of SimCLR's. Widening the predictor to the encoder's width reversed this ordering at every training seed, against both other methods. We release the simulator and the trained models.

1 Introduction

The paper asks whether a factor shared across views remains readable after masked latent prediction, and whether predictor capacity affects that readability. In a controlled simulator, the default predictor leaves the factor less readable than contrastive methods, while matching predictor width to encoder width reverses the comparison.

1 Introduction

  • The simulator keeps image factors separate from sampling noise, allowing a factor to be shared, redrawn between views, or held constant across training images.Views are sampled separately, so they do not share noise.Source: §1 · p. 1, Figure 1 · p. 2
  • The study compares I-JEPA∗ with SimCLR∗ and BYOL∗ on the same views and encoder architecture, probing known factor values from frozen representations.The star marks the authors’ implementations; the default predictor is narrower and shallower than the encoder.Source: §1 · p. 1
  • At the end of training, I-JEPA∗’s median shared-factor score was about 40% of SimCLR∗’s, while widening its predictor reversed the ordering against SimCLR∗ and BYOL∗ at every seed.Holding the factor constant produced similarly low scores; redrawing it between views raised the median.Source: §1 · pp. 1–2
  • The released simulator and trained models support testing other objectives and predictors against known factor values.A probe measures how readable a factor remains after training.Source: §1 · pp. 1–2

2 Setup

The setup uses a simulator with known component factors and independently sampled images to control what the two training views share. It compares objectives and predictor sizes, then probes frozen representations for how readable energy and shape remain.

2.1 Simulator

  • The simulator creates four-channel images from overlapping components with known positions, energies, and shapes; separate random streams control shape and image sampling, and larger N produces smoother images.Source: §2.1 · p. 2

2.2 Training Conditions and Objectives

  • Training pairs show the same components, positions, and energies, while conditions determine whether shape is shared, redrawn, constant, or both views come from one image.Source: §2.2 · p. 3
  • I-JEPA∗, SimCLR∗, and BYOL∗ use the same encoder architecture, data, optimiser, and training budget; I-JEPA∗ predictor depth and width are varied.Source: §2.2 · p. 3

2.3 Readability

  • A probe fitted on separate images predicts energy and energy-independent shape coordinates from frozen encoder tokens, measuring improvement in median absolute error over a median-prediction baseline.Source: §2.3 · p. 4

3 Results

The results compare factor readability under masked latent prediction and ask how predictor capacity, training data, and representation movement relate to it.

3.1 Readability of the Shared Factor

  • Making the predictor deeper raised readability at every seed, but only three of six deeper-predictor seeds reached the wider predictor’s level; the wider-predictor lead also held across probe variants.On the third probe variant, the wider predictor exceeded every BYOL seed and SimCLR at five of six seeds.Source: §3.1 · pp. 4–5
  • Shared I-JEPA reached about 81% of shared SimCLR’s energy score, a smaller gap than the 40% shape score, and widening raised both scores.With the wider predictor, every I-JEPA encoder scored above every SimCLR and BYOL encoder on energy too.Source: §3.1 · p. 4
  • At training end, default I-JEPA’s median shape readability was about 40% of SimCLR’s, while an encoder-width predictor put every I-JEPA seed above SimCLR and BYOL (Figure 3).BYOL scored like SimCLR after collapsed encoders were excluded.Source: §3.1 · p. 4, Figure 3 · p. 5

3.2 What Raises Readability

  • Redrawing shape between views raised default I-JEPA’s median readability, while holding shape constant across training images left scores about as low as in the shared condition.Five of six seeds scored higher in the redrawn condition; SimCLR and BYOL stayed ahead in both shared and constant conditions.Source: §3.2 · pp. 5–6
  • Training on grainy images produced the highest shape-readability score, at the top of the shared SimCLR and BYOL range; mixed images scored in between, while smooth and default training scored similarly.The grainy-first, mixed-second order held across three probe graininesses and in the single-view condition.Source: §3.2 · p. 6

3.3 Movement

  • Movement rose with readability as the predictor grew, but it did not consistently rank readability across objectives or training conditions (Figure 5).Default I-JEPA and SimCLR overlapped in movement despite different readability; constant I-JEPA moved more than shared I-JEPA despite overlapping readability.Source: Figure 5 · p. 6, §3.3 · pp. 6–7

4 Related Work

Prior work studies whether shared factors remain readable and how predictor capacity affects learned representations. The paper relates its findings to those questions while noting differences from the closest JEPA-style comparison.

4 Related Work

  • Prior work includes guarantees for recovering shared content under invertible observation models, controlled evidence that easy shared features can suppress competing features, and direct probes of known factors.The cited works are von Kügelgen et al. (2021), Chen et al. (2021), Sobal et al. (2022), and Pendharkar (2026).Source: §4 · p. 7
  • In the closest cited JEPA-style study, a feature held fixed across a future-latent pair stayed readable; here, default I-JEPA made the shared factor less readable than SimCLR and BYOL, though the tasks differ.The cited study varied feature persistence; this paper notes that the factor’s entry into the image also differs.Source: §4 · p. 7
  • Prior work links predictor capacity to retention of information that changes between views, while theory offers candidate explanations involving cross-view statistics, training dynamics, and trainable prediction heads.The cited capacity result is from Garrido et al. (2024); the candidate explanations cite Littwin et al. (2024), Tian et al. (2021), and Wen & Li (2022).Source: §4 · p. 7

5 Discussion and Conclusion

The discussion interprets the predictor-width result as consistent with a possible preference for easier-to-predict features, while emphasizing the study’s scope and unresolved mechanisms. It also proposes testing predictor capacity on natural-image views.

5 Discussion and Conclusion

  • One hypothesis is that a small predictor favors easier-to-predict encoder features at the expense of a weakly expressed factor; matching predictor and encoder width improved this factor’s readability.The paper contrasts this with I-JEPA ablations that favor a narrower predictor for downstream evaluation and reports that widening raised both shape and energy scores.Source: §5 · p. 7
  • Redrawing shape between views and training on grainier images both raised readability.These are reported as training-data effects; the paper has not established why they occurred.Source: §5 · p. 7
  • The movement measure rose with readability when the predictor grew, but did not consistently rank readability across objectives or training conditions.The reported experiments cover one factor in one simulator with small encoders.Source: §5 · p. 7
  • The probe comparisons cannot determine whether training removed shape information or only made it harder to read, although the trained-encoder comparisons use the same probe.Every trained encoder scored below the untrained median.Source: §5 · p. 7
  • A proposed next test is to vary predictor capacity on a measurable factor shared across natural-image views while fixing the encoder and training schedule.The authors expect the default predictor proportions to yield lower readability than SimCLR and increased capacity to narrow or reverse that gap.Source: §5 · p. 7

LLM Usage

The authors report that large language models assisted with coding, experiments, analysis, figures, and manuscript work under author direction and review.

LLM Usage

  • Large language models assisted with coding, experiment execution, analysis, figure production, and manuscript drafting and editing under the author’s direction and review.Study design, verification of reported results, and scientific interpretation remained with the author, who takes responsibility for the work.Source: LLM Usage · p. 8

A Simulator Details

The simulator builds images from separately generated particle showers, separating each particle’s shape from the random sampling that produces image deposits.

Image construction

  • Each image adds together several particles’ showers, with particle energies summing to total image energy and entry positions spread around a common axis.Table 2 summarizes the geometry and particle-sampling settings.Source: A Simulator Details · p. 10, Table 2 · p. 10

Shape generation

  • Stage 1 draws each particle’s shape parameters α and T from an energy-dependent distribution; T locates the depth-profile maximum and α controls its shape.The shape parameters also set the spread within each layer through a radial kernel depending on shower age.Source: A Simulator Details · p. 10
  • In the constant condition, every component uses α = 4.456 and T = 6.606, independent of its energy.Source: A Simulator Details · p. 10

Image sampling

  • Stage 2 samples points within each layer according to the depth profile and radial kernel, then bins them onto the image grid.More points reduce fluctuations between independent samplings of the same profile.Source: A Simulator Details · p. 10

B.1 Encoder Training

The appendix specifies encoder inputs and training settings, comparison-method implementations, and the probes used to measure factor readability and representation movement.

B.1 Encoder Training

  • The default encoder-training recipe follows the I-JEPA optimisation settings, with data simulated on the fly.The token embeds a 4 × 4 tile across all four image channels; empty cells map to zero and occupied cells to values up to about 1.1.Source: §B.1 · p. 10, Table 3 · p. 11
  • The predictor sweeps vary width over 12, 24, 48, 96, and 192 or depth over 1, 3, and 6, with the shared comparison using 48/3, 48/6, and 96/3.The shared comparison is evaluated at epoch 500.Source: §B.1 · p. 10
  • With default GPU kernels, same-seed training runs diverged within five epochs, whereas deterministic kernels produced identical checkpoints.Source: §B.1 · p. 11

B.2 Comparison Settings

  • SimCLR∗ and BYOL∗ use the same configurations as I-JEPA∗ outside the objective block, with unmasked mean-pooled views and projection heads discarded for probing.SimCLR∗ uses InfoNCE; BYOL∗ uses an exponential-moving-average target encoder and projector.Source: §B.2 · p. 11
  • Narrowing both projectors’ hidden width from 384 to 96 or 48, with BYOL∗’s predictor kept at 384, did not lower either method’s shared-condition median; at every width, each shared I-JEPA∗ encoder with the wider predictor outscored every non-collapsed encoder.BYOL∗ collapsed in two, three, and one of six seeds at widths 384, 96, and 48, respectively.Source: §B.2 · p. 11

B.3 Probe and Movement Measurement

  • The readability probe predicts standardized simulator targets from frozen representations, using separate fitting, selection, and test image sets.The main probe selects a checkpoint by median absolute error and reports how much of the baseline z1 error it removes.Source: §B.3 · pp. 11–12
  • Movement is measured as the ratio of mean cosine distance between same-image replicas to mean distance between different images.Evaluations use 64 images with eight replicas each, and report the median ratio across token positions.Source: §B.3 · p. 12

B.4 Probe Controls

Probe controls and predictor-capacity experiments qualify the readability comparisons, while energy scores move toward the untrained level as predictor capacity grows.

B.4 Probe Controls

  • MAE∗ scored 0.283 and every MAE∗ encoder exceeded the other trained encoders, while target-versus-online encoder copies differed by at most 0.0110 in readability.Single-view I-JEPA∗ scored 0.029; the single-view and two-view constructions differ in both image generation and available context.Source: §B.4 · p. 12
  • Across every probe, default-predictor shared I-JEPA∗ had a lower median than shared SimCLR∗, and both remained below MAE∗.Source: §B.4 · p. 12
  • The untrained readability score depends on probe budget: untrained fits kept improving until 11,700–39,300 steps, while trained medians did not move between 10,000 and 40,000 steps.Source: §B.4 · p. 12
  • On raw image patches, the probe scored 0.322 for z1; fixed per-dimension standardization raised this to 0.396, yet the highest encoder score remained below the lowest untrained initialization.The respective bounds were 0.287 for MAE∗ and 0.328 for the lowest untrained initialization.Source: §B.4 · p. 12
  • On three probes, the lowest wider-predictor score exceeded the highest SimCLR∗ or BYOL∗ score: 0.222 vs 0.193, 0.209 vs 0.157, and 0.148 vs 0.109.These are joint, single-target, and 7.8k-parameter probe scores, respectively; on the standardized probe, the highest SimCLR∗ score was 0.244 and the lowest wider-predictor score was 0.231.Source: Table 4 · p. 13

C.1 Predictor Capacity

  • In single-view mixed-image sweeps, readability rose with predictor depth and from width 24 to encoder width, with no further gain beyond that width.In the shared condition, deeper-predictor seeds split: three reached the wider predictor’s level and three stayed within the SimCLR∗ and BYOL∗ range.Source: §C.1 · p. 13
  • Shared SimCLR∗ was below default-predictor I-JEPA∗ at every seed from epochs 20 to 100, then above it at five of six seeds at epochs 400 and 500.With the default predictor, readability declined after epoch 150; deeper and wider predictors showed no sustained late decline.Source: §C.1 · p. 13

C.2 Energy

  • Shared I-JEPA∗ had median energy score d(E) 0.314 versus 0.389 for SimCLR∗ and 0.494 for the untrained encoder; deeper and wider predictors raised it to 0.413 and 0.457.I-JEPA∗ was also below SimCLR∗ in the constant condition.Source: §C.2 · p. 13

C.3 Training Graininess

Training-image graininess and probe-image graininess show distinct readability patterns, although one training-data comparison changes both image difficulty and graininess.

C.3 Training Graininess

  • The grainiest training images produced the highest trained readability in both shared and single-view conditions, even as the untrained score rose with smoother images.Figure 7 plots readability against probe graininess for four training-graininess choices at 40,000 steps; filled markers indicate training-matched graininess.Source: §C.3 · p. 13, Figure 7 · p. 14
  • With smoother probe images, readability rose for encoders trained on grainy or mixed images but changed little for those trained on default or smooth images.Source: §C.3 · p. 13
  • Dropping the grainiest images from the mixed training set yielded median readability 0.082, but also made training images easier to read, so the comparison cannot isolate which change lowered the score.The altered set began at N = 200 rather than N = 100.Source: §C.3 · p. 14

Related papers

Loading 2610.32562v1…