PaperlayerView exact revision on arXiv

Source-linked AI summary

Improving Signals, Declining Encoder: A Known-Answer Test of Label-Free Monitoring in Self-Supervised Learning

Nilotpal Kakati

arXiv:2610.35766v1cs.LGcs.CVphysics.data-an

TL;DR

Loss and label-free feature statistics judge self-supervised training, but do not measure whether an encoder keeps what it has learned, a question that cannot be checked checkpoint by checkpoint on natural data with unknown desired representations. The authors train small ViT encoders with a reduced two-view LeJEPA recipe on synthetic images with known factors, then track factor readability and eleven label-free signals across checkpoints. In nine of ten default runs, readability rose then fell sharply while training loss kept its downward trend and RankMe rose.

  • Problem

    Loss and label-free feature statistics judge self-supervised training, but do not measure whether an encoder keeps what it has learned, a question that cannot be checked checkpoint by checkpoint on natural data with unknown desired representations.

    Source: §1 · p. 1

  • Method

    The authors train small ViT encoders with a reduced two-view LeJEPA recipe on synthetic images with known factors, then track factor readability and eleven label-free signals across checkpoints.

    Source: §1 · p. 1, §2 · p. 2, §3.1 · p. 3

  • Results

    In nine of ten default runs, readability rose then fell sharply while training loss kept its downward trend and RankMe rose.

    Source: §3.1 · p. 3

  • Takeaways & Limitations

    In this LeJEPA case study, a signal that keeps improving does not show that an encoder keeps what it has learned.

    Source: §9 · p. 7

    The evidence is limited to one image generator, small encoders, a reduced two-view LeJEPA recipe, and MAE as the only other objective; it does not establish how common the decline is on natural images or at scale.

    Source: §8 · p. 7

Abstract

from arXiv · show

Self-supervised training is monitored without labels, through the loss and statistics of the learned features. We report a case where we know the encoders lost part of what they had learned, so we can check whether these signals noticed. We trained LeJEPA encoders on synthetic images with known generating factors. In nine of ten seeds, the readability of those factors fell abruptly and stayed down, nearly always along the same direction in factor space. Meanwhile the loss kept falling, and most of the eleven label-free signals we tracked kept improving and rated the declining runs better than runs that did not decline. Standard change-point detection rarely flagged the decline near its onset, and selecting checkpoints by these signals mostly chose ones after it.

1 Introduction

Self-supervised training is judged using loss and label-free feature statistics, but these do not measure whether an encoder retains what it learned. Using synthetic images with known factors, the paper finds that readability can fall while those signals suggest progress.

1 Introduction

  • Existing feature statistics and loss do not measure whether an encoder keeps what it has already learned, and that cannot be checked on natural data with unknown desired representations.Source: §1 · p. 1
  • Nine of ten default LeJEPA runs abruptly lost readability of learned factors, while loss kept falling and RankMe rose.The decline recurred across seeds and held under nonlinear probes and a separately generated test split; MAE did not decline on the same images.Source: §1 · p. 1
  • Most of eleven label-free signals kept improving through the declines and rated declining runs better than runs without declines.Even the best signal and change-point pairing, selected after the fact, alarmed near onset in only about a third of declines; checkpoint selection mostly chose post-decline checkpoints.Source: §1 · p. 1
  • Figure 1 contrasts a pair of renders sharing shape factors and energy with the nine default runs aligned at their readability-defined decline onsets.The plot shows readability staying below its pre-onset level even as loss trends downward and RankMe rises.Source: Figure 1 · p. 2

2 Experimental Setup and Measurements

The experiments use a synthetic image generator with known shape and energy factors, enabling checkpoint-wise readability measurements. Small ViT encoders are trained with a reduced two-view LeJEPA recipe, alongside MAE and architecture comparisons.

Images and views

  • The generator creates four-channel 64×64 images from two shape coordinates and energy, with two views rendered independently using the same factors but different sampled elements.The number of sampled elements controls graininess, and parameter µ controls the shape-coordinate distribution.Source: §2 · p. 2

Training configuration

  • The main model is a ViT-Nano encoder with a projector trained using LeJEPA’s agreement and SIGReg objective on two whole-image views.The default projector has hidden width 384; a width-2048 projector is the second main configuration, and MAE and ViT-Tiny provide objective and architecture comparisons.Source: §2 · p. 2

Measuring readability

  • Readability is the average held-out R2 of ridge readouts for z1, z2, and log energy from frozen, layer-normalized, position-averaged final-layer tokens.Readouts are fitted and scored on separate generated images matched to training in µ and N; the images are single renders, not pairs.Source: §2 · p. 2
  • Effective rank is measured with RankMe as the exponential entropy of normalized singular values of the same features.Source: §2 · p. 3

Decline and onset

  • A run counts as declining when its largest readability peak from epoch 20 onward exceeds its final value by at least 0.07.Onset is the first post-peak checkpoint with a fifth of that drop; the rule was fixed after inspecting most run trajectories.Source: §2 · p. 3

3 Readability Declines While Label-Free Signals Improve

In this synthetic-image LeJEPA study, readability fell in most default runs while loss and most label-free signals continued in their preferred directions. Change-point alarms were often late or absent, and signal-based checkpoint choices generally did not avoid the decline.

3.1 The Decline and the Signals Around It

  • Nine of ten default runs showed a sharp readability decline, with all three factors becoming less readable and nearly every decline losing a similar factor combination.Some runs partly recovered, but none returned to its peak; loss kept trending downward and RankMe rose.Source: §3.1 · p. 3, Figure 2 · p. 3
  • The eleven tracked signals include loss, SIGReg, feature standard deviation, LiDAR, and seven metrics summarizing feature spectra, relations across images, or intrinsic dimension.Figure 3 compares decline runs aligned at onset with non-decline runs aligned near the median onset.Source: §3.1 · p. 3, Figure 3 · p. 4
  • After onset, eight of eleven signals rated decline runs better than non-decline runs at every post-onset checkpoint; α-ReQ rated them worse, while loss and κ were mixed.The comparison aligns declines at onset and non-declines at epoch 100; the groups differed in their mix of µ settings, and changing the non-decline alignment mostly left verdicts unchanged except for SIGReg.Source: §3.1 · pp. 3–4, Figure 3 · p. 4

3.2 Alarms and Checkpoint Choice

  • Only 9 of 26 scored declines were alarmed within ten epochs of onset by the best signal–method combinations chosen after the fact, and some alarms also occurred without declines.The audit used three standard change-point methods with thresholds set on control runs from another shape distribution.Source: §3.2 · p. 4
  • Checkpoint selection was an offline comparison: each signal chose its best saved checkpoint in its preferred direction, with cost measured against the hindsight-best candidate.It was not a stopping rule.Source: §3.2 · p. 4
  • Loss minimization and RankMe selected checkpoints at or after onset in every decline run, while LiDAR did so in all but one.Each incurred a median readability cost of 0.21, equal to taking the final checkpoint; α-ReQ selected early, while the other five metrics usually selected post-onset checkpoints at similar cost.Source: §3.2 · p. 4
  • Apart from α-ReQ, selecting checkpoints by the signals did about as well as simply taking the final checkpoint.Source: §3.2 · p. 5

4 Checking and Understanding the Decline

The checks show that the readability decline persists across readouts and nearly always removes the same factor combination. Training changes alter its occurrence, with partial shape refresh eliminating it in the tested runs.

4.1 Readout and Objective Checks

  • Across 13 decline runs, the drop persisted across four readouts and changed little on a separate test split; RankMe rose in all 13 and in 11 after mean subtraction, while all ten masked-autoencoder runs ended at their peaks.The nonlinear probes did not beat pooled ridge in most comparisons, and projector-output RankMe stayed near its maximum at both checkpoints.Source: §4.1 · p. 5

4.2 A Shared Direction

  • Nearly every decline removed readability along almost the same factor-space direction; that direction mixed all three factors, while other combinations remained readable.The exception declined twice; splitting its trajectory places its first decline along the common direction and its second nearly perpendicular to it.Source: §4.2 · p. 5

4.3 Changes That Altered the Decline

  • Changes to the optimizer, objective, and encoder did not remove the decline; three of ten runs declined with the recommended SIGReg weight, and one of two ViT-Tiny runs declined.The decline also occurred without weight decay and with a constant learning rate, but three seeds per change were insufficient to determine whether either affected its frequency.Source: §4.3 · pp. 5–6
  • No partially shape-refreshed run declined, versus nine of ten default runs on the same images and probes; refreshed runs also stayed near their peaks through 1000 epochs.The refreshed views had partly correlated shape coordinates while energy remained shared.Source: §4.3 · p. 5, Figure 4 · p. 6
  • Declines were common at µ = 0.4 and 0.7, uncommon within 300 epochs at µ = 0.9, and absent at µ = 0.97 even after 1000 epochs.Every µ = 0.9 run declined by 1000 epochs; some µ = 0.97 runs fell mid-run and recovered.Source: §4.3 · p. 5, Figure 4 · p. 6
  • No run declined with less grainy images, while grainier images split between declines and poor learning from the start.With the cleanest images, runs did not peak far enough above an untrained encoder’s readability to assess the decline.Source: §4.3 · p. 5, Figure 4 · p. 6

5 LiDAR Flags the Decline When Its Groups Redraw Shape

LiDAR flagged most declines on time when its evaluation groups redrew shape, but rarely did with training-view groups. This altered grouping is not standard LiDAR and requires knowledge of the image factors.

5 LiDAR Flags the Decline When Its Groups Redraw Shape

  • With shape redrawn within each group, LiDAR flagged most declines on time with few false alarms; with training-view groups, it rarely did.Redrawing shape makes energy the main difference between groups, so the largest spectral direction follows energy and weakens as energy readability falls, causing LiDAR to rise.Source: §5 · p. 6
  • The redrawn-shape augmentation is not standard LiDAR, requires separating shape from energy, and makes LiDAR prefer checkpoints after the decline when used as a quality score.Because LiDAR rises during the decline with these groups, it reads that decline as an improvement.Source: §5 · p. 6

6 Related Work

Prior work compared label-free metrics across trained models or checkpoints; this paper instead tests whether they notice within-run losses. Related results on factor learning and embedding dynamics address different questions or describe a similar directional pattern.

6 Related Work

  • Prior comparisons assess metrics across trained models or checkpoints, whereas this study asks whether signals notice a single run losing learned information; its result does not contradict those comparisons.The cited work includes RankMe and LiDAR [Garrido et al., 2023; Thilak et al., 2024a], LeJEPA loss [Balestriero & LeCun, 2025], and broad metric comparisons [Arputharaj et al., 2026].Source: §6 · p. 6
  • The LeJEPA optimum theorem concerns Gaussian factor recovery with nonidentical view factors, unlike this paper’s identical-factor views, and does not determine training-path declines.The original LeJEPA guarantee concerns embedding distributions, not which factors embeddings retain [Balestriero & LeCun, 2025].Source: §6 · p. 7
  • Simon et al. (2023) report stepwise embedding learning; here, a similar single-direction pattern removes factor readability instead of adding it.Source: §6 · p. 7

7 Discussion

Effective rank tracks how evenly features spread across directions, not which factors they encode, so it can rise as readability falls. The discussion considers feature suppression and optimizer instability as possible explanations, while noting unresolved evidence.

7 Discussion

  • RankMe rose even after mean subtraction in most runs, so the falling share of the feature mean explains only part of the effective-rank increase.The feature mean was the largest spectral direction, and its share fell as the spectrum became more even.Source: §7 · p. 7
  • Feature suppression could remove an already learned feature, and observations suggest the common direction was weak from the start; neither observation explains its energy component.Refreshing shape between views removed the decline, although energy remained shared; the hardest directions in the readout checks placed little weight on energy.Source: §7 · p. 7
  • Optimizer instability remains possible: the checks establish only that weight decay and a changing learning rate are not necessary for the decline.Source: §7 · p. 7

8 Limitations

The evidence is limited to one synthetic image generator, small encoders, and a reduced two-view LeJEPA recipe; it does not establish how often the decline occurs in natural-image training or at scale.

8 Limitations

  • The study uses one image generator, small encoders, and a reduced two-view LeJEPA recipe; natural images and the published eight-view recipe at scale remain untested.The authors also note that only MAE besides LeJEPA learned on this generator, and contrastive training’s behavior is unknown.Source: §8 · p. 7
  • Readability probes show that factors became harder to read, not that their information was lost.Source: §8 · p. 7
  • Only three standard change-point methods were tested, with thresholds set on control runs from another shape distribution.Rare alarms therefore do not establish that these signals cannot detect the decline.Source: §8 · p. 7

9 Conclusion

In this LeJEPA case study, encoders lost learned factor readability even as training loss and most label-free signals improved. The signals often selected post-decline checkpoints, and change-point detection rarely caught declines near onset.

9 Conclusion

  • Encoders lost learned readability while training loss and most label-free signals kept improving.Source: §9 · p. 7
  • At best, change-point detection on the signals caught one-third of declines in time, while checkpoints chosen by loss or RankMe came at or after onset.Those checkpoints had worse readability than earlier ones.Source: §9 · p. 7
  • Improving label-free signals do not show that an encoder retains what it has learned.Source: §9 · p. 7

LLM Usage

The paper reports LLM assistance under author direction and review. It describes the synthetic generator and the training and probe procedures used in the study.

LLM Usage

  • LLMs assisted with coding, experiments, analysis, figures, and manuscript drafting and editing; study design, result verification, and scientific interpretation remained with the author.The author takes responsibility for the work.Source: LLM Usage · p. 8

A The Generator

  • The generator produces four-channel 64×64 calorimeter images from shape coordinates and energy, with two training views sharing the intensity profile but differing in sampled deposits.Energy scales intensity and also shifts typical peak position and width; Figure 5 compares renders at N = 30, 100, 300, and 1000.Source: A The Generator · p. 11, Figure 5 · p. 11

B Training and Probe Details

  • Training and probe details specify the optimizer and schedules, onset and signal calculations, reduced LeJEPA settings, and readout and alternative-model procedures.The reduced recipe uses two views, λ = 0.02, 256 SIGReg projections, and integration range [−3, 3], rather than the cited recommended settings.Source: B Training and Probe Details · pp. 11–12

C Readout and Direction Checks

Readout checks confirm the decline on an independent test split and show that it concentrates along a consistent factor direction even as rank measures rise. Occurrence comparisons qualify how checkpoints and training settings should be interpreted.

C Readout and Direction Checks

  • 2,000-image independent rescoring changed pooled ridge and both learned probes by less than 0.02 mean shape R2; both probes still showed the decline.Each learned probe scored at or below pooled ridge at the final checkpoint in 12 of 13 runs, for shape and log energy separately.Source: C Readout and Direction Checks · p. 12
  • RankMe rose in all 13 runs, including 11 of 13 after mean subtraction; projector RankMe medians exceeded 15.9 of 16 at both checkpoints.The squared-singular-value form and participation ratio each rose in 11 of 13 runs.Source: C Readout and Direction Checks · pp. 12–13
  • In 55 of 56 declines, the lost direction nearly matched the reference (minimum absolute cosine 0.996); the first two factor correlations stayed above 0.98 while the third fell.Directional loss compares pre-onset-to-final R2 drops with 101 windows from 43 nondeclining runs; windows overlap and share the reference direction.Source: C Readout and Direction Checks · p. 13
  • The reference direction ranked in the bottom SNR tenth (lowest-direction cosines 0.86 untrained, 0.91 pre-onset); the hardest R2 direction contrasted z1 and z2, unlike the reference's log E weight of 0.567.The hardest direction assigned log E an absolute weight of at most 0.17.Source: C Readout and Direction Checks · p. 13
  • In 21 of 56 declines, the pre-onset checkpoint was before the peak while readability was still rising; in the other 35, it was at most 16% into the decline.The pre-onset rule was fixed after inspecting most run trajectories.Source: C Readout and Direction Checks · p. 13

D Occurrence and Comparison Details

  • Partial shape refresh makes the two views' shape coordinates partly correlated while energy stays shared; the 1000-epoch refresh runs lack a matched shared-view comparison.The condition table covers otherwise-unshown configurations at N = 100; six wide-projector runs trained for 150 epochs are omitted from Figure 4.Source: D Occurrence and Comparison Details · pp. 13–14, Table 1 · p. 13
  • At N = 300, the median peak was 0.30 above untrained readability, whereas at N = 1000 it was only 0.16 above; N = 30 and N = 100 decline drops overlapped.At N = 30, three declines dropped by 0.22–0.26; the nine N = 100 declines dropped by 0.14–0.28.Source: D Occurrence and Comparison Details · p. 14
  • The oracle readout uses 24 physics-summary features per layer, including raw intensity and intensity-weighted coordinate statistics, with the encoder readout's ridge procedure.The summaries are computed after undoing the input log transform.Source: D Occurrence and Comparison Details · p. 14

E Feature Metrics and Checkpoint Selection

Across the alarm audit, most signals did not reliably flag declines near onset, and checkpoint selection by loss or feature metrics often favored post-decline checkpoints. LiDAR’s alarm performance depended strongly on how its input groups were formed.

E Feature Metrics and Checkpoint Selection

  • Loss minimization’s median readability cost was 0.15 for z1, 0.29 for z2, and 0.20 for log E in decline runs.In the 28 decline runs, readability had a median peak of 0.97 and final value of 0.77.Source: E Feature Metrics and Checkpoint… · p. 14
  • At aligned onsets, nine signals rated decline runs better after onset; before onset, rankings depended on alignment for every series.After onset, all signals except κ and α-ReQ rated decline runs better; the loss did so at all but one checkpoint, and the other eight at every checkpoint.Source: E Feature Metrics and Checkpoint… · p. 15

F Alarm Audit

  • The alarm audit applied CUSUM, EWMA, and Page–Hinkley with thresholds set from controls, scoring a matched set of 26 decline runs and 13 non-decline runs.Thresholds were 1.1 times the largest control statistic, and an alarm was timely within ten epochs of onset; neither threshold procedure had a nominal false-alarm guarantee.Source: F Alarm Audit · pp. 15–16
  • The best timely alarm counts reached 9/26 for TwoNN and uncentred α-ReQ, but those methods also alarmed in 6–7/13 and 5/13 non-decline runs, respectively.SIGReg also reached 9/26 timely alarms with EWMA under the initial procedure and alarmed in 2/13 non-decline runs.Source: F Alarm Audit · p. 15

G LiDAR Groups and Checks

  • With redrawn-shape groups, LiDAR’s leading eigenvalue fell in all 28 declines, and its share shift accounted for a median 98% of the log LiDAR change.Projections onto the leading direction correlated with the ridge log-energy prediction at median absolute 0.98.Source: G LiDAR Groups and Checks · p. 17
  • With redrawn-shape groups, LiDAR CUSUM and EWMA flagged 23–25 of 26 declines in time at each tested δ, with at most one false alarm among 13 non-declines.The contrast with training-view groups held across a second render set and runs using their own training distributions.Source: G LiDAR Groups and Checks · p. 17
  • LiDAR used as a quality score with redrawn-shape groups chose a checkpoint at or after onset in all 28 decline runs, at about the same median readability cost as loss.Source: G LiDAR Groups and Checks · p. 17

Related papers

Loading 2610.35766v1…