Which Tasks Survive Self-Supervised Learning?

1 INTRODUCTION

• The paper asks which downstream tasks survive same-instance SSL and what geometry and few-shot behavior follow from their recoverability.Source: §1 · p. 1
• Captured posterior energy measures how much of a task's posterior score remains in the represented function space, independently of the learning objective.Source: §1 · p. 2
• For centered, whitened representations, recoverability determines directional CDNV, probe–centroid alignment, few-shot nearest-centroid guarantees, and joint centroid geometry; a canonical two-view objective selects leading cross-view-stable modes.The task posterior's overlap with the selected subspace characterizes its semantic recoverability.Source: §1 · p. 2, Figure 1 · p. 3, §1.1 · p. 3

1.1 RELATED WORK

• The paper links task-relevant representation geometry to the population subspaces selected by two-view objectives, making its focus task-conditional rather than global.Prior work examines directional geometry and spectral views of SSL; this paper connects those lines to ask which downstream posterior scores survive.Source: §1.1 · p. 2

2.2 DOWNSTREAM FEW-SHOT LEARNING

• Downstream quality is evaluated by few-shot classification from independently sampled labeled examples per class, focusing on nearest-class-centroid classification.This operationalizes task survival as learnability from a small labeled support set.Source: §2 · p. 3, §2.2 · p. 3

2.1 LATENT INSTANCE MODEL AND TWO-VIEW GENERATION

• A latent instance determines a balanced binary semantic label, while potentially containing information beyond that label.Source: §2.1 · p. 3
• Two views form a positive pair by being drawn independently conditional on the sample associated with the same latent instance.The paper gives random image augmentations such as cropping, color jitter, and blur as examples.Source: §2.1 · p. 3

3.1 CAPTURED POSTERIOR ENERGY

• Captured posterior energy B(F) is the squared norm of the task posterior score projected onto the function space represented by F.It is task-dependent and does not require SSL or whitening.Source: §3.1 · pp. 3–4

3.2.1 EXACT DIRECTIONAL-COLLAPSE LAW

• For centered, whitened representations, directional CDNV equals (1 − B(F))/(2B(F)), vanishing as recoverability approaches one and diverging as it approaches zero.Global CDNV can remain large even when directional CDNV vanishes, so global collapse is not necessary for task geometry to simplify.Source: §3.2.1 · p. 4

3.2.2 PROBE–CENTROID ALIGNMENT

• The population MSE probe and nearest-centroid classifier share a separating direction, and the probe's squared norm equals B(F).Source: §3.2.2 · p. 4

3.2.3 FEW-SHOT TRANSFER GUARANTEES

• For centered, whitened features, the m-shot nearest-centroid error bound is controlled by B(F), separating uncaptured task information from finite-sample centroid estimation.Without whitening, the coordinate-invariant analogue uses rewhitening or a Mahalanobis metric.Source: §3.2.3 · pp. 4–5

3.2.4 MULTITASK SEMANTIC GEOMETRY

• Well-recovered tasks have strong semantic axes that become nearly orthogonal when task correlations and residual uncertainty are small.Their joint centroids approach an axis-aligned hyperrectangle whose side lengths depend on task-specific capture values.Source: §3.2.4 · p. 5

4.1 A CANONICAL TWO-VIEW POPULATION OBJECTIVE

• Whitening-based W-MSE rewards agreement between two views while fixing the feature metric, making it an analyzable rule for selecting an r-dimensional function space.Strict whitening is the clean setting for identifying the selected subspace and deriving exact geometry; related covariance-regularized and prediction-based views are treated separately.Source: §4.1 · pp. 5–6

4.2 THE TWO-VIEW OPERATOR AND THE SELECTED SUBSPACE

• The two-view conditional expectation operator is self-adjoint and positive semidefinite, with eigenvalues measuring how stable corresponding modes are across views.When λr > 0 and λr > λr+1, the population optimum spans the leading r eigenfunctions and achieves the sum of their eigenvalues.Source: §4.2 · p. 6

4.3 CLOSED-FORM RECOVERABILITY AT THE SSL OPTIMUM

• At the population optimum, task recoverability is the posterior energy projected onto the leading r two-view eigenfunctions.The two-view objective selects these modes without downstream labels; labels enter when evaluating a task's posterior overlap with the selected subspace.Source: §4.3 · p. 6, §4.2 · p. 6
• Greater spectral overlap predicts lower directional CDNV, stronger few-shot nearest-centroid classification, and more pronounced semantic axes.Across tasks, recoverability determines approximate hyperrectangle side lengths; when augmentations preserve a task, its posterior can be increasingly captured as representation dimension grows.Source: §4.3 · p. 7

5.1 SETTINGS

• Experiments use CelebA's 40 balanced binary attributes and several trained or pretrained encoders, retaining leading paired-view spectral directions to vary representation rank.Captured posterior energy is estimated from label-feature covariance and feature covariance, accounting for deviations from exact whitening.Source: §5.1 · p. 7, (a) W-MSE-CelebA (b) I-JEPA-IM1K · p. 8

5.2 RESULTS

• The most recoverable CelebA attributes approach B(t)r ≈1 at moderate ranks, while the mean across all 40 attributes remains lower and SSL exceeds random initialization across a broad rank range.Attributes also differ markedly in saturation rate and attainable recoverability.Source: §5.2 · p. 8
• Predicted and observed directional CDNV agree closely across ranks and SSL representations, spanning more than two orders of magnitude.Recoverability and directional CDNV are estimated on separate data splits.Source: §5.2 · pp. 8–9
• Across CelebA attributes, empirical few-shot NCC error decreases with task recoverability, and the bound tightens as support size m grows.The bound remains above observed error throughout evaluated configurations and is more informative for well-recovered tasks.Source: Figure 5 · p. 8, §5.2 · p. 9
• Across encoders, higher mean task recoverability is associated with lower normalized centroid RMSE and smaller task-axis overlap.Illustrative held-out joint centroids are compared with the predicted factorial hyperrectangle; quantitative panels include eligible triplets without recoverability or orthogonality screening.Source: Figure 6 · p. 8, §5.2 · p. 9
• For ImageNet-pretrained encoders, spectral-subspace recoverability reconstructions have median absolute discrepancies below 0.05 across models, while CelebA-trained W-MSE and VICReg show near-exact agreement.The spectral subspace is estimated without downstream labels; for I-JEPA, the paired-view operator is a surrogate analysis kernel rather than its pretraining masking distribution.Source: §5.2 · p. 9, Figure 7 · p. 9

• Semantic recoverability unifies accounts of directional neural collapse, probe–centroid alignment, multitask centroid structure, and few-shot performance.Source: §6 · p. 9
• The main results are derived primarily for population-optimal, centered and whitened representations.The work also analyzes a covariance-regularized surrogate.Source: §6 · p. 9
• Extending the theory to finite-sample training and imperfect optimization remains future work.Source: §6 · p. 9
• The main analysis focuses on balanced binary tasks and conditionally i.i.d. two-view generation; multiclass tasks and more general view dependencies are future directions.Source: §6 · p. 9

• ChatGPT and Claude Code assisted with theoretical frameworks, mathematical claims, proof strategies, and feedback on experimental methodology.Source: AI USE STATEMENT · p. 10
• The tools also assisted with experimental design and implementation, research brainstorming, literature exploration, and manuscript drafting and editing.Source: AI USE STATEMENT · p. 10
• The authors reviewed AI-assisted arguments, derivations, designs, and code, and take full responsibility for the paper's final content.Source: AI USE STATEMENT · p. 10

• The study concerns theoretical and empirical properties of representations learned through self-supervised learning.Source: ETHICS STATEMENT · p. 10
• Experiments use existing datasets and pretrained models for research purposes, without collecting new personal data or conducting new human-subject studies.Source: ETHICS STATEMENT · p. 10
• The work does not develop or evaluate a high-risk application.Source: ETHICS STATEMENT · p. 10

• The main text states the assumptions and formal results, and the appendix provides complete proofs.Source: REPRODUCIBILITY STATEMENT · p. 10
• The experimental sections and supplementary material document datasets, pretrained representations, evaluation protocols, preprocessing, training procedures, and implementation details.Source: REPRODUCIBILITY STATEMENT · p. 10
• The paper specifies procedures for estimating semantic recoverability, directional CDNV, spectral structure, and few-shot transfer performance.Source: REPRODUCIBILITY STATEMENT · p. 10

Datasets and evaluation protocol

• CelebA uses an identity-disjoint train/validation/test split; dSprites and 3DShapes use specified factor subsets and held-out test splits.Recoverability and directional-CDNV analyses use disjoint training subsets A and B, with spectral estimates computed on A and directional CDNV measured on B.Source: A IMPLEMENTATION DETAILS · p. 17

Evaluation preprocessing

• Label-dependent measurements use clean, single-view images, while augmented CelebA pairs are used to estimate the paired-view spectral basis.Synthetic images are kept at native resolution and upsampled for pretrained encoders when needed; inputs use ImageNet-1K channel statistics.Source: A IMPLEMENTATION DETAILS · p. 17

VICReg and W-MSE augmentations

• CelebA-trained VICReg and W-MSE share random-resized-crop, flip, color-jitter, grayscale, and blur augmentations, with a minimum crop scale of 0.2.The pipeline omits solarization and uses the same blur probability for both views.Source: A IMPLEMENTATION DETAILS · p. 17

Architectures and training

• CelebA-trained VICReg and W-MSE use ResNet-50 backbones and train from scratch for 1000 epochs; analyses generally use pooled backbone features.VICReg and W-MSE use specified projection heads and AdamW training; W-MSE independently whitens view sub-batches.Source: A IMPLEMENTATION DETAILS · pp. 17–18

Pretrained encoders

• The study also evaluates public ImageNet-1K-pretrained VICReg, Barlow Twins, and I-JEPA encoders without fine-tuning.The ResNet-50 models yield 2048-dimensional pooled features; I-JEPA uses averaged final-layer-normalized patch tokens for a 1280-dimensional representation.Source: A IMPLEMENTATION DETAILS · pp. 17–18

B.1 ADDITIONAL CELEBA RESULTS

• On CelebA, recoverability varies across attributes, held-out directional CDNV closely matches its prediction, and NCC error decreases with recoverability across models, ranks, and shot counts.The additional spectral plots compare direct recoverability with its reconstruction from the paired-view spectral subspace.Source: §B.1 · pp. 18–19, Figure 9 · p. 19
• Our W-MSE bound is non-vacuous in several rank–shot settings and falls below 0.5 at m = 1, while earlier bounds remain above 0.5 across the reported attributes and shot counts.The minimizing rank increases with support size, matching the bound’s rank–shot tradeoff.Source: §B.1 · pp. 19–20, Table 1 · p. 20

B.2 RESULTS ON SYNTHETIC DATASETS

• Across dSprites and 3DShapes, the recoverability–geometry relationships qualitatively match CelebA for the evaluated pretrained encoders.The figures cover rank-dependent recoverability, predicted versus observed directional CDNV, and held-out multitask centroid geometry.Source: §B.2 · p. 20, Figure 11 · p. 20, Figure 12 · p. 21

C COVARIANCE-REGULARIZED SSL OBJECTIVES

• The covariance-regularized surrogate preserves the two-view operator’s selected spectral subspace, but its geometry depends on three quantities; strict-whitening formulas return as γ →∞.The setup uses a covariance penalty controlled by γ and centered features.Source: C COVARIANCE-REGULARIZED SSL OBJECTIVES · p. 21, §C.1 · p. 21, §C.2 · p. 22

C.3 STRUCTURE OF THE POPULATION OPTIMUM

• With a positive eigengap at rank r, the regularized optimum can be population-rewhitened, recovering the strict-whitening optimum and its multitask geometry.Without rewhitening, covariance regularization can distort Euclidean geometry; the corresponding nearest-centroid rule uses a Mahalanobis metric.Source: §C.3 · pp. 22–23

C.4 HOW THE CDNV FORMULAS CHANGE

• The covariance penalty changes few-shot behavior through Bγ, Cγ, and Sγ: lowering γ amplifies selected modes, increasing both total variance and task-direction signal.The resulting behavior depends on how posterior coefficients overlap with the modes amplified by the regularizer.Source: §C.4 · p. 24

D PREDICTION-BASED SSL OBJECTIVES AS REDUCED-RANK REGRESSION

• A stylized whitened prediction objective reduces to a spectral objective and selects the top-r eigenspace of the two-view operator under the stated eigengap condition.Its cross-view feature-correlation norm is maximized by the span of the leading operator modes.Source: D PREDICTION-BASED SSL OBJECTIVES AS… · p. 24, §D.1 · p. 24, §D.2 · p. 25

D.3 CONSEQUENCES FOR RECOVERABILITY, CDNV, AND DOWNSTREAM GUARANTEES

• Because the prediction surrogate selects the same subspace as the whitened population objective, its recoverability, CDNV, few-shot, and multitask-geometry conclusions carry over.The appendix presents this as a population-level connection and does not model algorithmic details such as stop-gradient or EMA teachers.Source: D PREDICTION-BASED SSL OBJECTIVES AS… · p. 24, §D.3 · p. 25

E.1 PROOFS OF THE RESULTS IN THE MAIN TEXT

• For centered, whitened features and balanced binary labels, overall and directional CDNV are exactly (r - B(F))/(2B(F)) and (1 - B(F))/(2B(F)), respectively.Both ratios are +∞ when B(F) = 0.Source: §E.1 · p. 26
• The unique population MSE probe has zero intercept and weight equal to half the class-mean gap, so its axis aligns with the centroid axis.Balancedness and centering give w* = E[YF(X)] = Δ/2.Source: §E.1 · p. 26
• The few-shot NCC error is bounded by 1 - B(F) + (r - B(F))/m + (1 - B(F))/(1 - B(F) + 2mB(F)).The bound applies for m-shot support sets and extends by continuity when B(F) = 0.Source: §E.1 · pp. 26–27
• Task-relevant directions are projections of posterior functions into the feature space, and their overlap is bounded using task-label correlation and conditional noise without assuming independent labels.The theorem treats balanced tasks with positive recoverability.Source: §E.1 · pp. 27–28
• With an eigengap at rank r, the SSL objective’s optimal feature coordinates span the leading r eigenfunctions of the two-view operator.Those eigenfunctions achieve the sum of the top r eigenvalues, and every optimum has the same coordinate span.Source: §E.1 · p. 28

E.2 PROOFS FOR APPENDIX C

• For the regularized objective, every maximizer uses the leading eigenfunctions with squared scales 1 + λj/γ, up to an orthogonal output transformation.The eigengap fixes the maximizing subspace; rotations within repeated-eigenvalue eigenspaces are absorbed into the output transformation.Source: §E.2 · pp. 28–29