PaperlayerView exact revision on arXiv

Source-linked AI summary

RankMe: Assessing the downstream performance of pretrained self-supervised representations by their rank

Quentin Garrido, Randall Balestriero, Laurent Najman, Yann Lecun

arXiv:2210.02885v3cs.LGcs.AIcs.CV

TL;DR

JE-SSL lacks reliable label-free ways to assess representations when reconstruction cues and labeled validation data are unavailable. The paper introduces RankMe, an untuned effective-rank criterion based on embedding singular values, and shows that it predicts downstream performance and supports hyperparameter selection with nearly no performance loss relative to label-guided selection. RankMe is therefore a practical tool for evaluating JE-SSL representations in label-scarce settings.

  • Problem

    JE-SSL deployment is difficult because it lacks reconstruction-based visual cues and often requires labels to judge representation quality and tune models.

  • Method

    RankMe assesses JE-SSL representations using the Shannon entropy of normalized embedding singular values, without training or tunable parameters.

  • Results

    RankMe consistently informs downstream performance across methods and architectures and retrieves or sometimes surpasses most performance from label-guided hyperparameter search.

  • Takeaways & Limitations

    RankMe enables label-free model and hyperparameter selection for JE-SSL, including evaluation on different downstream datasets.

  • Takeaways & Limitations

    The empirical study focuses on linear classification, with only promising evidence for nonlinear classification, and excludes finetuning.

Abstract

from arXiv · show

Joint-Embedding Self Supervised Learning (JE-SSL) has seen a rapid development, with the emergence of many method variations but only few principled guidelines that would help practitioners to successfully deploy them. The main reason for that pitfall comes from JE-SSL's core principle of not employing any input reconstruction therefore lacking visual cues of unsuccessful training. Adding non informative loss values to that, it becomes difficult to deploy SSL on a new dataset for which no labels can help to judge the quality of the learned representation. In this study, we develop a simple unsupervised criterion that is indicative of the quality of the learned JE-SSL representations: their effective rank. Albeit simple and computationally friendly, this method -- coined RankMe -- allows one to assess the performance of JE-SSL representations, even on different downstream datasets, without requiring any labels. A further benefit of RankMe is that it does not have any training or hyper-parameters to tune. Through thorough empirical experiments involving hundreds of training episodes, we demonstrate how RankMe can be used for hyperparameter selection with nearly no reduction in final performance compared to the current selection method that involve a dataset's labels. We hope that RankMe will facilitate the deployment of JE-SSL towards domains that do not have the opportunity to rely on labels for representations' quality assessment.

1. Introduction

JE-SSL deployment is difficult without labels or visual reconstruction cues for judging representation quality. RankMe addresses this gap as a label-free, untuned criterion that assesses downstream performance and supports hyperparameter selection.

  • Motivation: JE-SSL hyperparameter and model selection commonly relies on labeled-dataset performance, limiting deployment on new unlabeled datasets.Without labels and visual cues from reconstruction, tuning remains challenging.
  • RankMe: RankMe estimates representation quality from the singular-value distribution of learned embeddings without training, labels, or tunable parameters.Its construction is motivated by the relationship between classifier input rank and training performance.
  • Evidence: RankMe informs downstream performance across JE-SSL methods, architectures, and evaluation settings, including in-distribution and out-of-distribution datasets.The study evaluates methods including VICReg, SimCLR, DINO, and variants.
  • Hyperparameter selection: RankMe retrieves and sometimes surpasses most performance found by manual label-guided hyperparameter search without using labels.The paper reports this capability on both in-domain and out-of-domain datasets.
  • Evaluation: Experiments cover 11 datasets and 110 models across five methods, including linear and nonlinear probing without downstream labels or task samples.These experiments test whether RankMe distinguishes successful from sub-optimal JE-SSL training.

2. Background

JE-SSL research includes contrastive and non-contrastive approaches whose representations can exhibit dimensional collapse. This study formalizes whether representation rank can identify successful training and quantify collapse-related performance differences.

  • JE-SSL families: JE-SSL methods comprise contrastive and non-contrastive families that differ in how they prevent representational collapse.Contrastive methods separate negative samples, whereas non-contrastive methods use other mechanisms.
  • Representation spectrum: Prior work established dimensional collapse as a theoretical and empirical phenomenon in JE-SSL representations.Studies examined whitening, augmentations, partially collapsed encoders, and practical causes of collapse.
  • Figure 1: Figure 1 relates encoder-output classification performance to projector-output RankMe values for in-distribution and out-of-distribution ImageNet evaluations.The figure also compares supervised linear-probe selection with unsupervised RankMe selection.
  • Open problem: Existing studies informally favor representations with less collapse, but had not formally tested whether this identifies successful models or quantified collapse.Formal quantification of collapse for improving representations is the stated goal of this study.

3. RankMe Consistently Predicts Downstream performances From Representations

RankMe estimates representation quality from the embedding singular-value distribution and is evaluated as a label-free predictor of downstream performance. Across methods, datasets, architectures, and evaluation protocols, higher rank generally indicates better performance, but rank alone is not sufficient for comparing different approaches.

  • RankMe: A Simple Method and Its Implementation: RankMe is a smooth rank measure based on the normalized singular values of the source dataset’s embedding matrix.Unlike classical rank, it avoids choosing a hard nonzero threshold and also quantifies embedding whitening.
  • RankMe: A Simple Method and Its Implementation: RankMe should compare runs of a given method because embedding rank is not the only factor affecting performance.The authors estimate RankMe accurately using 25,600 samples in practice.
  • RankMe Predicts Linear Probing performance Even on Unseen Datasets: RankMe is compared with label-based linear evaluation and other unsupervised criteria, while finetuning is excluded because it can alter pretrained representations.The paper also notes that dimensional collapse is not the only factor determining performance, so cross-method comparisons should be avoided.
  • Experimental Methods and Datasets Considered: Across five JE-SSL methods and eleven datasets, RankMe distinguishes successful from sub-optimal training without labels or downstream task samples.The evaluation covers linear and nonlinear probing regimes.
  • RankMe as a Prediction of Linear Classification Accuracy: For a given method, higher embedding rank generally improves performance on both the pretraining dataset and downstream datasets, although high rank is only a necessary condition.On StanfordCars, the best VICReg performance does not occur at the highest rank, while full-rank embeddings still perform well.
  • RankMe Also Holds for Non-linear Probing: The rank-performance relationship persists with a three-layer MLP head and k-NN evaluation, indicating that RankMe extends beyond linear evaluation.Figure 3 reports the same conclusion before and after the projector and for k-NN on ImageNet.

4. RankMe for Label-Free Cross-Validation

RankMe selects JE-SSL hyperparameters by choosing the model with the highest embedding rank, without labels or tuning. Across embedding, representation, OOD, iNat-18, and finetuning evaluations, it generally retrieves performance close to oracle-based selection.

  • Selection procedure: RankMe selects the hyperparameter value producing the highest rank, using the first value reaching a maximum when ordered values tie.The procedure is illustrated for loss and optimization hyperparameters, including temperature, learning rate, and weight decay.
  • Embedding evaluation: Across ImageNet embedding evaluations, RankMe retrieves most oracle-selected performance, with average gaps below half a point.On OOD datasets, RankMe sometimes improves performance and has similar average performance to ImageNet validation selection.
  • Representation evaluation: For representations before the projector, RankMe trails the ImageNet oracle but remains within less than half a point on average while requiring no labels or tunable parameters.RankMe also outperforms α-ReQ in most settings and avoids similarly severe worst-case drops.
  • Beyond ImageNet: On iNat-18 pretraining, RankMe matches the iNat-18 oracle for VICReg and slightly improves OOD performance, while SimCLR shows a small drop versus oracle selection.For VICReg, RankMe also beats the ImageNet oracle and α-ReQ by a significant margin.
  • Finetuning benchmarks: In finetuning-based benchmarks, RankMe retrieves most task-oracle performance, with small drops in semi-supervised ImageNet evaluation and matching or increased performance for object detection.These results suggest RankMe remains useful even though finetuning changes pretrained weights and may change representation rank.
  • Supporting analyses: RankMe’s selection behavior is supported by cross-dataset rank transfer and correlations between embedding and representation performance.Figure 5 reports rank transfer across source and target datasets and highly correlated train and test accuracy across datasets.

5. RankMe: From Theory to Implementation

RankMe is theoretically motivated by the relationship between embedding rank and approximation or classification performance, then extended empirically to unseen target datasets. Its use depends on assumptions about probing, monotonicity, and rank transfer across distributions.

  • Theory: RankMe is constructed by relating embedding rank to approximation and classification performance.The theoretical development uses approximation bounds and Cover’s theorem to motivate rank as a performance indicator.
  • Theory: Maximum training accuracy in linear regression or classification increases with embedding rank, plateauing for classification once rank exceeds the number of classes.
  • Theory: A higher RankMe value predicts greater training performance only when comparing models from the same method, because rank is not the sole performance factor.
  • Empirical assumptions: Extending RankMe to target performance requires verifying that probes do not overfit, embedding and representation performance are monotonically linked, and source and target ranks are related.
  • Empirical assumptions: Rank scales linearly across ImageNet and iNaturalist representations, with Pearson correlations above 0.99, whereas StanfordCars shows weaker scaling because of limited ImageNet overlap.

6. Conclusion

The paper concludes that dimensional collapse provides a label-free metric for evaluating JE-SSL representations and selecting models. RankMe shows promise beyond linear classification, but broader validation remains future work.

  • Dimensional collapse can evaluate JE-SSL models from training data alone, without labels, additional training, or tunable parameters.
  • RankMe provides promising results in nonlinear classification despite the paper’s primary focus on linear classification.
  • RankMe is competitive with oracle-based hyperparameter selection, supporting its use where labels are scarce.
  • The authors identify more varied scenarios as future work for establishing RankMe’s broader usefulness in designing self-supervised methods.
  • Methods studied: The implementation study covers SimCLR, VICReg, VICReg-exp, and VICReg-ctr, with losses combining invariance, variance, and covariance-related components.

B. Visualizations on iNaturalist-18

On iNaturalist-18, RankMe reproduces the trends observed on ImageNet for SimCLR and VICReg. Because the representation dimension is 2048, RankMe values above that threshold are clipped.

  • RankMe produces the same trend on iNaturalist-18 as on ImageNet for both SimCLR and VICReg.
  • 2048 is the practical maximum RankMe value because the embedding manifold cannot exceed the 2048-dimensional representation size.

C. Applicability to cluster based methods

The paper extends RankMe to DINO by analyzing collapse around its projector and clustering prototypes. RankMe can assess DINO performance and select hyperparameters using either embeddings or prototypes.

  • DINO structure: Dimensional collapse occurs before DINO’s clustering layer and also affects the clustering prototypes, though at different levels.
  • DINO structure: DINO’s projection head can be interpreted as a classical projector followed by a clustering layer whose weights are clustering prototypes.
  • RankMe evaluation: RankMe measures DINO performance and enables hyperparameter selection, including when computed directly on prototypes.
  • RankMe evaluation: Prototype-based RankMe produces results similar to embedding-based RankMe and matches the ImageNet oracle in the reported comparison.

D. Complete visualizations on all datasets

Additional visualizations largely reproduce the earlier relationship between representation rank and downstream performance across datasets. Higher rank generally supports better generalization, although Food101 shows nearly identical performance across methods.

  • The additional visualizations reinforce RankMe’s earlier validation.
  • Higher rank generally helps downstream generalization across the additional datasets.The relationship is visible for both embeddings and representations.
  • Food101 is an exception, with almost identical performance across methods.

E. Detailed results for α-ReQ

The appendix examines α-ReQ as an alternative rank-related criterion and finds weak direct alignment with downstream performance. Its power-law assumption fails for collapsed embeddings, whereas the entropic and classical rank estimators agree closely.

  • α-ReQ and performance: α-ReQ shows no clear increase in downstream performance as α approaches one.This holds both when α-ReQ is measured on representations and when applied to embeddings.
  • α-ReQ and performance: α-ReQ can still produce good hyperparameter selections despite its weak visible performance trend.
  • α-ReQ and performance: Performance sometimes improves as α decreases, including on StanfordCars, iNaturalist18, and ImageNet.Lower α corresponds to a more uniform singular-value distribution, consistent with RankMe’s objective.
  • Power-law assumption: The power-law prior holds for non-collapsed embeddings but fails for collapsed embeddings.For collapsed embeddings, it poorly approximates the main rank plateau formed by the highest singular values.
  • Rank estimators: The entropic and classical rank estimates correlate with a Pearson coefficient of 0.99.The entropic estimator additionally accounts for the degree of whitening in the embeddings.

G. Convergence of the rank estimators

Rank estimation converges quickly with increasing sample size, while the classical and entropic estimators produce highly similar qualitative conclusions. The rank relationship also transfers from source to target datasets.

  • Estimator convergence: The rank estimates converge extremely quickly on ImageNet, especially for VICReg.
  • Estimator convergence: More than 95% of the final rank is obtained with 10000 samples for both VICReg and SimCLR.The entropic estimator converges more slowly because it is more sensitive to singular values.
  • Transfer across datasets: Embedding rank transfers from source datasets to target datasets.The transfer estimates use 25600 images from each respective dataset.
  • Rank and performance: Higher ImageNet rank leads to improved performance across methods and datasets, with an even clearer link for representations.
  • Estimator comparison: The classical and entropic estimators yield extremely similar behaviors despite differing exact values.

I.2. Evaluation

The evaluation uses specified downstream optimization settings and reports top-1 results for representations, embeddings, and post-projector rank across multiple experimental runs. Tables organize the corresponding hyperparameters and outcomes.

  • Evaluation setup: Downstream evaluation uses optimization parameters specified in Table S2.
  • Evaluation setup: The appendix includes hyperparameter tables for all runs and continued entries.
  • Reported metrics: The appendix reports top-1 accuracies for representations across all settings.
  • Reported metrics: It separately reports top-1 accuracies for embeddings across all settings.
  • Reported metrics: Additional tables report rank after the projector across all settings.
Loading 2210.02885v3…