A Comparative Study of Label-free Representation Quality Metrics in Deep Learning

• Label-free metrics compute representation properties solely from features, offering a training-free alternative to exhaustive downstream evaluation when selecting among increasingly many public models.Existing evaluations often cover narrow architectural and training configurations, limiting general conclusions about reliability.Source: §1 · pp. 1–2
• The paper groups label-free metrics into covariance spectrum-based, pairwise relation-based, and manifold-based families, and places metrics within each family in a common framework.Source: §1 · p. 2
• Controlled synthetic experiments characterize how covariance spectrum-based metrics respond to eigenspectrum shape and representation dimensionality.Source: §1 · p. 2
• A systematic study tests metric correlations with test accuracy from an Ordinary Least Squares fit and their consistency across architecture classes and training objectives.Source: §1 · p. 2
• The results indicate that label-free metrics should be used cautiously because their reliability is influenced by model architecture and training objective.Because these metrics use geometric information without downstream-task knowledge, some unreliability is expected a priori.Source: §1 · p. 2

• Self-supervised methods based on contrastive learning, self-distillation, and masked reconstruction produce representations competitive with supervised methods, especially under fine-tuning.Source: §2 · p. 2

Label-dependent metrics

• Label-dependent evaluation includes linear probing, KNN or clustering, fine-tuning, LEEP, LogME, and PACTran, but these methods require target labels and fine-tuning is substantially more costly.Source: §2 · pp. 2–3

Label-free metrics

• Label-free metrics span spectral, relational, and manifold-based families, yet many were proposed or validated in isolation and some rely on unavailable projector outputs rather than backbone representations.Source: §2 · p. 3

• Existing direct comparisons rarely test how architecture class, representation dimension, or training objective influence metric reliability, despite metrics favouring distinct representation geometries.Source: §2 · p. 3

3 A Taxonomy for Label-Free Metrics

• Metrics are computed on backbone outputs for self-supervised models and penultimate-layer activations for supervised models, excluding projectors and classification heads.For self-supervised models, the projector MLP is discarded and metrics operate on X rather than projector embeddings Z.Source: §3 · p. 3, Figure 1 · p. 4
• The taxonomy groups metrics into spectral, relational, and manifold families, capturing global variance structure, pairwise relationships, and representation-manifold geometry, respectively.Spectral metrics use covariance eigenspectra or singular values; relational metrics use pairwise interactions; manifold metrics target dimensionality and local structure.Source: §3 · pp. 3–4, §3.1 · p. 4, §3.2 · p. 6

3.1 Spectral Metrics

• Spectral metrics characterize rank collapse and dimensional utilization through covariance eigenvalues or singular values, with RankMe measuring effective dimensionality without assuming power-law spectral decay.RankMe increases when variance is distributed across more directions, whereas α-ReQ interprets power-law decay and NE Sum measures whitening; condition number can reflect solver sensitivity rather than representation quality.Source: §3.1 · pp. 4–6

3.2 Relational Metrics

• Relational metrics use pairwise geometry: Self-Cluster is highest for collapsed representations and lowest for uniform hyperspherical distributions, while DSE measures diffusion-spectrum entropy.Self-Cluster and DSE depend on related pairwise structure and exhibit opposite behavior: clustering increases Self-Cluster but decreases DSE; DSE is also sensitive to kernel bandwidth.Source: §3.2 · pp. 6–8

3.3 Manifold Metrics

• Manifold metrics estimate geometry beyond linear subspaces, with TwoNN intrinsic dimension recovering the dimensionality of curved manifolds and lower ID associated with better generalisation.The evaluation computes ID from repeated 90%-subsamples within batches and averages across batches; CLID is excluded because it requires labels.Source: §3.3 · p. 8

4.1 Sensitivity of Spectral Metrics

• Synthetic power-law spectra show that RankMe and NE Sum peak at α = 0, whereas κ is symmetric, minimized near α = 0, and grows toward spectral extremes.RankMe and NE Sum scale with representation dimension d, while κ becomes unbounded as |α| increases; dividing by d enables comparison across dimensions.Source: §4.1 · pp. 9–10
• No spectral metric has uniform sensitivity: NE Sum⋆ is most sensitive near α = 0, RankMe⋆ is less sensitive there, and κ⋆ is largely uninformative near α = 0.For α > 0, RankMe⋆ and NE Sum⋆ decrease with d even after normalization, while for α ≤ 0 they are insensitive to d.Source: §4.1 · pp. 9–10

4.2 Correlation of Metrics with Test Accuracy

• Relational metrics are nearly mirror images, with Self-Cluster and DSE correlations ranging from −0.999 to −0.998, but their apparent strength depends on the probe.They show mild significant OLS correlations on selected datasets and moderate-to-strong KNN correlations across all six datasets because KNN directly uses nearest-neighbour structure.Source: §4.2 · pp. 11–12
• Training objective also changes reliability: ID is significant for SSL models across five object-classification datasets, while Self-Cluster is strong for SSL but insignificant for supervised models.For SSL, ID correlations range from ρ ∈[−0.88, −0.46] and Self-Cluster from ρ ∈[−0.91, −0.63]; NE Sum is reliable across all six datasets for SSL but only half for supervised models.Source: §4.2 · pp. 14–15
• Architecture class strongly confounds metric reliability: RankMe is significant in 23 of 24 architecture–dataset cells, while ID is insignificant for ResNet on four of six datasets.Models from different architecture classes therefore should not be ranked jointly using these metrics; ID remains reliable mainly on generic object classification and Flowers-102.Source: §4.2 · p. 13, Table 3 · p. 15

Spectral metrics

• Spectral metrics are generally unreliable: α-ReQ is fit-range dependent, RankMe correlates weakly in aggregate, and NE Sum predicts performance only for SSL models.RankMe’s entropy construction may reward variance spread that is unrelated to task-relevant information, whereas NE Sum reflects whitening encouraged by SSL but not supervised cross-entropy training.Source: §5 · p. 16, Table 4 · p. 17, Figure 7 · p. 18

Manifold-based metrics

• ID is the most reliable predictor across architectures and objectives, with ρ ∈[−0.95, −0.26], while remaining significant on five of six datasets after controlling for parameter count.Its correlation magnitude is comparable under OLS and KNN probes (mean ratio 1.04), and it is strongest on coarse object-category datasets but weakest on Places-365.Source: Table 4 · p. 17, §5 · pp. 17–18

Relation-based metrics

• Relation-based metrics are effective for SSL representations but uninformative for supervised ones, with DSE and Self-Cluster showing ρ ≈−0.999 and Self-Cluster varying by architecture.Self-Cluster is significant on all six datasets for SSL models, reliable on 5 of 6 ResNet datasets, 3 of 6 ConvNeXT and RegNet datasets, and 0 of 6 ViT datasets.Source: §5 · p. 17, Table 4 · p. 17

Limitations

• The study’s conclusions are limited by OLS regression assumptions, incomplete analysis of nonlinear probes, only 19 SSL+Sup models, imbalanced architecture counts, and vision-only classification experiments.The authors call for larger, more balanced model pools and evaluation across other modalities and task types.Source: §5 · p. 19

6 Conclusion

• Across 260 models, intrinsic dimensionality (ID) is the most broadly reliable metric, with directionally consistent correlations more often than any other metric across architectures and training objectives.ID remains robust to the evaluation probe choice, including OLS and KNN.Source: §6 · p. 19
• Metric reliability is strongly moderated by architecture class and training objective, effects that aggregate results obscure.This dependence is reported across the study’s 260-model analysis and referenced in Table 2a.Source: §6 · p. 19
• ID reliability decreases as tasks move beyond coarse object-category classification, suggesting that ID tracks representation-manifold separability relative to target-label structure.Spectral and relational metrics are reliable only within specific regimes.Source: §6 · p. 19
• The metrics are grouped into spectral-based, pairwise relation-based, and manifold-based families, whose complementary information could motivate combinations such as CLID (Lu et al., 2023).The study also investigates spectral-metric sensitivity to spectral shape and representation dimension.Source: §6 · p. 19

A Approximation of DSE

• For ℓ2-normalised representations, the squared Euclidean distance satisfies ∥xi − xj∥2 = 2 − 2xi · xj.This identity is the starting point for the DSE approximation.Source: A Approximation of DSE · p. 23
• The DSE approximation uses a default setting of σ = 10, with the supplied derivation stating that a relevant quantity is constant.The appendix also states the conditions σ2 ≤ 0.01 and |xi · xj| ≤ 1 and invokes a first-order Taylor expansion of exp.Source: A Approximation of DSE · p. 23

B.1 α-ReQ Fitting-Range Sensitivity

• The α-ReQ ranges (10, 0.9d) and (10, 0.8d) produce near-identical correlations across all ten datasets, indicating tail truncation is not material.These are the two paper-adjacent fitting ranges compared in Figure 8.Source: §B.1 · p. 23
• Including dominant head eigenvalues substantially changes α-ReQ correlations, with Flowers-102 shifting from +0.31 to −0.29 under the (0, d/2) range.Both correlations are reported as significant and point in opposite directions.Source: §B.1 · p. 23
• Because fitting ranges are arbitrary and lack clear theoretical justification, the |α −1| ≈0 hypothesis depends on the selected range.The comparison covers five fitting ranges, including the paper heuristic, a narrower variant, and three alternatives.Source: §B.1 · p. 23

B.2 ID (TwoNN) Estimation Noise

• 7.6% median and 15.2% 95th-percentile across-batch noise leave TwoNN ID estimation noise small relative to the signal driving correlations.Figure 9 evaluates 2,600 model–dataset pairs and reports across-batch variability of per-batch mean ID.Source: §B.2 · p. 23
• 4.5% median and 8.2% 95th-percentile intra-batch noise show that TwoNN ID estimates are usually stable within a batch.Intra-batch noise measures variability across 20 independent 90%-subsamples within each batch.Source: §B.2 · p. 23
• Most model–dataset pairs show under 10% relative TwoNN ID noise, although both intra-batch and across-batch distributions are right-skewed.The reported noise is evaluated across 260 models and 10 datasets.Source: §B.2 · p. 23

B.3 DSE Estimation Noise

• DSE estimation noise is low and stable across the model pool, with a median relative standard deviation of 0.8%.Source: §B.3 · p. 23
• Only 5% of DSE estimates exceed 2.2% relative standard deviation, as indicated by the 95th percentile.Source: §B.3 · p. 23
• TwoNN ID estimation is noisier, with median intra-batch and across-batch relative standard deviations of 4.5% and 7.6%, respectively.Source: Figure 9 · p. 24

Partial correlation results

• Partial Spearman correlations were computed across six datasets for 255 models, excluding five models without available parameter counts, with points coloured by log10 parameter count.The correction removes shared variance between ID and accuracy attributable to model capacity.Source: §B.4 · pp. 24–25, Figure 11 · p. 25
• ID remains significantly associated with accuracy after controlling for log parameter count on CIFAR-100, ImageNet, Flowers-102, Food-101, and one other dataset, but not Places-365.Partial ρ values are −0.434, −0.669, −0.664, and −0.210 respectively for the four reported datasets; Places-365 has partial ρ = −0.019, p = 0.759.Source: Figure 11 · p. 25, §B.4 · p. 25

Model inventory

• Tables 5 and 6 summarize torch hub, Hugging Face, and timm models by architecture, objective, input resolution, parameters, GMACs, and training-data provenance.The reported backbone statistics exclude classifier heads, and GMACs correspond to the listed input resolution.Source: Table 5 · p. 26, Table 6 · p. 26

Architectures and training

• The model pool includes ConvNeXT, ViT, and other architectures trained with supervised, self-supervised, or combined objectives across ImageNet, curated web, image–text, Instagram, and Flickr data.Examples include DINOv2/v3, OpenAI CLIP, LAION, MetaCLIP-2B, and other listed training sources.Source: Table 5 · p. 26, Table 6 · p. 26, C Models used · p. 29

Model complexity

• Model scale and computational cost vary substantially, from lightweight EfficientNet and ConvNeXT variants to billion-parameter ViTs with much higher GMACs.The tables provide parameters and GMACs for each frozen feature backbone, including models evaluated at different input resolutions.Source: Table 6 · p. 26, C Models used · p. 29

D Results on a Broader Dataset

• The broader-dataset evaluation reports Spearman correlations across datasets, using official train/test splits except for explicitly defined EuroSAT and Places-365 procedures.EuroSAT uses a deterministic stratified 80/20 split seeded with random.Random(0), while Places-365 uses standard train/validation splits with 256×256 decoding via PIL draft mode.Source: D Results on a Broader Dataset · p. 30

D.1 OLS linear probe

• The OLS analysis includes results stratified by the broader dataset evaluation.The supplied passage identifies the stratified OLS results but does not provide their numerical correlations.Source: §D.1 · p. 30
• OLS results compare spectral, relation-based, and manifold-based metrics against test accuracy across 10 evaluation datasets grouped by task type.Table 7 reports statistical significance after Benjamini–Hochberg correction, with non-significant cells greyed out.Source: Table 7 · p. 31
• OLS correlations for ImageNet-1k require caution because ImageNet-1k is used in the training pipeline of most backbones.This caveat limits direct interpretation of ImageNet-1k correlations as an independent evaluation signal.Source: §D.1 · p. 31

D.2 KNN probe

• The KNN subsection’s accompanying table organizes correlations by metric construction and architecture class across natural image datasets.The table caption describes OLS test accuracy, preferred metric directions, and Benjamini–Hochberg significance correction, so those labels are retained as reported.Source: Table 8 · p. 32
• ImageNet-1k correlations in the KNN analysis should likewise be interpreted cautiously because most backbones use ImageNet-1k during training.The caveat applies specifically to interpreting correlations involving ImageNet-1k.Source: §D.2 · p. 32

E The curious case of ViT models

• For ViT models, Table 9 evaluates Spearman correlations between representation-quality metrics and OLS test accuracy across fine-grained datasets, stratified by architecture class.Metrics are grouped into spectral, relation-based, and manifold-based constructions, with significance assessed after Benjamini–Hochberg correction.Source: Table 9 · p. 33

F.1 SGD results

• The SGD analysis freezes each backbone and trains only a linear probe with momentum 0.9 and no weight decay.This follows standard linear-evaluation practice.Source: §F.1 · p. 33
• The probe learning rate is optimized with Optuna over [1 × 10−4, 3 × 10−1] using three rounds of hyperparameter search.The final learning rate is then used with a cosine scheduler.Source: §F.1 · p. 33
• The SGD probe is trained for 100 epochs with ℓ2-normalized features, as commonly used in foundation-model evaluation.The analysis trains only approximately 120 models, a subset of those in Tab. C.Source: §F.1 · p. 33

• Tables 10–26 report Spearman correlations between representation metrics and OLS or KNN accuracy across task types, architecture classes, training objectives, and representation dimensions.Metrics are grouped as spectral, relation-based, or manifold-based, with statistical significance assessed after Benjamini–Hochberg correction.Source: Table 11 · p. 35, Table 12 · p. 36, Table 13 · p. 37
• Correlations involving ImageNet-1k should be interpreted cautiously because it is used in the training pipeline of most backbones.Source: §F.2 · pp. 35, 41
• Table 27 evaluates representation metrics against SGD-trained linear-probe accuracy across CIFAR-10, CIFAR-100, and ImageNet-1k under α < 1, α > 1, and unconstrained regimes.Bold underlined values indicate statistical significance at p < 0.05.Source: Table 27 · p. 50
• DSE values are dominated by ViT models at extreme values, while near-zero values primarily arise from Supervised and SSL+Sup models rather than SSL models.This suggests DSE may predict SSL models more reliably, although the SSL sample is small, and may be unreliable for Supervised and SSL+Sup models.Source: Figure 12 · p. 51
• Figures 13–16 compare DSE, Self-Cluster, RankMe, RankMe⋆, NE Sum, and ID against KNN probe accuracy across six datasets using Spearman correlations and annotated p-values.Marker size represents each model’s representation dimensionality, scaled relative to the highest-dimensional model.Source: Figure 13 · p. 52, Figure 14 · p. 53, Figure 15 · p. 54