Source-linked AI summary
A Comparative Study of Label-free Representation Quality Metrics in Deep Learning
Daniel Richards Arputharaj, Daniel Jönsson, Gabriel Eilertsen
TL;DR
Selecting among many pretrained representations is difficult because exhaustive downstream evaluation is expensive and existing label-free metrics have uncertain reliability across configurations. The paper unifies these metrics into three construction-based families, studies spectral sensitivity, and evaluates them across 260 vision models and six datasets. Intrinsic dimensionality is the most broadly reliable predictor, but metric reliability varies with architecture class and training objective.
Problem
The reliability of label-free representation metrics across architectures, training objectives, and datasets remains insufficiently understood, despite their computational efficiency.
Method
The study groups metrics into three families, analyzes spectral sensitivity with synthetic experiments, and compares metric correlations with downstream accuracy across models and configurations.
Results
Low intrinsic dimensionality is the most consistent indicator of representation quality, while relational metrics are especially predictive for self-supervised representations.
Takeaways & Limitations
Label-free metrics require interpretation by architecture class and training objective rather than universal application across models.
Takeaways & Limitations
Spectral metrics have non-uniform sensitivity to eigenspectrum shape, and κ’s apparent predictive power can arise from the OLS solver rather than representation quality.
Abstract
from arXiv · showhide
We present a comparative study of label-free metrics for assessing the quality of representations in deep neural networks to understand their reliability under a wide variety of configurations. We group existing label-free metrics into three families based on their construction and analytically establish connections between metrics within the same family. We then characterise the sensitivity of spectral metrics through controlled synthetic experiments. Finally, all label-free metrics are evaluated against downstream task accuracy across a diverse set of 260 vision models on six datasets spanning generic object classification, fine-grained object classification, scene recognition and geospatial task, stratifying results by architecture class and training objective. We find that intrinsic dimensionality (ID) is the most reliable predictor among the metrics considered. However, the reliability of all metrics, including ID, is moderated by architecture class and training objective. Our results provide a clearer understanding of what label-free representation quality metrics measure, when they are reliable, and how to interpret them in practice.
1 Introduction
The growing supply of pretrained models makes exhaustive downstream evaluation impractical, motivating label-free metrics computed directly from representations. This study compares their construction, spectral sensitivity, and reliability across architectures, objectives, and tasks.
- Label-free metrics assess representation properties from features alone, avoiding training and offering a computationally efficient alternative for model selection.
- Existing evaluations provide limited evidence because they often test a single metric across narrow architectural and training configurations.
- The study groups label-free metrics into covariance spectrum-based, pairwise relation-based, and manifold-based families, with a common framework highlighting within-family similarities.
- Controlled synthetic experiments characterise how spectral metrics respond to eigenspectrum shape and representation dimensionality.
- The evaluation tests metric correlations with downstream accuracy and consistency across architecture classes and training objectives.
- Results suggest label-free metrics should be used cautiously because their reliability depends on model architecture and training objective.
2 Related Work
Related work includes label-dependent transfer proxies and newer label-free metrics motivated by spectral, relational, and geometric properties. Direct comparisons remain uncommon and have rarely examined architecture, representation dimension, and training objective systematically.
- Representation learning seeks informative features that support efficient transfer to downstream tasks, with self-supervised methods becoming a dominant approach.
- Linear probing, K-nearest-neighbour classification, and clustering use frozen features as task-aware proxies for representation quality.
- LEEP, LogME, and PACTran estimate transfer suitability using target labels, motivating completely label-free evaluation metrics.
- Label-free metrics have been proposed from spectral properties of covariance matrices and other theoretical perspectives to avoid dependence on labels.
- Prior direct comparisons reported aggregate results without systematically examining architecture class, representation dimension, or training objective.
3 A Taxonomy for Label-Free Metrics
The paper organizes label-free representation quality metrics into spectral, relational, and manifold-based families, then connects metrics within each family. These metrics analyze representations directly through covariance structure, pairwise geometry, or intrinsic manifold properties.
- Representation input: The analysis uses representation matrix X, with metrics computed from backbone or penultimate-layer features rather than projector embeddings or classification heads.For SSL models, the projector is discarded; supervised models use penultimate activations.
- Spectral metrics: Spectral metrics use covariance eigenvalues or representation singular values to assess global structure and avoid rank collapse.Rank collapse concentrates variance in a small number of directions, whereas effective rank reflects how broadly variance is distributed.
- Spectral metrics: α-ReQ estimates spectral decay under a power-law assumption, with smaller α indicating slower decay and larger effective dimensionality.The exponent is estimated from a log-log fit to the eigenvalue spectrum, using a heuristic index range in the experiments.
- Spectral metrics: RankMe measures effective dimensionality from the Shannon entropy of normalized singular values without assuming a power-law spectrum.Higher RankMe indicates variance distributed across more directions and less dimensional collapse.
- Relational metrics: Relational metrics characterize representation geometry through pairwise interactions among samples, typically encoded in an N × N relational matrix.Their formulations differ in how pairwise relationships are quantified.
- Relational metrics: Self-Cluster increases for concentrated representations and reaches its minimum when representations are uniformly distributed on the hypersphere.It is linked to the Frobenius norm of the normalized Gram matrix and is negatively correlated with diffusion spectral entropy.
- Relational metrics: Diffusion spectral entropy is high for a uniform eigenspectrum and low when a few eigenvalues dominate, but extreme kernel bandwidths reduce its sensitivity.The metric is computed per batch and averaged across batches.
4 Comparative Analysis
The metrics respond differently to eigenspectrum shape and show varying associations with downstream accuracy. ID is generally the most consistent predictor, but reliability changes substantially with architecture class and training objective.
- Sensitivity of Spectral Metrics: RankMe and NE Sum peak at α = 0, whereas κ is minimized near α = 0 and grows as |α| increases.RankMe and NE Sum measure properties maximized by uniform spectra, while κ depends on the largest-to-smallest singular-value ratio.
- Sensitivity of Spectral Metrics: NE Sum⋆ is most sensitive near α = 0, RankMe⋆ is less sensitive there, and κ⋆ is largely uninformative near α = 0.The three metrics therefore provide complementary but non-uniform sensitivity across spectral shapes.
- Aggregate Correlations: ID is the only aggregate metric reliably correlated across all datasets, with strongest correlations on generic object classification and weaker correlations on Places-365 and EuroSAT.Its reported correlations range from ρ ∈[−0.63, −0.60] on generic object classification to ρ = −0.18 on Places-365 and ρ = −0.21 on EuroSAT.
- Aggregate Correlations: Relational metrics show mild correlations under OLS but moderate-to-strong correlations with KNN, consistent with their nearest-neighbour construction.Self-Cluster and DSE capture similar phenomena in opposite directions and are therefore highly related.
- Aggregate Correlations: κ correlations with OLS accuracy do not generally survive KNN, while RankMe and NE Sum show mild-to-moderate correlations whose strengths vary by probe.The strongest reported κ correlation is ρ = 0.55 on EuroSAT, and its OLS association is linked to solver conditioning.
- Architecture Effects: Architecture class substantially moderates metric reliability, so models from different architecture classes should not be ranked jointly using these metrics.RankMe becomes significant in all but one of twenty-four architecture–dataset cells, while ID is insignificant for ResNet on four of six datasets.
- Architecture Effects: ID remains reliable on generic object and Flowers-102 datasets but declines on Food-101 and specialised datasets, including significance in only one architecture class on EuroSAT.For ResNet, ID is significant only on ImageNet and Flowers-102.
- Architecture Effects: Self-Cluster is reliable for ResNet on five datasets but insignificant for ViT across all datasets because ViT values collapse near one.The near-unit values arise predominantly in supervised and SSL+Sup ViT models.
5 Discussion
The discussion finds that metric reliability depends strongly on architecture and training objective, making aggregate correlations difficult to interpret. Across the evaluated settings, intrinsic dimensionality is the most consistent indicator, although important methodological and scope limitations remain.
- Spectral metrics are generally not reliable: Spectral metrics have metric-specific sensitivities and are generally unreliable as standalone indicators of downstream performance.α-ReQ depends on the fitted eigenspectrum range, while RankMe can increase with dimensionality even when added dimensions may represent noise.
- Relation-based metrics are effective for SSL-based representations: Relation-based metrics are strong predictors for self-supervised representations but largely uninformative for supervised ones.Self-Cluster is significant on all six datasets for SSL models, whereas its reliability varies by architecture and is absent for ViT models.
- Manifold-based metrics show the strongest aggregate generalisation: Intrinsic dimensionality is the most consistent indicator of representation quality, showing relationships with test accuracy across architecture classes and training objectives.Its correlations range from ρ ∈[−0.95, −0.26], and it remains significant on five of six datasets after controlling for parameter count.
- Limitations: The analysis uses OLS probe accuracy, an assumption that limits conclusions about nonlinear heads, and includes an unbalanced vision-model pool with few SSL+Sup models.The study covers only vision models and image-classification tasks, so applicability to other modalities and task types remains unexplored.
6 Conclusion
The study organizes label-free representation metrics into three construction-based families and examines their relationships and sensitivity. It also analyzes spectral behavior under controlled changes in spectral shape and representation dimension.
- 6 Conclusion: The metrics are grouped into spectral-based, pairwise relation-based, and manifold-based families according to their construction.Where possible, the paper reformulates metrics to highlight similarities within and across these families.
- 6 Conclusion: The study investigates how spectral metrics respond to spectral shape and representation dimension.
- 6 Conclusion: For ℓ2-normalised representations, squared Euclidean distance is determined by the inner product through ∥xi − xj∥2 = 2 − 2xi · xj.
- 6 Conclusion: The default spectral-kernel setting uses σ = 10, with a small-σ approximation justified when σ2 ≤ 0.01 and |xi · xj| ≤ 1.
B Metric Estimation Stability
Metric-estimation stability is assessed across model–dataset configurations, with intrinsic dimensionality showing measurable within-batch and across-batch variability. Spectral fitting-range choices can also affect relationships with downstream accuracy.
- B Metric Estimation Stability: The two tail-truncated α-ReQ fitting ranges produce near-identical correlations with OLS accuracy across all ten datasets.The ranges compared are (10, 0.9d) and (10, 0.8d), suggesting tail truncation is not a material choice.
- B Metric Estimation Stability: TwoNN ID estimation has median intra-batch noise of 4.5% and median across-batch noise of 7.6% across 2,600 model–dataset pairs.The corresponding 95th percentiles are 8.2% and 15.2%, respectively.
B.3 DSE Estimation Noise
DSE estimation is highly stable across the evaluated model pool, while the accompanying spectral sensitivity analysis compares α-based correlations across fitting ranges. The DSE noise analysis uses relative standard deviation across batches.
- B.3 DSE Estimation Noise: 0.8% is the median relative standard deviation of DSE across batches, with a 95th percentile of 2.2%.These values indicate stable batched DSE estimation across the model pool.
- B.3 DSE Estimation Noise: Figure 8 compares Spearman correlations of α and |α − 1| with OLS test accuracy across five fitting ranges and ten datasets.
B.4 ID Partial Correlation Controlling for Model Capacity
The analysis tests whether ID–accuracy associations persist after controlling for model capacity. ID remains a significant predictor on five of six datasets, with Places-365 as the nonsignificant exception.
- B.4 ID Partial Correlation Controlling for Model Capacity: Figure 11 displays ID versus OLS test accuracy across six datasets, colouring points by log10 parameter count and annotating partial Spearman correlations.The analysis includes 255 models because parameter counts were unavailable for five models.
- B.4 ID Partial Correlation Controlling for Model Capacity: ID remains a significant predictor on five of six datasets after removing shared variance with log parameter count.
- B.4 ID Partial Correlation Controlling for Model Capacity: On Places-365, the partial ID–accuracy correlation is −0.019 with p = 0.759 and is not significant.
- B.4 ID Partial Correlation Controlling for Model Capacity: The reported partial correlations are CIFAR-100 −0.434, ImageNet −0.669, Flowers-102 −0.664, and Food-101 −0.210.All four reported associations have p ≤ 0.001.
C Models used
The study draws models from Hugging Face, timm, and Torch Hub, documenting backbone characteristics and computational costs for frozen feature extractors.
- Model sources: Models are drawn from Hugging Face, timm, and Torch Hub collections.The supplied tables summarize model characteristics for the evaluated frozen backbones.
- Model characteristics: Backbone summaries report parameter counts and GMACs after removing classifier heads.GMACs are measured at each model’s listed input resolution.
- Training provenance: The model inventory includes pretraining-to-fine-tuning chains and abbreviations for several large-scale data sources.Examples include DINOv2/v3 curated web data, OpenAI CLIP data, LAION, Instagram, Flickr, and MetaCLIP-2B web data.
- Architecture coverage: The ConvNeXT entries include supervised ImageNet-1k models across femto, pico, and nano variants.The listed entries vary in input resolution, parameter count, and GMACs.
D Results on a Broader Dataset
Broader evaluations report Spearman correlations between representation metrics and OLS accuracy across datasets, while highlighting dataset-split details and an ImageNet-1k interpretation caveat.
- Evaluation scope: Spearman correlations are reported for a broader dataset set, with main-text datasets retained for comparison.The broader analysis is described alongside the main-text results.
- Dataset construction: EuroSAT uses a deterministic stratified 80/20 train/test split seeded with random.Random(0).The split compensates for the absence of an official EuroSAT partition and ensures disjoint, reproducible sets.
- Task-level analysis: The OLS results are also summarized across ten evaluation datasets grouped by task type.Table 7 organizes metrics as spectral, relation-based, and manifold-based.
- Interpretation caveat: ImageNet-1k correlations should be interpreted cautiously because ImageNet-1k appears in most backbones’ training pipelines.This caveat applies to the reported correlation analyses.
- Stratified analysis: The OLS results are additionally stratified by architecture class across natural-image datasets.Table 8 organizes correlations by architecture class and metric construction.
E The curious case of ViT models
The ViT-focused analysis examines architecture-stratified metric correlations under a standardized frozen-backbone linear-probe protocol.
- ViT comparison: The analysis compares metric correlations with OLS accuracy across architecture classes on fine-grained datasets.Table 9 organizes metrics by construction and reports architecture-stratified Spearman correlations.
- Evaluation protocol: The linear probe freezes the backbone and trains only the probe with SGD, momentum 0.9, and no weight decay.The learning rate is selected with Optuna over [1 × 10−4, 3 × 10−1].
- Evaluation protocol: Training uses three hyper-parameter-search rounds, cosine learning-rate scheduling, and 100 epochs.Features are also ℓ2-normalised before evaluation.
F.2 Summary of the results
The broader results are organized by training objective, dataset type, architecture, representation dimension, and probe type, using Spearman correlations and significance correction. Across these analyses, ImageNet-1k correlations carry an explicit training-pipeline caveat.
- Training objective: Tables 11–13 stratify OLS correlations by training objective across natural-image, fine-grained, and specialised datasets.Metrics are grouped as spectral, relation-based, and manifold-based, with Benjamini–Hochberg correction for significance.
- Interpretation caveat: ImageNet-1k correlation results should be interpreted cautiously because it is used in most backbones’ training pipelines.This caveat is repeated across the reported stratified analyses.
- Representation dimension: Tables 14–16 stratify OLS correlations by representation dimension across natural-image, fine-grained, and specialised datasets.The tables retain the same metric-family organization and significance procedure.
- KNN evaluation: Tables 24–26 stratify KNN correlations by representation dimension across natural-image, fine-grained, and specialised datasets.The metric families remain spectral, relation-based, and manifold-based.