Benchmarking Self-Supervised 3D Medical Vision Foundation Models and Transferability Estimation

1 Introduction

• Published evaluations use different task collections, data partitions, preprocessing, fine-tuning settings and baselines, limiting direct comparisons between models.A result establishes an advantage only within its reported setting, not superiority over models evaluated elsewhere.Source: §1 · pp. 2–3
• Transferability estimation uses pretrained outputs or frozen features, often with target labels, to predict relative post-adaptation performance without fine-tuning every candidate.Existing estimators draw on label relationships, simple readouts, class separation or feature information; their performance varies with the models, tasks and evaluation setting.Source: §1 · p. 3
• Task requirements differ: segmentation needs local evidence, while classification and regression require patient-level feature variation related to outcomes.A global feature summary may miss small structures, and converting continuous outcomes to categories can discard ordering.Source: §1 · p. 3
• Across 14 tasks, no candidate ranked among the top three on every task, and matched comparisons showed that model advantages depend on architecture, objective and pretraining data.The benchmark includes segmentation, classification and regression tasks across several anatomies and MRI and CT inputs.Source: §1 · p. 3
• Evaluating 23 candidates, including pretrained encoders and scratch controls, on common tasks provides a shared reference for comparing model design and task suitability.Three-fold downstream outcomes support comparisons under the specified adaptation protocols.Source: §1 · p. 3
• InsightRank combines target-appropriate frozen-feature statistics with architecture-aware pairwise comparisons to rank candidates before fine-tuning.Its evaluation considers ranking agreement, three-model shortlist performance and recorded computational cost.Source: §1 · p. 4

2.1 A controlled benchmark of 23 models across 14 medical imaging tasks

• All 23 candidates were evaluated on the same 14 tasks, yielding 322 three-fold model–task means across nine segmentation and five image-level tasks.The pool comprised 21 pretrained encoders and two scratch controls; tasks covered several anatomies and MRI and CT inputs.Source: §2.1 · pp. 4, 6

2.2 No foundation model ranks in the top three on every downstream task

• No candidate ranked in the Top 3 across all 14 tasks, and leaders varied among segmentation, classification and regression tasks.Seven candidates led the nine segmentation tasks; the task leaders also differed across the five image-level tasks.Source: §2.2 · p. 6, Figure 3 · p. 10

2.3 Architecture and pretraining choices distinguish model performance

• ResEncL ranked higher than Primus-M in 57 of 70 matched task comparisons, while objective and pretraining-data effects varied across encoder families and tasks.The matched architecture comparison covered five pairs with the same full-OpenMind source and objective; data-subset checkpoints also showed repeated gains and losses across tasks.Source: §2.3 · p. 8

2.4 Benchmarking established transferability estimators on 3D medical tasks

• Across the predefined four-task panel, conventional estimators showed mixed agreement: segmentation CC-FV reached Kendall τ-b 0.447, while image-level LogME averaged 0.105 across four layers.CC-FV’s three-model shortlist incurred 1.253 Dice percentage points of mean regret; image-level LogME’s deepest-layer mean was 0.055.Source: §2.4 · pp. 10–11

2.5 InsightRank combines target- and architecture-aware evidence

• InsightRank’s mean Kendall τ-b was 0.431 across nine segmentation tasks and 0.352 across five classification and regression tasks.Its segmentation weighted Kendall correlation was 0.549 on the predefined four-task panel, versus 0.365 for CC-FV; image-level full-ranking correlations exceeded those of matched conventional estimators.Source: §2.5 · pp. 11, 13, Figure 4 · p. 12, Table 1 · p. 13

2.6 Three-model shortlists retain strong segmentation models but miss image-level winners

• InsightRank’s three-model shortlist included a task winner on four of nine segmentation tasks but none of the five image-level tasks.Across segmentation tasks, mean regret was 0.239 Dice percentage points; Top 3 regret was unavailable for the 23-model image-level panel.Source: §2.6 · pp. 13–14, Figure 5 · p. 14

2.7 Fine-tuning three selected models reduces recorded segmentation job time

• Across seven segmentation tasks, the three-model workflow’s recorded job time was 85.8% below exhaustive adaptation.The 257.8-hour subtotal included 245.5 hours of selected-model training and 12.4 hours of feature extraction; feature-statistic computation was excluded, and no matched image-level timing comparison was available.Source: §2.7 · pp. 14–15

3 Discussion

• 57 of 70 matched-objective comparisons favored ResEncL over Primus-M, yet no candidate ranked in the top three on every task; architecture, objective, and data composition matter jointly.Subset pretraining and externally pretrained weights also varied in performance across targets.Source: §3 · p. 15
• Lesions, organs, and arterial trees place different demands on segmentation, and leaders vary even within brain MRI; scratch controls also remained competitive on several regression tasks.DINO3D led brain-age and post-stroke-interval estimation but ranked lower on survival, while CNN-VF showed a different balance of top-three placements and worst-task performance.Source: §3 · pp. 15–16
• InsightRank uses local voxel and boundary evidence for segmentation, but patient-feature variation and neighbour-outcome agreement for classification and regression.Architecture-aware pairwise comparisons improved on global voting of the same segmentation indicators, while combining effective dimensionality and neighbour-outcome agreement improved image-level ranking correlations over matched conventional estimators.Source: §3 · p. 16
• 0.239 Dice percentage points was InsightRank’s mean loss versus exhaustive selection across nine segmentation tasks; recorded job time fell 85.8% across seven, while image-level Top 3 contained no task winner.The image-level winner count does not establish shortlist regret, which was unavailable for those tasks.Source: §3 · p. 16
• Generalization is bounded by an OpenMind-centred candidate pool and brain-weighted task set; decoder, optimization, and data differences prevent attributing matched-comparison results to one design choice.Image-level tasks all contributed to InsightRank development, unlike the four segmentation evaluation tasks held apart from rule development.Source: §3 · pp. 16–17
• Common downstream evaluation and explicit model selection provide complementary evidence for judging where pretrained models transfer and how efficiently suitable candidates can be identified.The discussion calls for evaluating models across representative targets rather than relying on corpus size or modality matching alone.Source: §3 · pp. 15, 17

4.1 Benchmark design, model pool and downstream tasks

• The benchmark evaluates 23 candidates—21 pretrained encoders and two architecture-matched scratch controls—on nine segmentation, two classification and three regression tasks.The pool includes 11 ResEncL-like CNNs and 12 transformers, with OpenMind and external pretraining sources represented.Source: §4.1 · pp. 17–18
• Tasks span multiple anatomies and modalities, including private hospital-curated coronary-plaque and bridge-vessel CTA cohorts whose reported results remain provisional.Bridge-vessel instance labels are combined into one foreground class; 44 unmatched series are encoded as all-background.Source: §4.1 · p. 18

4.2 Downstream training and reference rankings

• Segmentation fine-tunes the full pretrained network and output layer, while classification and regression use task-specific configurations and 100-epoch training.Segmentation generally uses 150 epochs with foreground oversampling and spatial and intensity augmentation; image-level tasks use an effective batch size of 16.Source: §4.2 · pp. 18–19
• Candidates are ranked separately within each task using mean Dice for segmentation, mean AUROC for classification, or negative mean absolute error for regression.Segmentation evaluates the final epoch-150 checkpoint; classification and regression select the checkpoint with the lowest validation loss, and fold metrics are averaged equally.Source: §4.2 · pp. 18–19

4.3 Target-data sampling and frozen-feature extraction

• All candidates use fixed, shared target-data samples, with encoders in evaluation mode and gradient calculation disabled.Segmentation retains features at labelled voxel locations; classification and regression retain one feature vector per patient at selected depths.Source: §4.3 · p. 19
• Segmentation sampling retains at most 100 patients, two cases per patient, and bounded background, foreground and boundary points at common coordinates.Each case contributes at most 2,560 background points, 256 foreground points and 200 ground-truth boundary anchors; four encoder depths are stored in FP16.Source: §4.3 · p. 19
• Image-level sampling retains one case per patient, while spatial features or patch tokens are averaged into patient vectors at four architecture-specific depths.Regression sampling uses outcome quintiles but InsightRank's outcome-agreement statistic retains continuous outcomes; scratch features use fixed random initializations not verified against fine-tuning runs.Source: §4.3 · p. 20

4.4 Representation statistics for segmentation and image-level prediction

• Class-balanced nearest-centroid accuracy, G, assesses voxel-class separability, with equal class weighting limiting the influence of abundant background voxels.Finite case scores are averaged within patients and then equally across patients; centroids and queries come from the same case.Source: §4.4 · pp. 20–21
• Activation amplitude, N, measures feature-response magnitude under fixed extraction, but depends on feature scale, channel count and normalization placement.It measures response amplitude rather than class information.Source: §4.4 · p. 21
• Balanced centroid margin, B, compares the relative distance to the correct class and its closest competitor, with larger raw margins indicating stronger correct-class advantage.Class and case margins are averaged with equal weighting across eligible classes and patients.Source: §4.4 · p. 21
• Effective dimensionality, C, measures the breadth of patient-feature variation, while neighbour-outcome agreement, R, tests whether nearby patients have similar outcomes.C uses covariance-proportional eigenvalues without PCA reduction; R uses AUROC for binary labels and Spearman correlation for regression, averaging the two deepest representations.Source: §4.4 · pp. 21–22

4.5 Architecture-aware pairwise ranking

• Each encoder pair receives one indicator according to architecture groups and target type, and the oriented indicators are combined through their orderings.Indicators have different units, so pairwise comparison uses their relative ordering rather than pooling their raw values.Source: §4.5 · p. 22
• Segmentation pairs use G for CNN–CNN comparisons, −N for transformer–transformer comparisons and −B across architectures; image-level transformer pairs use R and other pairs use C.These assignments and directions were fixed during development without task-specific fusion weights; negative B is a ranking choice although larger raw B indicates greater centroid separation.Source: §4.5 · p. 22
• Net-win aggregation assigns +1, −1 or zero to wins, losses or ties and ranks models by their total pairwise score.The score depends on the candidate pool: pairwise cycles may occur, and architecture-group sizes change the balance of comparisons.Source: §4.5 · pp. 22–23

4.6 Baseline transferability estimators and adaptations

• LogME, H-score, GBC, NLEEP and CC-FV assess complementary frozen-feature properties, with CC-FV included for segmentation.LogME evaluates linear feature-to-target evidence, H-score compares between-class variation with overall covariance, GBC measures Gaussian-class overlap, and NLEEP uses feature-derived mixture assignments.Source: §4.6 · p. 23
• Segmentation comparators use final encoder representations, capped deterministic sampling, at most 64 PCA dimensions, and diagonal-Gaussian approximations for CC-FV and GBC.The CC-FV comparison evaluates the scalable diagonal-Gaussian approximation rather than its published full-covariance formulation.Source: §4.6 · p. 23
• Image-level comparisons use LogME, H-score, spherical-Gaussian GBC and NLEEP, with quintile adaptations for categorical regression estimators.LogME accepts continuous regression targets; original source-posterior LEEP and NCE have no reported scores because source-class probabilities are unavailable.Source: §4.6 · p. 23

4.7 Evaluation design and controlled comparisons

• The controls use the stated candidate pool and task-specific downstream references across nine segmentation tasks and five image-level tasks.A predefined comparator analysis covers HNTS-MRG, KiTS19, Coronary-Plaque and CoronaryBridgeVessel; the coronary tasks retain their annotation qualifications.Source: §4.7 · p. 23
• Pair-type concordance measures indicator agreement with downstream ordering within encoder-pair categories, giving task-level concordances equal weight.Predicted ties count as one-half agreement, while downstream ties are excluded.Source: §4.7 · p. 24
• Image-level controls compare C alone, R alone and a binary CNN-first prior to distinguish feature-statistic information from encoder-group ordering.Budget controls compare same-size uniform random subsets and a fixed cross-task model ordering; candidate-pool sensitivity analyses vary the pool while keeping the primary rule fixed.Source: §4.7 · p. 24

4.8 Ranking accuracy and model-selection performance

• Ranking correlations assess model-pool ordering, while Top k hit and regret assess the performance retained when fine-tuning only k candidates.Task summaries give equal weight to tasks within each analysis, with segmentation and classification/regression reported separately.Source: §4.8 · p. 24
• Segmentation uses hyperbolic weighted Kendall correlation as its primary rank endpoint, while image-level tasks use Kendall τ-b and Spearman ρ as principal endpoints.The segmentation implementation rounds inputs to ten decimal places; rank correlations retain ties, and constant score vectors have undefined correlation.Source: §4.8 · p. 24
• Shortlist regret compares selected candidates with the best full-pool candidate, using Dice percentage points for segmentation and normalized task-range loss for image-level Top 1.Image-level Top 3 regret is unavailable, and its normalized loss and the segmentation Dice-point loss use separate scales.Source: §4.8 · pp. 24–25
• Segmentation resolves score ties by stable descending scores and ascending model IDs, whereas image-level endpoints average over uniform selection within a cutoff tie.Observed three-fold means and task means are descriptive; individual model contrasts have no confidence intervals, and no confirmatory hypothesis tests are used.Source: §4.8 · p. 25

4.9 Computational cost analysis

• Computational accounting separates feature extraction, statistic calculation and downstream training, and sums recorded job durations as cumulative job time.The comparison is exhaustive fine-tuning versus feature extraction for all candidates followed by fine-tuning only selected models.Source: §4.9 · p. 25
• Segmentation timing covers seven tasks with complete 150-epoch logs for all 23 candidates across folds 0–2; exhaustive fine-tuning sums 483 job records.The predicted Top 3 sums 63 training records plus all-candidate feature-extraction time; two tasks are omitted for unavailable logs, and feature-statistic computation is excluded.Source: §4.9 · p. 25
• Classification and regression have no matched full-pool end-to-end timing comparison, so their ranking results do not support direct extrapolation of segmentation time reductions.Image-level computational cost depends on sample size and feature dimension; per depth, effective dimensionality costs O(n^2d + n^3), and exact neighbour distances cost O(n^2d).Source: §4.9 · p. 25

• Public datasets are accessible through their providers, subject to the providers’ access terms.Source: Data availability · p. 28
• Aggregate results are reported in the text, figures, and tables.Source: Data availability · p. 28
• The preprint omits the complete model–task score matrix and individual-level data from two private, hospital-curated coronary CTA cohorts.Source: Data availability · p. 28