Source-linked AI summary
Label-Free Foundational Model Selection for Medical Image Classification under Distribution Shift via Pseudo Label Discrepancy
Juan Iñaki Larrea, Lucas Mansilla, Enzo Ferrante
TL;DR
Under inter-institutional distribution shift, choosing among foundation models is difficult because target labels are rarely available. The paper repurposes SUDO into AURCC, a label-free ranking criterion using unlabeled target predictions, and finds rankings that track target performance, especially when labeled source data are scarce.
Problem
Under distribution shift, selecting the best foundation model for an unlabeled target domain is difficult because target-domain labels are rarely available.
Method
AURCC partitions unlabeled target data by predicted probabilities, measures pseudo-label discrepancy within regions, and aggregates these discrepancies into a label-free model-ranking score.
Results
Across three inter-hospital scenarios, AURCC strongly correlates with ground-truth target performance and is competitive with or superior to source-AUC ranking, particularly with small labeled sources.
Takeaways & Limitations
AURCC reduces dependence on target labels for choosing among chest X-ray foundation models and is most useful when labeled source data are scarce.
Takeaways & Limitations
Target subsampling to match the source positive rate was used as an experimental control, although it is not required by the method.
Abstract
from arXiv · showhide
Foundation models are increasingly deployed for medical image analysis. However, under the inter-institutional distribution shift typical of deployment, their performance varies widely and cannot be known without target-domain labels, which are rarely available. This leaves a practical question unresolved: given several candidate foundational models and labeled-data from a source domain, which one to deploy in an unlabeled target domain? We propose a label-free selection criterion built on SUDO, a framework for evaluating clinical AI systems without ground-truth annotations. SUDO partitions the unlabeled target data by predicted probability and, for each region, measures a pseudo-label discrepancy reflecting class contamination; aggregated across regions, this yields a score (AURCC) requiring neither target annotation nor fine-tuning. We show that AURCC can be used to rank a variety of vision-language models (BioMedCLIP, CXR-CLIP, CheXzero, MedCLIP, MedImageInsight, CLIP) on chest X-ray classification across three inter-hospital shift scenarios, under zero-shot and MLP-probe regimes. The AURCC ranking recovers the ground-truth ranking with Spearman rho up to 0.943 (p<0.05). Against the natural baseline of ranking by held-out source accuracy, AURCC is competitive when the labeled source is large and yields a more accurate ranking once it is small; the regime of interest in resource-constrained settings.
1 Introduction
Distribution shift can substantially degrade medical-image model performance, while target-domain annotations needed to measure that degradation are often unavailable. This work asks how to rank candidate foundation models for an unlabeled target domain using labeled source data.
- Distribution shift arises when clinical images differ across scanners, hospitals, patient populations, or acquisition protocols, potentially degrading performance substantially.
- Target-domain performance conventionally requires ground-truth annotations, but obtaining expert radiologist labels is costly and often inaccessible.
- Vision-language foundation models support zero-shot and lightweight probe-based transfer with little or no supervision.
- SUDO partitions unlabeled target data by predicted probabilities, measures pseudo-label discrepancy within regions, and aggregates these discrepancies into AURCC for model ranking.The proposed ranking uses source labels while target images only generate the probabilities driving partitioning.
- The framework ranks candidate models by AURCC without target labels, using embeddings produced from labeled source and unlabeled target datasets.
2 Methodology
The method ranks pretrained foundation models under distribution shift by applying SUDO to target probabilities and aggregating regional pseudo-label discrepancies into AURCC. It evaluates zero-shot and MLP-probe probabilities across six chest X-ray models and inter-institutional datasets.
- Each candidate foundation model receives an AURCC score computed without target labels, intended to approximate its true target-performance ranking.
- SUDO ranking criterion: For each model, positive-class target probabilities are discretized into K quantile bins, each containing unlabeled target samples.
- SUDO ranking criterion: Two auxiliary logistic regressions use opposing pseudo-label assumptions for each bin, producing held-out source AUC values s0(b) and s1(b).
- SUDO ranking criterion: Large |SUDOb(ϕm)| indicates class dominance, whereas values near zero indicate contamination; scores are averaged over 300 stochastic runs.
- SUDO ranking criterion: AURCC aggregates bin scores by tracing reliability against target coverage while sweeping symmetric quantile thresholds, yielding a score in [0, 1].
- Models and regimes: The study compares zero-shot probabilities with a source-trained shallow MLP probe and evaluates BioMedCLIP, CheXzero, CXR-CLIP, MedCLIP, MedImageInsight, and CLIP-base.
- Data and evaluation: The task is binary pneumonia classification across MIMIC-CXR, NIH ChestX-ray14, CheXpert, and PadChest, with PadChest serving as the shifted target domain.
- Data and evaluation: Source data are split 80/20, supporting probe training and pseudo-label sampling on the training split and auxiliary evaluation on held-out data.
3 Results
AURCC closely recovers target-performance rankings across inter-hospital scenarios, while its advantage becomes clearer as held-out source labels shrink. The MP ranking is stable across region counts and outperforms source-AUC under the smallest tested sample sizes.
- 3.1 Label-free model ranking: ρ = 0.943 (p < 0.05) for AURCC in every MP scenario and K value, indicating stable rankings across the number of probability regions.The comparison uses Spearman correlation with ground-truth AUCtarget.
- 3.1 Label-free model ranking: AURCC orders models consistently with true target AUC, correctly placing the general-domain baseline below the medical models in all three scenarios.Intermediate-model ordering is not perfect: one pair is locally swapped in each scenario.
- 3.2 Sensitivity to held-out source size: At N = 50, AURCC outperforms source-AUC in all three scenarios, including ρ = 0.683 versus ρ = 0.605 for MIMIC.For NIH at N = 100, AURCC achieves ρ = 0.911 versus ρ = 0.749 for source AUC.
- 3.2 Sensitivity to held-out source size: AURCC consistently outperforms source-AUC for NIH and CheXpert at every tested held-out-source size, with wider gaps as the source shrinks.The sensitivity analysis averages Spearman ρ over 20 stratified subsampling repetitions.
4 Discussion
The study proposes AURCC as a label-free criterion for selecting foundation models under distribution shift, using unlabeled target images when source labels are scarce. Its demonstrated scope is chest-X-ray pneumonia classification with PadChest as the single target domain.
- 4 Discussion: AURCC ranks foundation models using unlabeled target images and correlates strongly with ground-truth target performance across three inter-hospital scenarios toward PadChest.In MP, it is competitive with or superior to source-AUC, while in ZS the two criteria perform comparably.
- 4 Discussion: AURCC outperforms source-AUC at every tested fraction for NIH and CheXpert and produces better rankings for N = 50 in all scenarios.The discussion attributes this signal to pseudo-label discrepancy across probability regions and target-distribution structure.
- 4 Discussion: Generalization to other tasks, modalities, and multi-class settings remains to be validated because evaluation is restricted to pneumonia classification and PadChest.The three source datasets provide independent shift scenarios, but only one target domain is evaluated.
- 4 Discussion: The pipeline uses embeddings and runs on CPU, enabling local model ranking on commodity hardware in resource-constrained settings.The paper frames this as reducing dependence on scarce target labels during model selection.