PaperlayerView exact revision on arXiv

Source-linked AI summary

CRS-Bench: A Reference-Relative Reliability Benchmark for Medical Image Encoders

Xingtao Lin, Hangqi Ren, Caiwan Sun, You Chen

arXiv:2608.22059v1eess.IVcs.AIcs.CV

TL;DR

Medical encoder selection cannot rely on clean-test discrimination alone because encoders with similar AUROC may differ in calibration, label efficiency, and robustness. CRS-Bench evaluates 15 encoder families across three clinical domains using four reliability dimensions and a reference-relative Clinical Reliability Score. The benchmark finds that AUROC and CRS are not decision-equivalent and supports a stable leading tier rather than a single resolved winner.

  • Problem

    Clean-test discrimination alone does not characterize encoder reliability because calibration, label efficiency, and robustness can differ despite similar AUROC.

  • Method

    CRS-Bench evaluates 15 pretrained encoder families across three clinical domains under matched conditions and summarizes four reliability dimensions with a Pareto-aware, reference-relative CRS.

  • Results

    21 of 105 pairwise encoder orderings reverse between AUROC and CRS, with mean absolute rank displacement of 1.87; PanDerm, MedSigLIP, and MedGemma form a stable leading tier.

  • Takeaways & Limitations

    Encoder selection should use reliability profiles alongside clean-test discrimination rather than treating AUROC as decision-equivalent to multi-axis reliability.

  • Takeaways & Limitations

    The benchmark is a comparative summary rather than a clinical-utility endpoint and excludes prospective validation, subgroup analysis, operating-point selection, and human–AI interaction.

Abstract

from arXiv · show

Pretrained image encoders are central to medical image classification, where expert annotation is costly and task-specific cohorts are often limited. As the model space expands from general-purpose to broad-medical and specialty-specific encoders, selecting the representation becomes a substantive modeling decision. Clean-test discrimination alone is insufficient for this purpose: encoders with similar AUROC can differ in calibration, label efficiency, and stability under acquisition perturbations or distribution shift. We introduce CRS-Bench, a controlled benchmark for multi-objective medical encoder selection. CRS-Bench evaluates 15 pretrained encoder families across dermatology, ophthalmology, and radiology using ISIC 2019, APTOS 2019, and CheXpert, with CheXpert-to-MIMIC-CXR as an observed institutional shift, yielding 17,575 controlled run records and 3,515 seed-aggregated metric rows. Each encoder is characterized along four operational reliability dimensions: discrimination, calibration, label efficiency, and robustness. We summarize these dimensions using the Clinical Reliability Score (CRS), a Pareto-aware, reference-relative score combining dominance, profile balance, and worst-axis performance. AUROC and CRS are positively associated but not decision-equivalent: 21 of 105 pairwise orderings reverse, with a mean absolute rank displacement of 1.87. Paired-seed bootstrap analysis identifies PanDerm, MedSigLIP, and MedGemma as a stable leading reliability tier rather than a statistically resolved single leader. CRS-Bench provides a controlled framework for selecting medical image encoders from multi-axis reliability profiles rather than clean-test AUROC alone.

1 Introduction

CRS-Bench addresses encoder selection as a multi-objective reliability problem because clean-test discrimination alone does not capture calibration, label efficiency, or robustness. It evaluates 15 encoder families and summarizes these dimensions with a reference-relative Clinical Reliability Score.

  • Motivation: Encoder selection is difficult because the available models span general-purpose, broad-medical, and specialty-specific pretraining regimes.The benchmark covers dermatology, ophthalmology, and radiology applications.
  • Motivation: Similar AUROC can conceal material differences in calibration, supervision demand, and stability under perturbation or distribution shift.Thus, a clean-split ranking may not reflect broader reliability.
  • Benchmark: CRS-Bench evaluates 15 pretrained encoder families across three clinical domains under matched data, perturbation, seed, and tuning conditions.The study produces 17,575 controlled run records and 3,515 seed-aggregated metric rows.
  • Benchmark: The benchmark characterizes each encoder using discrimination, calibration, label efficiency, and robustness, then combines them in the Clinical Reliability Score.CRS uses Pareto dominance, profile balance, and worst-axis performance.
  • Results: 21 of 105 pairwise encoder orderings reverse between AUROC and CRS, with mean absolute rank displacement of 1.87.Bootstrap analysis identifies PanDerm, MedSigLIP, and MedGemma as a stable leading tier rather than a statistically resolved single leader.
  • Contributions: The paper contributes a controlled multidomain benchmark, a comparable reference-relative score, criterion validation, and analyses of encoder behavior.These analyses cover specialization, calibration, feature depth, label scarcity, perturbations, prompting, and adaptation.

2 Related Work

Prior work benchmarks transfer, corruption robustness, and selected medical properties, but typically emphasizes narrower model sets or clean discrimination. CRS-Bench instead evaluates four reliability dimensions under one matched clinical protocol against an explicit reference panel.

  • Encoder landscape: General-purpose, broad-medical, and specialty encoders use diverse supervised, contrastive, self-supervised, biomedical, and domain-targeted pretraining strategies.The 15-family panel includes six generalist, four broad-medical or multimodal, and five specialty encoders.
  • Prior comparisons: Existing comparisons generally cover narrower model sets and emphasize clean discrimination or task-specific adaptation.This leaves limited direct guidance for multidimensional reliability-based selection.
  • Benchmarking reliability: VTAB and ELEVATER measure transfer, ImageNet-C standardizes corruption robustness, and medical benchmarks target selected properties such as few-shot adaptation, breadth, radiology, or fairness.Calibration and uncertainty studies separately show that discrimination does not guarantee reliable probabilities under shift.
  • CRS-Bench: CRS-Bench measures discrimination, calibration, label efficiency, and robustness under one controlled clinical protocol with an explicit reference panel.This design directly supports selection using a multidimensional reliability profile.

3 The CRS-Bench Benchmark

CRS-Bench defines encoder selection as comparison of pretrained representations across multiple observable reliability properties under matched downstream conditions. Its frozen-probing benchmark measures four dimensions, evaluates robustness and uncertainty, and aggregates them with a fixed reference-relative CRS.

  • Benchmark design: CRS-Bench separates measurement, score validation, and behavioral interpretation within a controlled encoder-selection workflow.The protocol covers matched probing, label scarcity, perturbations, institutional shift, score stability, uncertainty, and ranking transfer.
  • Selection problem: The benchmark compares pretrained representations jointly across discrimination, calibration, label efficiency, and robustness under matched downstream conditions.CRS summarizes the measured profile rather than estimating universal clinical utility.
  • Encoder panel: The 15-family panel spans general-purpose, broad-medical or multimodal, and specialty-specific encoders, with 43 of 45 encoder–dataset cells frozen.BioViL and CheXzero lack APTOS cells because their released interfaces are chest-radiology specific.
  • Evaluation protocol: Frozen probing fixes the downstream decision rule to isolate pretrained representation quality, while LoRA separately tests persistence after task-specific adaptation.The primary probe and optimization settings are matched across encoders.
  • Reliability dimensions: Label efficiency measures retention of each encoder’s own full-label performance rather than absolute low-label attainment.The full-label value appears only in the denominator, distinguishing relative retention from endpoint-favoring AULC.
  • Reliability dimensions: Robustness combines clean-to-perturbed performance drops and, for CheXpert, an observed MIMIC-CXR institutional shift.A corruption-only variant addresses the fact that only CheXpert contributes an observed institutional shift.
  • Clinical Reliability Score: CRS combines Pareto dominance, proximity to an ideal profile, and weakest-axis performance after orienting all dimensions so larger is better.The components summarize complementary aspects without treating raw metrics as directly commensurate.
  • Reference standard: The fixed 15-family reference panel and normalization anchors keep previously reported aggregate CRS values unchanged when future encoders are added.Default component weights prioritize Pareto-consistent comparative evidence.

4 Experimental Setup

The experimental setup uses three primary datasets and MIMIC-CXR as an external institutional-shift target, spanning distinct medical imaging modalities and acquisition pipelines.

  • Datasets: CheXpert contains 191,229 chest radiographs with 14 multilabel targets, ISIC 2019 contains 25,331 dermoscopy images across 8 classes, and APTOS 2019 contains 3,662 fundus photographs across 5 ordinal grades.MIMIC-CXR serves as the institutional-shift target for CheXpert-trained probes.

5 Results

CRS-Bench evaluates encoder reliability across multiple dimensions and shows that rankings remain broadly stable under uncertainty and reference-panel changes. The results identify distinct reliability profiles rather than a single accuracy-based ordering.

  • Results: The benchmark combines four-axis profiles with bootstrap, reference-stability, scalarization, transfer, and estimator-dependence analyses across controlled experiments.The evaluation includes 10,000 bootstrap replicates, 415 panel perturbations, and 30 insertion trials.
  • Multi-Axis Profiles and Sampling Uncertainty: PanDerm pairs competitive discrimination with the strongest robustness, while MedSigLIP and MedGemma concentrate more advantage in discrimination and low-label retention.BiomedCLIP is comparatively strong in calibration, and CheXzero combines lower discrimination with favorable calibration and robustness.
  • Multi-Axis Profiles and Sampling Uncertainty: 10,000 paired-seed bootstrap replicates support PanDerm, MedSigLIP, and MedGemma as a stable leading tier rather than a statistically resolved single leader.PanDerm and MedSigLIP attain rank 1 with probabilities 0.664 and 0.336, respectively; each of the three models has P(top-3) = 1.000.
  • Reference Stability and Incremental Extensibility: 415 panel perturbations yielded Kendall τ values of 0.975, 0.926, and 0.891, with Top-3 overlap of 1.000, 0.985, and 0.958 as panels were reduced.Agreement weakened gradually as the reference panel was thinned, but the leading tier remained stable.
  • Reference Stability and Incremental Extensibility: Fixed-reference insertion produced exactly zero displacement of existing scores, whereas cohort renormalization moved scores by mean, median, and maximum values of 0.053, 0.042, and 0.167.The fixed-reference rule preserves previously published scores when future encoders are added.

5.3 Construct Informativeness and Scalarization Stability

CRS is associated with AUROC but incorporates information that changes encoder decisions. Its rankings remain stable across substantial weighting perturbations, while the dimensions are not interchangeable.

  • Construct Informativeness: Relative label retention reduces AUROC-related correlations to 0.339 at encoder level and −0.342 across 43 cells, unlike AULC.AULC label efficiency is nearly a re-expression of full-label AUROC, whereas relative retention measures preservation under reduced supervision.
  • Construct Informativeness: The remaining reliability dimensions are dependent but non-interchangeable, with the largest absolute correlation equal to 0.625 at encoder level and 0.460 at cell level.Discrimination versus oriented calibration ranges from 0.764 on APTOS to −0.279 on CheXpert and −0.029 on ISIC.
  • Scalarization Stability: Across 1,000 axis- and component-weight perturbations, mean Kendall agreement was 0.986 and 0.985, with Top-3 unchanged in every draw.Alternative axis weights produced τ = 1.000, supporting leading-set stability without implying uniqueness or optimality of the default coefficients.
  • Construct Informativeness: 21 of 105 pairwise encoder choices reverse between AUROC and CRS, with mean absolute displacement of 1.87 ranks.The association remains positive, with τ = 0.600 and ρ = 0.811.

5.5 Transfer of the Ranking under Adaptation

The benchmark tests whether a reference-relative reliability ranking transfers under adaptation and held-out evaluation. Transfer is partial and task-dependent, so CRS is bounded as a suite-relative selection summary rather than a universal predictor.

  • Transfer under Adaptation: LoRA changes the representation itself, making agreement with frozen probing a test of ranking persistence after task-specific optimization.The matched adaptation arm recomputes discrimination, calibration, label efficiency, robustness, and CRS on the same evaluation grid.
  • Held-Out Transfer: Leave-one-dataset-out CRS correlations with held-out rankings were 0.550, 0.709, and 0.271 for ISIC, APTOS, and CheXpert, respectively.Top-3 overlap was 1.000, 0.333, and 0.667, and the held-out winner was recovered in no fold.
  • Held-Out Transfer: Removing MIMIC-CXR from CRS gave ρ = 0.529 for predicting MIMIC AUROC, compared with ρ = 0.864 for CheXpert AUROC.CheXpert AUROC was more directly aligned because the target was discrimination on a closely related radiographic dataset.
  • Decision Consequences beyond AUROC: Figure 3 reports aggregate AUROC–CRS association and rank displacement, showing that 21 of 105 pairwise orderings reverse with mean displacement 1.87.These reversals contextualize transfer by showing that CRS captures a ranking distinct from clean discrimination.
  • External Validity: CRS is bounded as a multi-objective selection summary under a declared suite rather than a universal predictor of unseen-task AUROC.Its external validity is therefore tied to the evaluated reference panel and clinical suite.

5.7 Sensitivity to Metric Representation

Sensitivity analyses show that the leading CRS conclusions persist across alternative metric representations and corruption-only robustness, while absolute scores can shift. Institutional-shift analysis also reveals heterogeneous preservation across encoders.

  • Metric and robustness sensitivity: τ = 1.000 with adaptive ECE and τ = 0.981 with Brier preserve the CRS ranking and leading tier.These results indicate that the primary conclusion is not determined by fixed-width ECE binning.
  • Metric and robustness sensitivity: τ = 0.923/0.949 for QWK/ordinal-MAE on APTOS preserves the Top-3 under ordinal-task metrics.
  • Metric and robustness sensitivity: τ = 0.962 under matched corruption-only robustness preserves the Top-3/Top-5, although mean CRS moves by 0.031.
  • Institutional shift: 0.717 versus 0.540 macro-AUROC is retained by medically pretrained versus generalist encoders under CheXpert→MIMIC-CXR shift.Welch p = 0.0029, but preservation varies within the medical group.
  • Institutional shift: MedSigLIP/MedGemma show relative AUROC gaps of 0.040/0.039 under the observed institutional shift.The shift exposes preservation differences that clean source-domain AUROC does not reveal.

5.9 Calibration Repair and Corruption-Induced Miscalibration

Post-hoc calibration substantially improves probability quality without materially changing discrimination, while corruption can degrade calibration even when class ranking remains useful. These are complementary reliability failure and repair modes.

  • Calibration repair: 54–77% ECE reduction occurs across six primary tests while mean ∆AUROC is approximately +0.001.Temperature, vector, isotonic, and Dirichlet scaling produce this improvement while preserving AUROC.
  • Calibration repair: 0.150→0.034 and 0.121→0.028 are the ECE changes for DINOv2/APTOS and MedSigLIP/APTOS.
  • Corruption-induced miscalibration: 0.128 mean ECE increase, with a worst value of 0.418, occurs for APTOS LLaVA-Med under corruption.
  • Corruption-induced miscalibration: 0.113/0.094 ECE increases affect MedGemma/MedSigLIP on ISIC, while CheXpert remains comparatively stable with a maximum of 0.038.
  • Corruption-induced miscalibration: r = 0.63 links discrimination loss and calibration loss across 43 cells, showing only moderate association.An AUROC-only robustness measure would not capture this calibration failure mode.

5.10 Representation Depth and Supervision Efficiency

Representation depth and supervision efficiency materially affect encoder reliability beyond final-layer clean AUROC. Medical pretraining is especially advantageous when labels are scarce, while prompting introduces additional variation.

  • Representation depth: 5/8 depth sweeps favor intermediate features rather than final blocks.
  • Representation depth: 1.3–1.6 AUROC points are gained by RAD-DINO around 50% depth, while DINOv2 prefers 75%.Shallower MedSigLIP/MedGemma features can also lower ECE.
  • Supervision efficiency: 0.884 AUROC is reached by MedGemma on APTOS with 1% of labels.
  • Supervision efficiency: 0.819/0.815/0.806 ISIC AUROC is reached by MedSigLIP/MedGemma/PanDerm at 1% labels.
  • Prompting versus feature probing: 74.7/51.0/82.0% versus 49.3/17.8/79.5% are MedGemma and prompted LLaVA-Med results on APTOS/ISIC/CheXpert.MedGemma remains below frozen-feature probing, and one APTOS chain-of-thought setting parses only 64.6% of cases.

6 Discussion and Conclusion

CRS-Bench frames encoder selection as a controlled, multi-objective comparison in which reliability profiles complement clean discrimination. The ranking supports a stable leading tier, but CRS remains a benchmark-level comparative summary rather than a clinical-utility endpoint.

  • Discussion and conclusion: 21 of 105 AUROC–CRS pairwise choices reverse, supporting profile-specific trade-offs rather than a universal winner.
  • Discussion and conclusion: PanDerm, MedSigLIP, and MedGemma form a stable leading tier rather than a statistically resolved single leader.
  • Discussion and conclusion: Reference-panel perturbations test score sensitivity, while fixed-anchor insertion keeps prior scores invariant as candidates are added.
  • Discussion and conclusion: Specialty pretraining, calibration, corruption, representation depth, and prompting add reliability variation beyond final-layer clean AUROC.
  • Scope and limitations: CRS is a benchmark-level comparative summary, not a clinical-utility endpoint, with prospective validation, subgroup analysis, operating-point selection, and human–AI interaction outside the protocol.
Loading 2608.22059v1…