Source-linked AI summary
Can You Trust Frozen Hematology Foundation Models under Acquisition Shift?
Jai Kumar Sharma, Peeyush Tapadiya
TL;DR
Frozen hematology encoders need evaluation beyond saturated in-domain accuracy because deployment spans acquisition domains and class priors. The paper audits 15 frozen encoders for cross-domain accuracy, calibration, exposure, and adaptation, finding that rankings and confidence often fail to transfer. CBR improves target-prior scenario means and partly improves calibration, but exposure confounding and residual exceptions remain.
Problem
Standard in-domain accuracy comparisons do not establish robustness, calibration transfer, pretraining independence, or safe adaptation under realistic WBC class-prior shift.
Method
The study evaluates frozen encoder features with source-fitted linear and 1-NN probes across four acquisition domains, while auditing exposure and testing label-free adaptation under balanced and shifted priors.
Results
Cross-domain accuracy and calibration degrade, rankings reorder, and class-balanced re-standardization improves all evaluated target-prior scenario means while only partially correcting calibration.
Takeaways & Limitations
Hematology FM benchmarks should jointly audit accuracy, calibration, pretraining exposure, and class-prior robustness rather than rely on in-domain accuracy.
Takeaways & Limitations
The benchmark has no leakage-free DinoBloom target, and CBR remains transductive, pseudo-label-dependent, and subject to encoder-level exceptions under extreme skew.
Abstract
from arXiv · showhide
Frozen hematology foundation-model (FM) embeddings reach near-saturated in-domain white-blood-cell (WBC) accuracy, but clinical deployment demands reliability across scanners, sites, stains and preparation pipelines. We audit 15 frozen encoders (hematology, pathology, and general vision) across four public single-cell acquisition domains along two axes: accuracy robustness and calibration. In-domain linear-probe macro-F1 is saturated (0.98-0.997), yet cross-dataset macro-F1 drops 34-72% and rankings re-order: DinoBloom-L, the in-domain best, falls to 10th of 15 on the most-shifted target (MLL23) at the benchmark's shared 224-px input, behind RedDino and several general and pathology encoders. Rank transfer is probe-dependent: 1-NN retrieval is more stable on average than a source-fitted linear head (median $ρ$ 0.65 vs 0.45), but neither probe universally predicts target robustness. Calibration also collapses: source-trained probes are nearly calibrated in-domain (expected calibration error, ECE, 0.004) but confidently wrong off-domain (ECE 0.35), and source-fitted temperature scaling transfers poorly. We further audit pretraining exposure and identify MLL23 as DinoBloom's internal cohort; because DinoBloom's only held-out dataset is also our source domain, this benchmark cannot isolate exposure from scanner-associated shift. Label-free adaptation and marginal-entropy-based model selection appear safe under balanced evaluation but fail under realistic WBC class-prior shift. Class-Balanced Re-standardization (CBR), a training-free pseudo-label-balanced feature normalization, improves all evaluated target-prior scenario means and partially improves calibration, although encoder-level exceptions and residual miscalibration remain. Hematology FM benchmarks must therefore jointly audit accuracy, calibration, exposure, and class-prior robustness.
1 Introduction
Standard in-domain evaluation can overstate the reliability of frozen hematology encoders under acquisition-domain shift. This audit therefore examines cross-domain accuracy, calibration, pretraining exposure, and class-prior robustness.
- Cross-domain deployment can make a classifier systematically and confidently wrong across scanners, laboratories, stains, and preparation protocols.
- In-domain frozen-encoder rankings do not establish whether accuracy or calibration transfers across acquisition domains.
- The audit evaluates 15 frozen encoders across cross-domain accuracy, calibration, exposure, and label-free adaptation under class-prior shift.
2 Related work
Prior work motivates evaluating scanner sensitivity, calibration transfer, pretraining exposure, and class-prior robustness together for frozen single-cell hematology embeddings. The paper extends existing adaptation and selection methods to this setting.
- Existing WBC studies show supervised classifiers degrade across imaging conditions, while this work tests frozen embeddings under cross-dataset acquisition shift and class imbalance.
- Pathology foundation-model research establishes scanner sensitivity but does not address frozen single-cell hematology embeddings, calibration transfer, exposure ambiguity, or class-prior stress tests.
- The paper ports calibration metrics and source-to-target temperature-transfer testing to frozen hematology foundation models while auditing pretraining overlap.
- CBR is presented as a training-free, single-batch, pseudo-label-balanced instance of class-balanced feature-statistics adaptation.
3 Method
The method freezes encoders, fits source-domain probes, and evaluates zero-shot transfer across acquisition domains. It combines accuracy and rank metrics with label-shift stress tests and pseudo-label-balanced adaptation.
- Each encoder’s frozen features are source-standardized and evaluated with both an L2-regularized logistic probe and a 1-NN probe on held-out domains.
- Macro-F1 is primary, complemented by rank correlation, effective robustness, relative gap, per-class recall, and bootstrap 95% confidence intervals.
- Label-free methods are tested on balanced targets and realistic skewed priors that mimic neutrophil-dominated deployment batches.
- CBR re-standardizes target embeddings using equal-weighted pseudo-label class means and pooled within-class standard deviations.
4 Experimental setup
The experiment uses one source and three zero-shot targets spanning distinct acquisition systems, with 15 frozen encoders evaluated at a shared 224 × 224 input. It also audits pretraining exposure and uses image-level splits.
- The source is Acevedo via BloodMNIST@224, containing 10,298 WBC images acquired with CellaVision DM96.
- Targets are MLL23, Matek-LMU, and Raabin, representing Metafer, M8, and smartphone-plus-Olympus acquisition settings.
- The benchmark evaluates 15 frozen encoders spanning hematology, pathology, general self-supervised or vision-language models, ImageNet models, and a supervised ResNet-18 baseline.
- DinoBloom’s internal LabAnonymous cohort corresponds to MLL23, leaving Acevedo as its only held-out dataset and making its other targets in-pretraining domains.
5 Results
Across acquisition shifts, saturated in-domain accuracy fails to predict encoder robustness, calibration collapses, and class-prior-aware adaptation is safer than global standardization.
- Accuracy robustness: In-domain macro-F1 is saturated at 0.98–0.997, yet cross-dataset macro-F1 drops 34–72% and rankings reorder substantially.DinoBloom-L falls from in-domain #1 to 10th of 15 on MLL23, 0.15 macro-F1 behind RedDino.
- Accuracy robustness: 1-NN rank transfer is more stable than linear-probe transfer on average, with median Spearman ρ 0.65 versus 0.45, but remains unreliable in some shifts.Acevedo→Raabin reaches only ρ 0.34, and MLL23 is hardest for both probes.
- Exposure audit: Pretraining exposure remains unresolved because MLL23 is in DinoBloom’s training corpus and Acevedo is the only held-out dataset, preventing clean isolation of exposure from scanner shift.The authors report exposure status without claiming or excluding a DinoBloom-specific leakage effect.
- Calibration: Off-domain calibration collapses: ECE rises from 0.004 in-domain to 0.35, while source-fitted temperature scaling transfers poorly.CBR plus source temperature scaling partially helps, but only oracle target temperature scaling substantially improves ECE, and it requires target labels.
- Adaptation and selection: Marginal-prediction entropy appears effective under balanced sampling but incurs 0.25–0.37 selection regret under skewed priors, leaving no reliable label-free deployment selector.Balanced evaluation therefore does not establish safe model selection for realistic WBC differentials.
- Adaptation and selection: Under realistic class-prior shift, global target standardization hurts, whereas CBR improves all 18 target-prior scenario means.CBR’s mean gain is +0.059, with 0/18 scenarios harmed; encoder-level exceptions and small-batch noise remain.
6 Discussion and limitations
The benchmark has important scope and interpretation limits, including exposure ambiguity, probe-dependent selection, transductive adaptation, class-prior exceptions, and restricted label and stain coverage.
- Scope boundaries: The five-class WBC intersection and one-stain-family-per-dataset design constrain how broadly the findings generalize.
- Exposure ambiguity: No leakage-free DinoBloom target is available, so exposure effects cannot be cleanly separated from scanner-associated shift.
- Selection limits: 1-NN rank transfer is more stable on average but remains not universally reliable for model selection.
- Adaptation limits: CBR is transductive and pseudo-label-dependent, reaches approximately 61% of the oracle, and has per-encoder exceptions under extreme skew.
7 Conclusion
The paper concludes that in-domain accuracy is insufficient for selecting robust frozen hematology encoders under scanner-associated shift. It recommends jointly auditing accuracy, calibration, exposure, and class-prior robustness, with class-balanced re-standardization as a partial mitigation.
- In-domain linear-probe accuracy does not predict which frozen encoder is robust under scanner-associated shift.
- Rank transfer is probe-dependent, while both target accuracy and calibration degrade and benchmark exposure remains ambiguous.
- Class-balanced re-standardization provides a partial mitigation, motivating joint audits of accuracy, calibration, exposure, and class-prior robustness.
Supplementary Material
The supplement provides controls and supporting analyses for scanner shift, rank instability, exposure, class-specific failures, calibration, adaptation, resolution, and probe sensitivity. Its results reinforce that label-free selection can fail under skew and that CBR only partially mitigates this failure.
- Scanner shift is measurable: Scanner decodability, subspace, and projection analyses support the claim that scanner shift is measurable.
- Clean rank is unstable: Selection-regret, paired-bootstrap, and source-target-matrix analyses support unstable clean rankings.
- Exposure and class-specific failures: A controlled exposure experiment supports auditing pretraining exposure, while class-specific analyses localize the MLL23 collapse.
- Calibration fails off-domain: Calibration analyses cover ECE, adaptive-ECE, NLL, Brier, and temperature scaling.
- Adaptation and selection: Marginal-entropy and agreement-style selection fails under skew, whereas CBR mitigates skewed-prior test-time-adaptation failure.
- Resolution and probe controls: Resolution and head-sensitivity controls show that the rank flip is not explained by resolution and that probe dependence is not a single-head artifact.
A.1 Scanner identity is linearly decodable from frozen embeddings
Frozen embeddings retain highly decodable scanner identity even after equalizing WBC-class composition across domains. This scanner-associated variation is not itself a reliable predictor of encoder robustness, but projecting scanner directions can improve cross-target performance.
- 0.990–1.000 scanner accuracy, with mean 0.997, shows near-perfect linear decodability across the class-balanced domains.
- 0.782–0.971 WBC5 accuracy, with mean 0.891, is lower than scanner decodability under the same auxiliary task.
- 0.990 mean within-class scanner accuracy shows that scanner identity remains decodable even when WBC class is held constant.
- A mean permutation-baseline accuracy of 0.25 confirms chance performance for shuffled domain labels.
- Approximately −0.09 Spearman ρ indicates no significant relationship between scanner decodability and mean cross-dataset performance drop.
A.2 Class–scanner subspace analysis
Global class–scanner overlap is small and does not explain encoder robustness; remaining variation is more closely associated with target-shift magnitude and encoder-specific class geometry.
- 0.03–0.16 entanglement does not correlate with cross-dataset drop (Spearman ρ=−0.03, p=0.94).
- Neither saturated scanner decodability nor global class–scanner overlap explains which encoder is most robust.The remaining variation appears more closely related to target-shift magnitude and encoder-specific class geometry than to a simple geometric feature-space property.
A.3 Scanner-direction projection as a diagnostic intervention
Projecting away top-3 label-free scanner directions improves cross-target macro-F1 without in-domain cost, supporting a scanner-associated component in source-probe failure.
- +0.063 mean crosstarget macro-F1 improvement occurs at zero in-domain cost (in-domain ∆= +0.000).The intervention projects features onto the orthogonal complement of top-3 between-domain scanner directions, retrains the source linear probe, and re-evaluates cross-scanner.
- General encoders benefit most, while DinoBloom improves only +0.01–0.05 and BiomedCLIP approximately 0.The weaker DinoBloom effect indicates its MLL23 failure is not fully captured by a removable global scanner-mean axis and reflects specific learned class geometry.
- Scanner-subspace projection is a diagnostic first-moment intervention, not the deployment method used in the main paper.It supports the low-order-shift interpretation and complements the CBR analysis.
A.4 Effective-robustness residuals are uninformative under saturated in-domain accuracy
Saturated in-domain accuracy makes effective-robustness residuals uninformative, while cross-domain rankings, selection, adaptation, and calibration reveal substantial deployment failures and caveats.
- 0.98–0.997 near-identical in-domain accuracy makes effective-robustness residuals ill-conditioned and unable to represent meaningful robustness above the line.DinoBloom-family residuals remain approximately −0.08 to −0.11 across targets, re-encoding cross-domain rank.
- Up to about 15 macro-F1 points are lost when selecting by in-domain accuracy on the most-shifted target.This regret is computed on balanced targets; realistic class imbalance changes absolute values, and label-free alternatives also fail under evaluated shifts.
- Marginal-entropy selection fails under skewed priors: regret rises to 0.25–0.37, while clean accuracy performs better across K=8–400.Under label shift, rewarding prediction diversity penalizes robust encoders that should produce skewed marginals.
- CBR improves scenario means across all 18 target-prior scenarios by +0.059, with 0/18 harmed, but 29/270 encoder–scenario cells remain negative.The negative cells concentrate in the two most skewed priors and the DinoBloom family; CBR is therefore positive on average, not uniformly per encoder.