PaperlayerView exact revision on arXiv

Source-linked AI summary

Scanner-Induced Domain Shifts Undermine the Robustness of Pathology Foundation Models

Erik Thiringer, Fredrik K. Gustafsson, Kajsa Ledesma Eriksson, Mattias Rantalainen

arXiv:2601.04163v1eess.IVcs.CVcs.LG

TL;DR

Scanner-induced shifts in whole-slide images remain insufficiently characterized for pathology foundation models, especially beyond AUC. The study evaluates 14 frozen feature extractors using repeated scans of 384 breast-cancer WSIs across five devices, combining embedding and supervised analyses. It finds that current PFMs are not reliably scanner-robust: scanner variation distorts embeddings and calibration even when AUC remains stable.

  • Problem

    Evidence is limited on whether PFMs remain robust to scanner-induced domain shifts, despite the importance of robustness beyond standard supervised performance for clinical deployment.

  • Method

    The study evaluates 14 frozen feature extractors on a controlled multiscanner dataset of 384 breast-cancer WSIs scanned on five devices using embedding, prediction-consistency, and calibration analyses.

  • Results

    Current PFMs are not invariant to scanner-induced shifts: scanner variability distorts embedding geometry and downstream calibration, while AUC often remains strong.

  • Takeaways & Limitations

    PFM development and evaluation should explicitly assess embedding stability and calibration across acquisition devices rather than relying only on accuracy-centric benchmarks.

  • Takeaways & Limitations

    The study focuses on breast-cancer resections, does not spatially register tiles across scanners, and uses ABMIL for downstream evaluation.

Abstract

from arXiv · show

Pathology foundation models (PFMs) have become central to computational pathology, aiming to offer general encoders for feature extraction from whole-slide images (WSIs). Despite strong benchmark performance, PFM robustness to real-world technical domain shifts, such as variability from whole-slide scanner devices, remains poorly understood. We systematically evaluated the robustness of 14 PFMs to scanner-induced variability, including state-of-the-art models, earlier self-supervised models, and a baseline trained on natural images. Using a multiscanner dataset of 384 breast cancer WSIs scanned on five devices, we isolated scanner effects independently from biological and laboratory confounders. Robustness is assessed via complementary unsupervised embedding analyses and a set of clinicopathological supervised prediction tasks. Our results demonstrate that current PFMs are not invariant to scanner-induced domain shifts. Most models encode pronounced scanner-specific variability in their embedding spaces. While AUC often remains stable, this masks a critical failure mode: scanner variability systematically alters the embedding space and impacts calibration of downstream model predictions, resulting in scanner-dependent bias that can impact reliability in clinical use cases. We further show that robustness is not a simple function of training data scale, model size, or model recency. None of the models provided reliable robustness against scanner-induced variability. While the models trained on the most diverse data, here represented by vision-language models, appear to have an advantage with respect to robustness, they underperformed on downstream supervised tasks. We conclude that development and evaluation of PFMs requires moving beyond accuracy-centric benchmarks toward explicit evaluation and optimisation of embedding stability and calibration under realistic acquisition variability.

1. Introduction

PFMs are increasingly used as frozen encoders, but scanner variability creates a clinically relevant robustness gap that standard performance metrics may miss. This study isolates scanner effects and evaluates embedding geometry, downstream predictions, consistency, and calibration across diverse feature extractors.

  • Motivation: Clinical histopathology images vary systematically across laboratories and scanner devices, creating domain shifts that can challenge model deployment.Scanner differences include colour response, resolution, compression, and image post-processing.
  • Motivation: Stable AUC does not ensure robustness because scanner shifts can alter feature representations, predicted probabilities, and decision thresholds.These effects may remain hidden in standard benchmarking protocols.
  • Study objective: The study uses identical tissue specimens scanned on multiple devices to isolate scanner-induced variability from biological and laboratory confounders.The controlled benchmark combines unsupervised embedding analyses with supervised downstream evaluations.
  • Study objective: Robustness is assessed across 14 frozen feature extractors, including contemporary PFMs, earlier models, vision-language models, and a natural-image ResNet baseline.The benchmark spans multiple generations and design paradigms.
  • Main findings: The study reports pronounced scanner-specific variability in most models and finds that robustness is not a simple function of training data scale, model size, or recency.Vision-language models appear more robust but did not rank among the top supervised-task performers.

2. Related Work

Prior work established scanner sensitivity but generally lacked controlled comparisons of identical WSIs across scanners and comprehensive PFM evaluation. This study extends that literature by assessing intrinsic embedding stability and clinically relevant calibration alongside predictive performance.

  • Existing gap: Earlier benchmarking studies did not directly compare the same WSIs scanned on different whole-slide scanners.Scanner-related colour variation can be substantial in practice.
  • Concurrent work: Concurrent studies confirmed scanner-induced shifts, but some focused on training-from-scratch segmentation or downstream mitigation using multiscanner data.Ryu et al. used patches from 48 WSIs, while Carloni et al. proposed a downstream loss requiring curated multiscanner data.
  • Concurrent work: Chai et al. evaluated scanner and staining variability mainly through accuracy and found partial mitigation from adding stain-varied slides during downstream training.Their evaluation included six PFMs and a ResNet-50 baseline.
  • Positioning: This work evaluates out-of-the-box embedding stability using geometric consistency metrics rather than applying a downstream robustness loss.The framework is more granular and clinically oriented than prior accuracy-focused evaluations.
  • Positioning: Across 14 feature extractors, scanner variability can shift predicted probabilities even when AUC remains stable, producing miscalibration and compromised clinical decision thresholds.The result identifies a failure mode overlooked by accuracy-centric benchmarking.

3. Methods

The study combines controlled multiscanner data, standardized preprocessing, frozen feature extractors, and supervised and unsupervised evaluations. Its dataset links repeated scans of the same breast-cancer slides with clinical labels and a separate training cohort.

  • Datasets: CHIME Multiscanner contains 384 H&E breast-tumour slides from 384 patients, each scanned on five devices, yielding 1920 digitized WSIs.The slides came from a single medical center and three scanner manufacturers.
  • Datasets: CHIME supports unsupervised embedding analysis and supervised prediction of ER, PR, HER2, and Nottingham histological grade.Clinical labels are available for most patients.
  • Datasets: TCGA-BRCA provides the supervised training cohort, with clinical factors overlapping those evaluated in CHIME Multiscanner.The selected cohort contains 825 primary breast-cancer samples.
  • Preprocessing: All WSIs undergo tissue segmentation and extraction of non-overlapping 256×256-pixel tiles at 0.435µm/pixel, standardizing spatial resolution across scanners.Blur filtering is applied to TCGA-BRCA but not CHIME to avoid scanner-dependent filtering in the evaluation data.
  • Feature extractors: The benchmark evaluates 14 frozen tile-level encoders: 13 PFMs and a natural-image-trained ResNet baseline.The models include large ViTs, earlier CNN-based models, vision-language PFMs, and a distilled robustness-focused model.
  • Feature extractors: The feature-extractor set spans models trained on datasets ranging from tens of thousands to more than 3 million WSIs and includes multiple architectures and pretraining paradigms.Table 2 records release date, learning approach, architecture, embedding dimensionality, magnification, training-set size, and data source.

4. Evaluation Framework

The evaluation framework measures scanner robustness at both tile and slide levels through embedding visualisation, geometric invariance metrics, and neighbourhood preservation. These analyses assess whether scanner changes preserve representations, distances, and local structure for identical specimens.

  • Embedding-space analysis: UMAP visualisations reveal whether scanner-specific signatures form distinct clusters in tile- and slide-level embedding spaces.Colour mixing indicates scanner-invariant representations.
  • Embedding construction: Slide-level embeddings are obtained by mean-pooling tile-level feature vectors, providing representations for whole-slide geometric analysis.The extractor maps each tile to a feature vector before aggregation.
  • Cross-scanner consistency: Average Pairwise Cosine Distance compares embeddings of identical slides across scanner pairs, with lower values indicating stronger scanner invariance.The metric is computed across 384 patients and visualised for 14 feature extractors across ten scanner pairs.
  • Local structure: 1-Nearest Neighbour Match Rate tests whether cross-scanner retrieval identifies the same physical slide, while values near 100% indicate preserved local neighbourhood structure.The metric evaluates relative ordering rather than absolute embedding differences.
  • Global structure: Mantel correlation compares patient-to-patient distance matrices across scanners to quantify preservation of global embedding geometry.Higher rM values mean that slide pairs similar under one scanner remain similar under another.
  • Distribution and neighbourhood stability: Mean Intra-Scanner Distance and Intersection-over-K quantify scanner-specific density shifts and shared neighbourhoods across all scanners.High IoK indicates that similar histological slides remain grouped together regardless of acquisition device.

4.2. Supervised Downstream Model Evaluation

The supervised evaluation tests whether scanner variability changes clinicopathological predictions, their cross-scanner consistency, or their probability calibration. It combines five breast-cancer tasks with repeated training, scanner-wise testing, and complementary performance and stability measures.

  • Task design: Five tasks predict ER, PR, HER2, binary NHG, and multiclass NHG status from H&E WSIs using models trained on TCGA-BRCA and tested on CHIME scanners.The design evaluates downstream behaviour across five acquisition devices.
  • Evaluation dimensions: The evaluation separates predictive performance, prediction consistency, and calibration stability to distinguish different effects of scanner variability.This framework tests whether shifts degrade discrimination, alter discrete decisions, or change predicted probabilities.
  • Prediction model: ABMIL with a learnable gated-attention mechanism aggregates tile-level embeddings into slide-level predictions for every downstream task.The implementation uses the CLAM-SB framework with instance-level clustering constraints disabled.
  • Training and testing: Ten stratified 80/20 TCGA-BRCA splits and five scanner evaluations produce 50 results per feature extractor and task combination.The same randomised splits are used across feature extractors to support fair comparison.
  • Performance and consistency: AUC measures discriminative performance, while Fleiss’ κ measures whether predicted class labels remain consistent across the five scanners.For multiclass NHG, performance is reported as one-vs-rest macro-averaged AUC.
  • Calibration stability: Calibration stability compares predicted probabilities for the same slides across scanner pairs, using diagonal agreement and LOWESS deviations as indicators of scanner-dependent bias.The procedure aggregates bootstrapped LOWESS curves across ten random seeds with 95% confidence intervals.

5. Results

Across unsupervised embedding and supervised prediction analyses, PFMs showed substantial scanner-dependent variation. CONCH, CONCHv1.5, and H0-mini were generally more robust, but robustness did not consistently translate into superior supervised performance or calibration stability.

  • Qualitative embedding analysis: CONCH and CONCHv1.5 showed greater scanner-domain overlap than most vision-only ViT-based PFMs in tile- and slide-level UMAP embeddings.Virchow also showed subtle inter-device separation between HXR1 and HXR2 that became more pronounced after slide-level aggregation.
  • Quantitative geometric embedding analysis: 81.8% MR1NN for CONCHv1.5 versus 25.8% for Phikon-v2 on G20X-PHIL demonstrated substantial differences in preserved local neighbourhood structure.The HXR1-HXR2 pair achieved near-perfect MR1NN across feature extractors.
  • Quantitative geometric embedding analysis: Inter-scanner distance distributions shifted substantially for many vision-only ViTs, indicating scanner-dependent changes in feature-space geometry and density.These shifts corresponded to scanner pairs with larger pairwise cosine distances, including G20X-PHIL for UNI and UNI2-h.
  • Quantitative geometric embedding analysis: CONCH and CONCHv1.5 ranked among the most robust models across Dcos, MR1NN, and Mantel correlation, preserving expressive cross-scanner representations.ResNet-IN achieved the lowest Dcos but ranked near the bottom on MR1NN and Mantel correlation, indicating that its apparent robustness reflected feature collapse.
  • Cross-model robustness patterns: Robustness was not uniformly improved by larger scale or newer versions, while H0-mini often outperformed H-Optimus-0 and CTransPath sometimes outperformed newer models.The results indicate that architectural and training choices were more informative than model size, recency, or pretraining dataset scale alone.
  • Supervised prediction performance: All ViT-based extractors achieved high AUC confidence intervals overall, yet scanner-specific performance variation appeared for tasks including NHG classification.Models strongest on robustness, including CONCH, CONCHv1.5, and H0-mini, were not top-ranked on supervised prediction tasks.
  • Prediction consistency: Fleiss’ κ ranged from approximately 0.4 for Phikon-v2 on HER2 to 0.85 for CONCHv1.5 on ER across downstream tasks.CONCH and CONCHv1.5 achieved the highest agreement for ER, PR, binary NHG, and multiclass NHG, while H0-mini exceeded its teacher on prediction consistency.

6. Discussion

Across embedding and downstream analyses, current PFMs remain sensitive to scanner-induced shifts despite strong discriminative performance. Robustness varies with model design and training strategy, while calibration instability remains a deployment concern.

  • Evaluation: 14 PFM feature extractors were evaluated using embedding geometry, supervised classification, prediction consistency, and calibration stability metrics.The benchmark used the CHIME Multiscanner dataset and controlled scanner effects independently of other variability sources.
  • Embedding robustness: Most vision-only ViT models show scanner-dependent clustering, larger pairwise cosine distances, and reduced neighbourhood consistency across scanner pairs.These effects appear in both tile- and slide-level representations despite extensive self-supervised data augmentation.
  • Embedding robustness: CONCH and CONCHv1.5 demonstrate greater scanner invariance across embedding metrics, likely reflecting their more diverse visual-semantic pretraining data.CONCH was pretrained on over one million image-caption pairs from PubMed figures and captions.
  • Model design: H0-mini consistently outperforms H-Optimus-0 across multiple embedding-stability and downstream-consistency measures.The result suggests robustness-oriented distillation can improve cross-scanner generalisation without increasing model size.
  • Model design: Robustness does not increase uniformly with model scale or recency, as UNI2-h, Virchow2, and H-Optimus-1 remain sensitive to scanner shifts.Architectural choices, pretraining objectives, and training data appear more consequential than size or release date alone.
  • Clinical deployment: High AUC can coexist with systematic scanner-dependent calibration shifts, making predicted probabilities and fixed decision thresholds unreliable across scanners.Supervised training partially mitigates scanner effects on discrete predictions but does not resolve calibration instability.
  • Clinical deployment: The study recommends calibration-aware objectives, scanner-aware augmentation, and geometric embedding-stability metrics alongside AUC.The authors also identify multiscanner validation as important for deploying current non-robust PFMs.
  • Limitations: The analysis is limited to breast cancer resections, does not spatially register tiles, uses ABMIL aggregation, and does not comprehensively evaluate mitigation strategies.These choices may obscure local scanner effects or limit generalisation to other tissues, preparations, aggregation methods, and newly released models.

7. Conclusion

The conclusion is that current PFMs are not invariant to scanner-induced variability: scanner shifts distort embeddings and calibration even when AUC remains strong. Robustness appears more related to training-data diversity and design strategies than to scale or recency, motivating explicit multiscanner robustness and calibration evaluation.

  • Conclusion: 14 pathology feature extractors show that current state-of-the-art PFMs are not invariant to scanner-induced domain shifts.Scanner variability distorts embedding geometry and downstream calibration despite strong AUC performance for most models.
  • Conclusion: None of the evaluated models exhibits satisfactory robustness to scanner-device variability.Robustness is not a simple function of training-data scale, model size, or model recency.
  • Conclusion: Training-data diversity, architectural design, and training strategies may influence robustness more than scale alone.Vision-language models show comparatively improved robustness, while robustness-oriented distillation may improve scanner invariance without increasing model size.
  • Implications: Future PFM development should explicitly prioritise robustness and calibration stability across acquisition devices through comprehensive multiscanner validation.The authors argue that scanner-induced shifts should inform research studies, trials, and clinical deployment.

Competing Interests

The competing-interests statement reports that MR is a co-founder and shareholder of Stratipath AB, while KLE is employed by Stratipath AB and holds employee stock options.

  • Competing interests: MR is a co-founder and shareholder of Stratipath AB; KLE is employed there and holds employee stock options.ET and FKG report no competing interests.

Declaration of Generative AI Use

The authors used Gemini 2.5 and ChatGPT 5.2 during manuscript preparation for language editing, drafting assistance, and correction suggestions, with authors retaining responsibility for the final content.

  • Generative AI use: Gemini 2.5 and ChatGPT 5.2 assisted with language editing, drafting, and identifying typographical and grammatical errors.The authors state that all generated content was critically reviewed, edited, and validated.

A. Supplementary Figures

The supplementary figures extend the supervised calibration-stability evaluation across feature extractors and scanner pairs for multiple clinicopathological tasks. They show scanner-dependent calibration shifts or LOWESS curves for multiclass NHG, ER, PR, HER2, and binary NHG status.

  • Multiclass NHG: The multiclass NHG supplement shows examples of scanner-dependent calibration shifts for all evaluated feature extractors.It extends Figure 9.
  • ER status: The ER-status supplement presents LOWESS curves across all feature extractors and scanner pairs.It follows the format and interpretation of Figure 10.
  • PR status: The PR-status supplement presents LOWESS curves across all feature extractors and scanner pairs.It follows the format and interpretation of Figure 10.
  • HER2 status: The HER2-status supplement presents LOWESS curves across all feature extractors and scanner pairs.It follows the format and interpretation of Figure 10.
  • Binary NHG: The binary NHG supplement presents LOWESS curves across all feature extractors and scanner pairs for the 1-versus-3 task.It follows the format and interpretation of Figure 10.

B. Experimental Details

The supplementary experimental details specify the hyperparameters used to train ABMIL models across all five clinical tasks in the supervised downstream evaluation.

  • ABMIL training: Table S1 lists the hyperparameters used to train ABMIL models on all five clinical tasks.
Loading 2601.04163v1…