Source-linked AI summary
A Clinical Benchmark of Public Self-Supervised Pathology Foundation Models
Gabriele Campanella, Shengjia Chen, Ruchika Verma, Jennifer Zeng, Aryeh Stock, Matt Croken, Brandon Veremis, Abdulkadir Elmas, Kuan-lin Huang, Ricky Kwan, Jane Houldsworth, Adam J. Schoenfeld, Chad Vanderbilt
TL;DR
Pathology foundation models need independent clinical benchmarking because public models differ in size, algorithm, and pretraining data, while whole-slide pathology presents distinctive data and computational challenges. The paper benchmarks public models on clinically relevant tasks using slide-level aggregation and cross-validation, finding that newer pathology models generally outperform ImageNet pretraining, with task-dependent effects of model size and pretraining-data composition. The results support comparing models by task and considering dataset composition alongside scale.
Problem
Public pathology foundation models differ in training algorithms, datasets, and sizes, creating a need for independent comparison across clinically relevant tasks and pathology settings.
Method
The study benchmarks public pathology foundation models on clinical datasets using frozen tile embeddings, Gated MIL Attention slide aggregation, and Monte Carlo cross-validation.
Results
Newer pathology models generally outperform ImageNet-pretrained features; model-size benefits vary by task, while DINO and DINOv2 models perform comparably on many disease-detection and biomarker tasks.
Takeaways & Limitations
Model selection should consider downstream task and pretraining-dataset composition rather than relying on model size alone.
Takeaways & Limitations
Scaling conclusions depend on the currently included tasks, and benchmark test data are withheld to reduce pretraining-data contamination.
Abstract
from arXiv · showhide
The use of self-supervised learning (SSL) to train pathology foundation models has increased substantially in the past few years. Notably, several models trained on large quantities of clinical data have been made publicly available in recent months. This will significantly enhance scientific research in computational pathology and help bridge the gap between research and clinical deployment. With the increase in availability of public foundation models of different sizes, trained using different algorithms on different datasets, it becomes important to establish a benchmark to compare the performance of such models on a variety of clinically relevant tasks spanning multiple organs and diseases. In this work, we present a collection of pathology datasets comprising clinical slides associated with clinically relevant endpoints including cancer diagnoses and a variety of biomarkers generated during standard hospital operation from two medical centers. We leverage these datasets to systematically assess the performance of public pathology foundation models and provide insights into best practices for training new foundation models and selecting appropriate pretrained models.
1 Introduction
Self-supervised pathology foundation models address the data and computational challenges of whole-slide imaging, but their varied training algorithms, datasets, and scales require systematic clinical benchmarking. This work introduces an updateable benchmark spanning multiple institutions, organs, diseases, and clinically relevant tasks.
- Motivation: Self-supervised learning enables pathology foundation models to learn from large unlabeled datasets and support diverse downstream tasks with little fine-tuning.Pathology models remain less mature than those in computer vision and natural language because medical data and computational resources are limited.
- Motivation: Whole-slide images create substantial data and hardware demands, motivating tile-level encoding followed by slide-level feature aggregation.The tile-encoding stage is the dominant computational cost, encouraging reliance on pretrained encoders.
- Prior work: Prior pathology foundation models vary widely in architecture, SSL algorithm, pretraining scale, tissue coverage, and downstream evaluation.Examples include CTransPath, iBOT-based models, UNI, Virchow, RudolfV, and Prov-GigaPath, trained on datasets ranging from millions to billions of tiles.
- Prior work: Existing evidence supports pathology-specific SSL over natural-image pretraining, while model development remains constrained by relatively small pathology datasets.The cited prior work also reports DINO outperforming MAE for pathology foundation-model pretraining.
- Contribution: The study introduces a clinical benchmark dataset that systematically compares public pathology foundation models across disease detection, biomarker prediction, and treatment outcome prediction.The benchmark uses clinical data generated during standard hospital operations from two institutions and is intended to expand as models and datasets become available.
2 Method
The benchmark evaluates frozen foundation-model representations by aggregating tile features into slide-level predictions, with fixed Monte Carlo cross-validation splits and replicated training.
- Representation evaluation: Frozen tile embeddings are aggregated with Gated MIL Attention and linearly classified to test the expressiveness of each foundation model’s feature space.GMA does not use tile spatial distribution, providing a simple slide-level evaluation of pretrained representations.
- Representation evaluation: Each slide is represented as a 2D tile-by-feature matrix, which GMA converts into a slide-level representation and class scores.Tiles are extracted at 20x magnification, corresponding to 0.5 microns per pixel.
- Validation: Monte Carlo cross-validation assigns 80% of samples to training and 20% to validation, using 20 fixed folds and two replicas per split.Results are averaged across the two replicas to assess stochastic variation during training.
3 Downstream Tasks
The downstream benchmark assembles clinical slide datasets from multiple institutions and scanners, covering disease detection and computational biomarker prediction with pathology and molecular endpoints.
- Task collection: The task collection spans clinically relevant datasets from multiple institutions and scanners, summarized in dedicated detection and biomarker task tables.The datasets are used to assess the representation power of pathology foundation models.
- 3.1 Disease Detection: Disease-detection cohorts include breast cancer, oral cancer, and inflammation-versus-normal mucosa classification tasks.The inflammation cohort contains 1,441 slides: 717 with active inflammation and 724 with normal mucosa.
- 3.2 Computational Biomarkers: The breast biomarker cohorts predict ER, PR, HER2, and HRD-related endpoints from clinical pathology data.The ER cohort contains 2,000 slides, while PR and HER2 cohorts contain 1,986 and 2,018 slides, respectively.
- 3.2 Computational Biomarkers: The benchmark also includes molecular biomarker prediction, such as EGFR mutation status in lung adenocarcinoma identified through MSK-IMPACT molecular analysis.The task links molecular assay results with corresponding digitized slides.
4 Foundation Models
The benchmark compares public pathology foundation models with an ImageNet-pretrained baseline and in-house DINO models, covering architectures and scales from small to very large encoders.
- Public and baseline models: Public models include CTransPath, UNI, Virchow, and Prov-GigaPath, alongside a truncated ResNet50 pretrained on ImageNet as a baseline.Virchow’s pretraining on MSKCC slides prevents assurance that its pretraining cohort is disjoint from benchmark tasks.
- In-house models: The study adds in-house ViT-small and ViT-base models trained with DINO on 423,563 clinical H&E slides from 88,035 cases and 76,794 patients.The pretraining collection covers 42 organs across pathology specialties and was constructed from normal hospital operations.
5 Results
Disease detection was consistently strong across encoders, whereas biomarker prediction was more variable and task-dependent. Larger models showed no clear benefit for detection, and computational cost did not consistently predict downstream performance.
- Disease Detection Tasks: AUCs exceeded 0.9 for all tested encoders across disease detection tasks, with ImageNet consistently underperforming pathology-trained encoders.
- Disease Detection Tasks: DINO- and DINOv2-trained models achieved comparable detection performance, making inference cost a possible model-selection consideration.
- Biomarker Prediction Tasks: Biomarker prediction varied more than detection; pathology-trained models generally outperformed ImageNet and CTransPath, with larger-model advantages limited to selected tasks.
- Biomarker Prediction Tasks: For lung NGS mutations, UNI and Prov-GigaPath achieved higher AUCs than several smaller models, potentially reflecting greater lung representation in their pretraining datasets.
- Biomarker Prediction Tasks: All models performed poorly for immune-checkpoint-inhibitor response prediction in NSCLC, with AUCs barely above chance; UNI averaged 0.6.
- Scaling Laws: Foundation Model Size: Detection performance showed no evidence of scaling with model size, while biomarker performance tended to increase with size but depended on task and possibly dataset composition.
- Scaling Laws: Computational Resources: Higher computational cost showed no evidence of improving detection or biomarker prediction; UNI outperformed the more resource-intensive SP22M and SP85M models on average.
6 Discussion
The discussion argues that independent clinical benchmarking is needed to guide foundation-model development and downstream model selection. Results suggest newer DINO/DINOv2 models generally outperform older baselines, while scaling benefits remain task- and dataset-dependent.
- Benchmark purpose: Independent clinical benchmarks help researchers improve pretraining and select models for downstream tasks under performance and computational constraints.The benchmark uses clinically generated data without further curation to represent biological and technical variability from real-world operations.
- Benchmark governance: The benchmark test data are withheld to reduce the risk of contamination from future foundation-model pretraining.The authors instead plan regular benchmark updates and a future API for external-model evaluation.
- Benchmark findings: Newer models generally outperform the ImageNet-pretrained encoder and, to a lesser degree, CTransPath.The comparison is summarized across the benchmark's disease detection and biomarker prediction analyses.
- Scaling and dataset composition: Model size has little apparent effect on disease detection, while larger models tend to help biomarker prediction depending on the task and pretraining-data composition.The discussion cautions that current analyses do not establish broad scaling laws for pathology tile encoders.
- Scaling and dataset composition: Dataset composition may be crucial for downstream performance, making greater pretraining-data curation and tissue-specific models plausible alternatives to general-purpose models.The authors expect incremental rather than dramatic gains with current SSL strategies.
- Open questions: Important pretraining questions remain unresolved for magnification choice, mixed-resolution training, and broader use of IHC slides.The paper states that these aspects could not be addressed because of limited evidence.
Declarations
The authors will release benchmarking code while withholding benchmark data to reduce potential future training-data leakage. Benchmark results will be continuously updated as public models become available.
- Code to run the benchmarks will be made available upon acceptance.
- Benchmark data are deliberately withheld because data leakage could compromise future foundation-model evaluation.The benchmark will be continuously updated with available public foundation models.