Source-linked AI summary
Virchow: A Million-Slide Digital Pathology Foundation Model
Eugene Vorontsov, Alican Bozkurt, Adam Casson, George Shaikovski, Michal Zelechowski, Siqi Liu, Kristen Severson, Eric Zimmermann, James Hall, Neil Tenenholtz, Nicolo Fusi, Philippe Mathieu, Alexander van Eck, Donghun Lee, Julian Viret, Eric Robert, Yi Kan Wang, Jeremy D. Kunz, Matthew C. H. Lee, Jan Bernhard, Ran A. Godrich, Gerard Oakley, Ewan Millar, Matthew Hanna, Juan Retamero, William A. Moye, Razik Yousfi, Christopher Kanan, David Klimstra, Brandon Rothrock, Thomas J. Fuchs
TL;DR
Computational pathology needs representations that capture the diverse patterns in H&E whole slide images for cancer diagnosis and related applications. Virchow addresses this with a 632M-parameter DINOv2-based vision transformer trained on 1.5 million slides, achieving strong cancer detection, tile-level, and biomarker-prediction results.
Problem
Computational pathology requires image representations that capture diverse pathology patterns, while prior foundation models used substantially smaller datasets and architectures.
Method
Virchow is a 632M-parameter vision transformer trained with DINOv2 self-supervised learning on 1.5 million H&E whole slide images.
Results
Virchow embeddings outperform smaller-scale foundation-model embeddings across tasks, including pan-cancer detection, out-of-distribution benchmarks, and biomarker prediction.
Takeaways & Limitations
The results support scaling pathology datasets and model architectures to improve computational pathology applications, including tasks with limited training data.
Takeaways & Limitations
The training dataset comes from one center with limited scanner types, and slide-level predictions require an aggregation model whose architectures were not deeply investigated.
Abstract
from arXiv · showhide
The use of artificial intelligence to enable precision medicine and decision support systems through the analysis of pathology images has the potential to revolutionize the diagnosis and treatment of cancer. Such applications will depend on models' abilities to capture the diverse patterns observed in pathology images. To address this challenge, we present Virchow, a foundation model for computational pathology. Using self-supervised learning empowered by the DINOv2 algorithm, Virchow is a vision transformer model with 632 million parameters trained on 1.5 million hematoxylin and eosin stained whole slide images from diverse tissue and specimen types, which is orders of magnitude more data than previous works. The Virchow model enables the development of a pan-cancer detection system with 0.949 overall specimen-level AUC across 17 different cancer types, while also achieving 0.937 AUC on 7 rare cancer types. The Virchow model sets the state-of-the-art on the internal and external image tile level benchmarks and slide level biomarker prediction tasks. The gains in performance highlight the importance of training on massive pathology image datasets, suggesting scaling up the data and network architecture can improve the accuracy for many high-impact computational pathology applications where limited amounts of training data are available.
1 Main
Virchow is a large pathology foundation model designed to capture diverse patterns in digitized H&E slides through self-supervised learning. It uses million-scale training data and supports downstream cancer detection and biomarker prediction tasks.
- 1 Main: Self-supervised foundation models learn reusable image embeddings without task-specific curated labels, supporting diverse pathology applications.Such applications include cancer detection, subtyping, biomarker quantification, mitotic counting, and survival prediction.
- 1 Main: Virchow is a 632M-parameter vision transformer trained with DINOv2 on approximately 1.5 million H&E whole slide images from approximately 100 thousand patients.The dataset includes cancerous and benign tissue, biopsy and resection specimens, and 17 high-level tissue types.
- 1 Main: Virchow embeddings support slide-level prediction by aggregating tile embeddings across whole slide images.DINOv2 processes global and local views of tissue tiles, which are then aggregated for downstream predictive tasks.
- 1 Main: The study evaluates Virchow across broad downstream computational pathology benchmarks, including cancer detection and biomarker prediction.The authors report comparisons across internal and external benchmarks, rare cancers, and out-of-distribution data.
- 1 Main: Virchow embeddings outperform or match baseline models across tested cancer types and achieve state-of-the-art biomarker prediction performance.The reported results motivate large-scale foundation models as a basis for robust computational pathology applications.
2 Results
Virchow embeddings were evaluated across slide-level cancer detection and biomarker tasks, tile-level linear probing benchmarks, and cell-feature analysis. Across these evaluations, Virchow generally matched or exceeded competing embeddings, including on rare cancers, external data, diverse tissue tasks, and biomarkers.
- 2.1 Pan-cancer detection: 0.949 overall AUC was achieved for pan-cancer detection with Virchow embeddings, versus 0.930 with Phikon and 0.904 with CTransPath.The differences were statistically significant with p « 0.001.
- 2.1 Pan-cancer detection: 0.937 AUC was achieved on rare cancers, although cervical and bone cancer detection remained below 0.9 across embeddings.Virchow reached 0.875 AUC for cervix, 0.841 for bone, and 0.954 for brain cancer detection.
- 2.1 Pan-cancer detection: 0.006 AUC was the Virchow performance drop from internal to external data, supporting robust out-of-distribution detection across cancer types and unseen tissues.The evaluation included 18.9% OOD tissues, and half of the specimens came from external institutions.
- 2.2 Biomarker detection: 0.972, 0.902, and 0.853 AUC were obtained for ColonMSI, BladderFGFR, and LungEGFR biomarker prediction, respectively.The reported confidence intervals were 0.950–0.989, 0.862–0.941, and 0.804–0.891.
- 2.3 Tile-level benchmarks: Virchow ranked top-1 by weighted F1 across all six tile-level tasks and matched or surpassed baseline performance across tissue types and cancer subtypes.It also showed strong performance on WILDS and CRC without stain normalization, with only a −0.005 weighted-F1 decline without normalization.
- 2.3 Tile-level benchmarks: Virchow’s first principal component highlighted malignant epithelium, while the second highlighted miscellaneous or inflammatory cells in CoNSeP images.The authors describe this as approximate semantic segmentation emerging from unsupervised features.
3 Discussion
Virchow extends computational pathology foundation-model scaling to 1.5 million WSIs and shows robust performance across external, out-of-distribution, rare-cancer, biomarker, and tile-level tasks. The authors note unresolved training-design questions and limitations involving data provenance, tile-level embeddings, aggregation, and clinical validation.
- 3 Discussion: The model’s 1.5 million-WSI training scale is over three thousand times ImageNet’s pixel volume and motivates larger models for diverse pathology features.The discussion presents this scale increase as a major expansion beyond prior computational-pathology training datasets.
- 3 Discussion: DINOv2 is not established as uniquely optimal because prior comparisons found no clear best self-supervised approach across 37 thousand WSIs.The authors identify the choice of self-supervised algorithm as an open design question, especially under long-tailed pathology-feature distributions.
- 3 Discussion: Histopathology-specific choices such as color augmentation remain open because stain and scanner variation may not reflect underlying pathology.The discussion distinguishes domain-specific training decisions from techniques transferred from natural-image settings.
- 3 Discussion: The training data comes from one center with limited scanner types, while embeddings remain tile-level and require a separately trained aggregation model.The authors also state that deep investigation of aggregator architectures and training procedures is beyond scope and that stratified clinical validation is required.
- 3 Discussion: Virchow embeddings outperform smaller-scale foundation models across pan-cancer, biomarker, and tile-level tasks, including external, out-of-distribution, and rare-cancer data.The authors attribute this performance and robustness to learning a rich representation of pathology-image diversity at scale.
4 Methods
Virchow was trained on nearly 1.5 million H&E WSIs from MSKCC using a 632-million-parameter ViT-H/14 and DINOv2, producing tile embeddings for downstream aggregation. Pan-cancer evaluation used internal and external specimens across 17 tissue types, with labels extracted from clinical reports and Agata aggregators trained against baseline embeddings.
- 4 Methods: WSIs were converted into 224 × 244 foreground tiles after downsampling and tissue-threshold filtering before training.Tiles were retained when they contained at least 25% tissue by area.
- 4 Methods: Virchow uses a 632-million-parameter ViT-H/14 trained with DINOv2 self-supervision on H&E WSIs from 119,629 patients across 17 tissue groups.The training set contains 1,488,550 WSIs scanned at 20× resolution with Leica scanners.
- 4 Methods: DINOv2 learns global and local tile representations through a student-teacher setup whose teacher is an exponential moving average of student networks.Virchow used modified warmup and teacher-temperature schedules with AdamW and float16 precision.
- 4 Methods: A Virchow tile embedding concatenates the class token with the mean of 256 other predicted tokens, producing a 2,560-dimensional representation.Phikon uses its class token, whereas CTransPath uses the mean of all tokens because it has no class token.
- 4 Methods: Pan-cancer models aggregate foreground-tile embeddings across specimens using Agata, with cancer labels extracted from synoptic and diagnostic reports.Training and validation comprised 177,742 slides from 47,839 specimens; evaluation covered 23,408 slides from 6,372 specimens across 17 tissue types.
- 4 Methods: The test set combines 15,941 MSKCC slides with 7,467 slides from institutions worldwide, and includes both common and rare cancer groups.None of the evaluation patients were seen during training.
4.5 Biomarker detection
Biomarker prediction treats each case as a binary classification problem and uses Agata to aggregate tile embeddings from H&E slides. Virchow achieved high AUCs across colon MSI, bladder FGFR, and lung EGFR tasks, while tile-level benchmarks span internal and public datasets with linear probing.
- 4.5 Biomarker detection: Biomarker tasks use binary case labels, patient-disjoint dataset splits, and one Agata aggregator network per biomarker.Positive labels indicate biomarker presence, and cases may contain one or more H&E slides sharing the same label.
- 4.5 Biomarker detection: The biomarker panel covers colon microsatellite instability, bladder FGFR alterations, and lung EGFR mutations using standard H&E images.These biomarkers are described as clinically relevant to cancer diagnosis and treatment.
- 4.5 Biomarker detection: The ColonMSI dataset contains 2,698 colorectal-cancer samples, including 288 MSI-H/dMMR-positive cases.IHC results are prioritized when both IHC and MSK-IMPACT sequencing outcomes are available.
- 4.6.2 Dataset description: Tile-level evaluation compares frozen-encoder embeddings with linear classifiers across an internal PanMSK benchmark and public datasets.PanMSK contains 3,999 held-out slides across 17 tissue types, including 1,456 slides with pathologist cancer annotations.
- 4.6.2 Dataset description: The public tile benchmarks include CRC, WILDS, MHIST, and PCam datasets with varied tissue types, magnifications, resolutions, and distribution shifts.CRC includes a stain-normalized and an unnormalized variant whose testing subset introduces distribution shift.
A.1 Early foundation models in computational pathology
Early computational pathology foundation models differed in training data, architecture size, and self-supervised objective, with these properties summarized across proposed models.
- Earlier models included a 28M-parameter SwinTransformer trained with a modified MoCoV3 procedure and an 86M-parameter ViT-B trained with iBOT.
- Proposed computational pathology foundation models varied in training-data scale, model architecture, and training objective.Table A1 summarizes these dimensions across the models.
A.2 Multi-tissue PanMSK dataset
PanMSK is a multi-tissue cancer-versus-benign tile dataset constructed from annotated slides and partitioned to balance tissue, label, slide, and tile distributions across splits.
- The dataset uses exhaustive annotations for selected prostate, breast, bladder, breast lymph node, and colon slides, while other tissue groups have non-exhaustive highlighted cancer regions.
- PanMSK splits are made at the slide level with no shared slides or tiles, approximately following a 7:1:2 train-validation-test ratio for both slides and tiles.
- Splits were algorithmically balanced across tissue groups and cancer/benign labels, with cancer tiles reduced in tissue groups having the largest counts.
- Table A3 reports per-tissue tile-level cancer classification performance, with overall performance computed after combining tiles across tissues.
A.3 Model training method
Virchow uses DINOv2 self-supervised training with a ViT-H architecture, matching student representations of masked and augmented views to teacher representations.
- Virchow uses a ViT-H architecture trained with the DINOv2 self-supervised learning method.
- Each tile produces 2 global crops and 8 local crops with random augmentations for student-teacher representation matching.
- The student receives randomly masked global crops, while the teacher receives unmasked versions and supplies target representations from opposite views.
A.4 Pan-cancer aggregator architecture details
The Agata aggregator converts tile embeddings into predictions through learned cross-attention followed by nonlinear classification layers.
- Agata uses cross-attention to learn which tiles contribute to the label decision.
- Unlike typical cross-attention, Agata parameterizes queries directly while deriving keys and values from tile embeddings through projection layers.
- The model produces 256-dimensional keys and 512-dimensional values, omitting scaling by √d_k = 16.
- After attention, ReLU-activated linear layers and a final softmax layer generate the output predictions.
- Figure A3 depicts Agata for specimen-level pan-cancer detection and biomarker detection tasks.
A.5 Tile-level benchmarks
The tile-level benchmark appendix reports multiple evaluation metrics for each model, with detailed results presented in Table A4.
- The reported metrics are accuracy, balanced accuracy, and weighted F1 score.
- Balanced accuracy averages true positive rate and true negative rate, while weighted F1 averages classwise F1 scores weighted by positive-sample counts.
- Table A4 presents downstream task linear probing evaluations for the tile-level benchmarks.
A.6 Acronyms
This section defines acronyms used throughout the paper, spanning pathology, machine learning, cancer biology, datasets, and evaluation terminology.
- AI denotes artificial intelligence; MIL denotes multiple instance learning; PCA denotes principal component analysis; SGD denotes stochastic gradient descent.
- AUC denotes area under the receiver operating characteristic curve, while ROC denotes receiver operating characteristic.
- H&E denotes hematoxylin and eosin; PCR denotes polymerase chain reaction; EMA denotes exponential moving average; GELU denotes Gaussian Error Linear Unit.
- MSI denotes microsatellite instability; dMMR denotes deficient mismatch repair; EGFR denotes epidermal growth factor receptor; FGFR denotes fibroblast growth factor receptor.
- MSK-IMPACT denotes MSK-Integrated Mutation Profiling of Actionable Targets, and MSKCC denotes Memorial Sloan Kettering Cancer Center.
- NSCLC denotes non-small cell lung cancer; PAIP denotes Pathology AI Platform; TCGA denotes The Cancer Genome Atlas; CoNSeP denotes colorectal nuclear segmentation and phenotypes.