Source-linked AI summary
Virchow2: Scaling Self-Supervised Mixed Magnification Models in Pathology
Eric Zimmermann, Eugene Vorontsov, Julian Viret, Adam Casson, Michal Zelechowski, George Shaikovski, Neil Tenenholtz, James Hall, David Klimstra, Razik Yousfi, Thomas Fuchs, Nicolo Fusi, Siqi Liu, Kristen Severson
TL;DR
The study asks which factors—data scale and diversity, model size, or training algorithms—most improve computational pathology foundation models. It scales data and models while introducing pathology-specific training modifications, finding that performance benefits from their combination, with methodological gains also possible at smaller scales.
Problem
The relative importance of data scale and diversity, model size, and training algorithms for computational pathology remains underexplored, and it is unknown whether natural-image scaling trends transfer without adaptation.
Method
The authors develop mixed-magnification Virchow2 and Virchow2G models by scaling pathology data and model size while adapting DINOv2 with domain-inspired modifications.
Results
Performance improved across in-distribution and out-of-distribution tile benchmarks, with average weighted F1 increasing from 0.944 to 0.966 in-distribution and from 0.877 to 0.894 out-of-distribution across the Virchow series.
Takeaways & Limitations
Pathology-specific improvements and data diversity can deliver substantial gains, while combining algorithmic, data, and model scaling continues to improve average performance.
Takeaways & Limitations
Further gains from model scaling may be difficult to measure on benchmarks where performance is already near saturation, and additional benchmarks may be needed to assess continued improvement.
Abstract
from arXiv · showhide
Foundation models are rapidly being developed for computational pathology applications. However, it remains an open question which factors are most important for downstream performance with data scale and diversity, model size, and training algorithm all playing a role. In this work, we propose algorithmic modifications, tailored for pathology, and we present the result of scaling both data and model size, surpassing previous studies in both dimensions. We introduce three new models: Virchow2, a 632 million parameter vision transformer, Virchow2G, a 1.9 billion parameter vision transformer, and Virchow2G Mini, a 22 million parameter distillation of Virchow2G, each trained with 3.1 million histopathology whole slide images, with diverse tissues, originating institutions, and stains. We achieve state of the art performance on 12 tile-level tasks, as compared to the top performing competing models. Our results suggest that data diversity and domain-specific methods can outperform models that only scale in the number of parameters, but, on average, performance benefits from the combination of domain-specific methods, data scale, and model scale.
1 Introduction
Computational pathology foundation models aim to learn generalizable representations from diverse unlabeled whole slide images, but the effects of scaling data, model size, and training methods remain underexplored. This work investigates these factors together through pathology-adapted training and larger, more diverse datasets.
- Motivation: Whole slide images support foundation models that learn general representations across stains, tissues, diseases, and clinical tasks without task-specific labels.These models target applications including diagnosis, disease subtyping, biomarker quantification, treatment-response estimation, and survival prediction.
- Open question: Scaling trends from natural-image foundation models may extend to computational pathology, but their applicability without domain adaptation remains unknown.Earlier CPath studies suggested benefits from dataset and model scaling, while comparative evidence remained limited.
- Evaluation: Figure 1 compares average weighted F1 across eight public-data tile benchmarks against model-parameter size and WSI dataset size.It also presents Virchow2G Mini, a 22M-parameter distillation designed to remain competitive with substantially larger models.
- Approach: Virchow2 scales training from 1.5M single-institution H&E WSIs to 3.1M WSIs from globally diverse institutions and stains, while using pathology-inspired training modifications.Virchow2 is a 632M-parameter ViT-H model.
- Approach: Virchow2G extends Virchow2 by increasing model size from 632M to 1.9B parameters, enabling evaluation across both data- and model-scaling axes.The study evaluates in-domain and out-of-domain benchmarks at multiple magnifications.
2 Background & related work
Computational pathology foundation models use varied datasets but largely converge on self-distillation methods such as DINOv2, leaving algorithm choice and dataset characterization comparatively underexplored. Related work spans open-access and proprietary collections ranging from tens of thousands of WSIs to millions.
- Self-supervised learning: Joint-embedding self-supervised learning learns representations through alignment between augmented views and diversity across observations to avoid collapse.Self-distillation variants have become prominent for no-tuning transfer in natural images and computational pathology.
- Related models: At least ten computational pathology foundation models had been proposed, with CTransPath among the earliest using a 28M-parameter Swin transformer trained on 15M tiles from 32K WSIs.The passage introduces the field’s rapid model proliferation and an early training scale.
- Training algorithms: Most computational pathology foundation models use DINOv2 or related iBOT, while comparative studies have not established a consistent best self-supervised algorithm.One study found no clear winner among DINO, MoCo v2, SwAV, and Barlow Twins, whereas another reported a large algorithm effect.
- Training data: Earlier models primarily used open-access TCGA data of approximately 30K WSIs, whereas newer proprietary datasets range from 100K to 1M WSIs.Training tile counts varied from 4–50M for earlier open-data models and reached billions for newer proprietary-data models.
3 Adapting self-supervised learning to the pathology domain
Pathology images differ from natural images in structure, redundancy, magnification, and color variation, motivating domain-specific self-supervised adaptations. The paper proposes extended-context translation and KDE-based diversity regularization to reduce morphological distortion and address clustered features.
- Pathology tiles are repetitive, pose-invariant, minimally color-varying, and acquired at multiple magnifications, unlike typical natural images.
- Self-supervised feature quality depends on how augmentation-generated views relate to downstream objectives.
- Domain-specific augmentations: Pathology-specific augmentation choices include stain variation, while standard solarization may provide poor color invariance for pathology.
- Domain-specific augmentations: Extended-context translation samples output tiles from larger source regions, minimizing resizing and morphological distortion while preserving overlap between views.The larger source field of view extends WSI context without padding artifacts or computational overhead.
- Accounting for tissue redundancy: KDE replaces KoLeo as a bounded diversity regularizer when tissue features may be clustered and arbitrarily close.KoLeo can produce losses approaching infinity for near-identical samples; KDE preserves population-level diversity promotion with bounded gradients.
- Accounting for tissue redundancy: The unnormalized von Mises-Fisher kernel is used for KDE because of its computational qualities and demonstrated success encouraging diverse embeddings.Its concentration parameter controls the kernel scaling.
4 Stabilizing self-supervised vision transformers at scale
Scaling self-supervised vision transformers introduces optimization and representation-collapse risks. The paper addresses these risks by considering adaptive optimization, update clipping, and distillation-based stabilization.
- Larger vision transformers experience increasingly problematic gradient norms across layers, and partial failures may require rollback.
- AdamW stability depends on warm-up, learning rate, and momentum choices, while gradient spikes can signal likely collapse events.
- StableAdamW mitigates unusually large weight updates with root-mean-square clipping, though lowering second-moment settings may reduce downstream performance.
- DINOv2 primarily avoids dimension collapse through asymmetric centered and sharpened student-teacher cross-entropy over class and patch tokens.
5 Ablations investigating learning in the pathology domain
The ablation study tests pathology-specific augmentations and regularizers before scaling, using linear-probe evaluation across diverse magnifications and datasets. Combining domain-inspired changes generally performs best, especially for in-distribution tasks.
- The ablation sweeps augmentation and regularization choices at limited scale to assess viability and interactions before model and data scaling.
- Ablation setup: The ablation dataset contains 1.5M H&E-stained WSIs from 120K patients across four magnifications and 17 tissue types.
- Ablation setup: Evaluation uses seven in-distribution and out-of-distribution tile-classification tasks across magnifications with linear probes on frozen teacher embeddings.
- Ablation results: 5/7 tasks achieve their best performance with all three domain-inspired changes, which also gives the best average in-distribution performance.
- Ablation results: ECT plus KDE has the best performance on 3/7 tasks and the highest average out-of-distribution performance, within 0.002 of all three changes.
- Ablation results: ECT alone slightly reduces out-of-distribution performance, whereas KDE alone slightly improves it; combining them produces a pronounced improvement.
6 Results of large-scale training
The models scale pathology-specific training across data, model size, magnification, and embedding choices. Virchow2 and Virchow2G improve tile-level benchmark performance, while distillation produces a much smaller competitive model.
- 6.1 Training data: 3.1M WSIs spanning diverse patients, institutions, stains, tissue types, diagnoses, and magnifications form the scaled Virchow2 training dataset.Virchow2 extends the earlier 1.5M-WSI dataset and includes nearly 200 recorded tissue types.
- 6.2 Training recipe: Pathology-specific training uses ECT, KDE diversity regularization, mixed global and local magnification views, and online balancing of long-tailed data.The models use 224 × 224 global tiles and 98 × 98 local tiles extracted from 392 × 392 source regions.
- 6.6 Scaling results: 0.966 average weighted F1 on ID tasks increased from 0.944 for Virchow to 0.971 for Virchow2G, while OOD F1 increased from 0.877 to 0.894.Virchow2 provided the larger ID gain, whereas further model scaling to Virchow2G produced a more modest additional increase.
- 6.6 Scaling results: Virchow2G achieves top performance across benchmark tasks, with mixed magnifications improving 5× and 10× ID tasks and supporting performance on 40× OOD tasks.The TILS weighted F1 score was 0.948 for Virchow2G versus 0.949 for the top score, indicating similar performance on that task.
- 6.5 Tile benchmarks: Virchow2G records the best average HEST-Benchmark Pearson correlation at 0.350, while model performance varies across the ten gene-expression tasks.The authors attribute inconsistency partly to fixed embedding hyperparameters, weak gene-expression signal in some H&E images, and low sample counts.
- 6.7 Distillation: A 22M-parameter Virchow2G Mini distilled from the 1.9B-parameter Virchow2G outperforms several larger models in-distribution and nearly matches Virchow out-of-distribution.Virchow2G Mini uses roughly 1% of Virchow2G’s parameters.
7 Discussion
Virchow2’s results support continued benefits from data and model scale, while pathology-specific training changes also improve performance. However, saturated or insufficiently diverse benchmarks limit confident conclusions about further gains and model quality.
- Scaling and methodological effects: Pathology-specific training modifications can significantly improve benchmark performance alongside scaling data and model size.The authors report benefits from domain-inspired adaptations, including KDE regularization and extended-context translation.
- Scaling and methodological effects: Mixed magnifications allow reduced random resizing while retaining scale variation relevant to pathology images.ECT avoids distorting cell morphology, and training across magnifications preserves scale diversity.
- Scaling and methodological effects: Without KDE, DINOv2 training collapsed before processing the 3.1M-slide dataset, and further stabilization was needed to scale from ViT-H to ViT-G.Virchow2 uses 632M parameters, whereas Virchow2G uses 1.9B parameters.
- Evaluation boundaries: No substantial model-scale improvements appeared on WILDS, CRC, CRC-No-Norm, or TILS beyond ViT-H, where several task F1 scores were already 0.95–0.99.The authors attribute possible limits on measurable gains to benchmark saturation or label noise.
- Evaluation boundaries: Further gains require diverse benchmarks that have not saturated, because benchmark quality determines whether model improvements can be assessed confidently.The authors identify MIDOG, DLBCL, and PanMSK as examples containing magnifications other than 20×.
A.1 Foundation models in CPath
Table A1 organizes computational pathology foundation models by training-slide count and compares their training-data scale, architecture size, and training objective.
- Foundation models in CPath: Table A1 compares computational pathology foundation models across training WSI count, model architecture size, and training objective.The works are ordered by the number of training whole slide images.
A.2 Hyperparameters
The appendix reports adapted Virchow2, Virchow2G, and Virchow2G Mini hyperparameters derived from DINOv2 training configurations.
- Virchow2: Table A2 lists Virchow2’s DINOv2-adapted hyperparameters and training on 512 NVIDIA V100 32GB GPUs.Default augmentations are excluded because they remain unchanged.
- Virchow2G: Table A3 lists Virchow2G’s DINOv2-adapted hyperparameters for training on 512 NVIDIA V100 32GB GPUs.Default augmentations are not included and remain unchanged.
- Virchow2G Mini: Table A4 lists Virchow2G Mini’s adapted hyperparameters for training on 256 NVIDIA V100 32GB GPUs.Default augmentations are not included and remain unchanged.
A.3 Detailed ablation results
The detailed ablation results report weighted F1 scores for in-domain and out-of-domain tasks, with statistical significance conventions and named training modifications.
- Ablation results: Tables A5 and A6 report weighted F1 scores for in-domain and out-of-domain tile tasks, respectively.All metrics are multiplied by 100 for readability.
- Evaluation conventions: Bold values indicate the best-performing model or results not significantly different at α = 0.05 under McNemar’s test.The appendix notes that statistical testing is not applied to average performance.
- Ablation results: Tables A7 and A8 report weighted F1 scores for [CLS]-only embeddings while identifying ECT, KDE, and SOL configurations.ECT denotes extended context translation, KDE the proposed regularizer, and SOL removal of solarization.
A.4 Scaling plots
Scaling analyses examine how performance varies with model and training-data size across in-distribution and out-of-distribution tasks. MIDOG results additionally show that aggregating patch-token information improves performance, especially for localized mitosis detection.
- Scaling plots: Performance averages across in-distribution and out-of-distribution tasks scale with model size and training data size.The figure compares scaling along both axes and distinguishes CLS-only from concatenated embeddings for models with a CLS token.
- MIDOG patch-token analysis: Patch-based aggregation improves MIDOG performance across all tested models.The evaluated aggregators include max-pooling and attention-based aggregation of patch tokens.
- MIDOG patch-token analysis: CLS+Mean substantially benefits MIDOG by concatenating the mean of patch tokens with the CLS embedding.This initial analysis motivated further testing of patch-token aggregation.
- MIDOG patch-token analysis: Virchow2G and H-optimus-0 achieve the highest overall performance among the larger models in the MIDOG analysis.The passage reports this comparison across the tested ViT-based computational pathology foundation models.
- Scaling plots: Tables A10 and A11 report weighted F1 results for in-distribution and out-of-distribution tile-level benchmark tasks after models saw 1B tiles.These tables compare Virchow2G Mini with a ViT-S/14 trained without distillation, although the supplied passage does not include their numeric results.