PaperlayerView exact revision on arXiv

Source-linked AI summary

Phikon-v2, A large and public feature extractor for biomarker prediction

Alexandre Filiot, Paul Jacob, Alice Mac Kain, Charlie Saillard

arXiv:2409.09173v1eess.IVcs.AIcs.CV

TL;DR

Histology foundation models have grown in scale, but larger models do not consistently perform better on biomarker prediction. The authors train and publicly release Phikon-v2 on a large public histology dataset, then benchmark it across external slide-level tasks. It performs competitively with state-of-the-art models, while the authors identify limits to general-purpose models and call for task- and organ-specific development.

  • Problem

    Larger histology foundation models do not systematically provide better or more robust representations, particularly for biomarker prediction.

    Source: §2 · p. 3

  • Method

    The authors train a DINOv2 ViT-L on 460 million tiles from public histology slides and compare it with 14 public feature extractors on slide-level tasks.

    Source: §1 · p. 2, §3.5 · p. 6

  • Results

    Phikon-v2 performs competitively with state-of-the-art models across eight slide-level biomarker tasks, while ensembling improves AUC by +1.75 over one-shot retraining across tasks and models (p<0.001).

    Source: §5 · p. 10, §4.3 · p. 10

  • Takeaways & Limitations

    The authors conclude that scaling can benefit downstream performance, but task- and organ-specific fine-tuning or distillation merits attention because general-purpose models may suit specific indications differently.

    Source: §5 · p. 10

    The authors did not assess iBOT-only scalability on PANCAN-XL, and current large foundation models remain general-purpose across tasks and target organs.

    Source: §5 · p. 10

Abstract

from arXiv · show

Gathering histopathology slides from over 100 publicly available cohorts, we compile a diverse dataset of 460 million pathology tiles covering more than 30 cancer sites. Using this dataset, we train a large self-supervised vision transformer using DINOv2 and publicly release one iteration of this model for further experimentation, coined Phikon-v2. While trained on publicly available histology slides, Phikon-v2 surpasses our previously released model (Phikon) and performs on par with other histopathology foundation models (FM) trained on proprietary data. Our benchmarks include eight slide-level tasks with results reported on external validation cohorts avoiding any data contamination between pre-training and evaluation datasets. Our downstream training procedure follows a simple yet robust ensembling strategy yielding a +1.75 AUC increase across tasks and models compared to one-shot retraining (p<0.001). We compare Phikon (ViT-B) and Phikon-v2 (ViT-L) against 14 different histology feature extractors, making our evaluation the most comprehensive to date. Our result support evidences that DINOv2 handles joint model and data scaling better than iBOT. Also, we show that recent scaling efforts are overall beneficial to downstream performance in the context of biomarker prediction with GigaPath and H-Optimus-0 (two ViT-g with 1.1B parameters each) standing out. However, the statistical margins between the latest top-performing FMs remain mostly non-significant; some even underperform on specific indications or tasks such as MSI prediction - deposed by a 13x smaller model developed internally. While latest foundation models may exhibit limitations for clinical deployment, they nonetheless offer excellent grounds for the development of more specialized and cost-efficient histology encoders fueling AI-guided diagnostic tools.

1 Introduction

Histopathology supports diagnosis and treatment planning, while self-supervised learning has advanced computational pathology by providing slide-derived representations for complex prediction tasks. The paper introduces Phikon-v2, a publicly released DINOv2-trained ViT-L built on public histology tiles and evaluated on slide-level tasks.

Computational pathology

  • Digital pathology and whole-slide images have enabled computational pathology approaches with potential applications in disease classification, treatment planning, and drug discovery.Source: §1 · p. 1
  • Self-supervised representations of whole-slide-image patches support predictions of complex outcomes, including genomic mutations, survival rates, and histological and molecular subtypes.Source: §1 · p. 2

Phikon-v2

  • Phikon-v2 is a DINOv2-trained ViT-L trained on 460 million histology tiles from more than 55,000 public slides, covering healthy tissues and over 30 cancer sites.The model is publicly available on Hugging Face.Source: §1 · p. 2
  • Phikon-v2 demonstrates competitive performance with state-of-the-art models on whole-slide biomarker prediction and disease-classification tasks.Source: §1 · p. 2

2 Related Work

Histology representation learning has progressed from patch-level self-supervised methods to large foundation models, alongside increasing data and model scale. Yet larger models do not consistently improve biomarker prediction, motivating this study’s comparison of Phikon-v2 with Phikon and other public encoders.

Self-supervised learning

Histology foundation models

  • Later models expanded architectures and datasets, including CTransPath, HIPT [32], and five publicly released models trained on 32.6 million TCGA tiles [28].Source: §2 · p. 2
  • Recent foundation models include Phikon, UNI, Virchow, RudolfV, and models using DINO or DINOv2; subsequent scaling efforts include Prov-GigaPath, H-Optimus-0, and Virchow2.Source: §2 · pp. 2–3

Scaling and its limits

  • Scaling has not consistently improved histology representations or biomarker prediction: reported saturation, parity, and limited gains challenge the assumption that larger models are always better [35] [40].Source: §2 · p. 3

Study aim

  • The study compares Phikon-v2 (ViT-L) with Phikon (ViT-B) and 14 public histology encoders to examine DINOv2 versus iBOT under joint dataset-size and model-size scaling.The stated scaling comparison is 10x in pre-training dataset size and 4x in model size.Source: §2 · p. 3

3 Materials & Methods

The methods assemble a heterogeneous public pre-training collection, train a DINOv2 ViT-L, and evaluate frozen feature extractors with a shared slide-level MIL protocol. The benchmark uses external validation cohorts, compares 14 public encoders, and reports AUC with corrected statistical tests.

3.1 Training dataset

  • PANCAN-XL combines 58,359 WSIs from 132 public datasets and four internal datasets, spanning over 30 cancer sites and normal tissues; 456,060,584 tiles were extracted at 20x magnification.The collection includes varied tissue sources, stains, and over 50 scanner types.Source: §3.1 · p. 3

3.2 Model and pre-training setup

  • Phikon-v2 uses a randomly initialized ViT-L trained with DINOv2 on PANCAN-XL tiles; the released iteration was taken at 100,000 iterations after seeing 400 million tiles.Training ran for 250,000 iterations overall; the selected model had seen 93% of PANCAN-XL.Source: §3.2 · pp. 3–4

3.3 Evaluation setup

  • Frozen feature extractors encode sampled WSI tiles, and ABMIL aggregates their features into slide-level predictions; most tasks use 5,000 tiles per slide, while ISUP uses 400.The protocol trains an attention-based, one-layer gated MIL model.Source: §3.3 · p. 4
  • For each task, five stratified splits and five-fold cross-validation train five initialization-specific models per training split; external-cohort predictions are ensembled across 25 models.A one-shot retraining strategy is also evaluated against this ensemble protocol.Source: §3.3 · pp. 4–5

3.4 Evaluation tasks

  • The eight WSI-level tasks span five cancer sites, with external validation cohorts kept free of overlap with PANCAN-XL despite overlap in downstream training cohorts.Source: §3.4 · p. 5
  • Other tasks predict breast ER and PR status, glioma IDH1 histomolecular subtype, and prostate ISUP grade.Source: §3.4 · p. 6
  • The benchmark includes kidney subtype classification, breast lymph-node metastasis detection, colorectal MSI prediction, and breast HER2 prediction.Source: §3.4 · pp. 5–6

3.5 Comparisons & baselines

  • Phikon-v2 is compared with 14 recent public feature extractors; the natural-image baseline is a DINOv2-trained ViT-L/14 distilled from a ViT-G/14.Table 1 organizes the evaluated models by architecture, self-supervised pre-training scheme, and dataset.Source: §3.5 · p. 6, Table 1 · p. 7

4 Results

Across external slide-level evaluations, GigaPath and H-Optimus-0 lead overall, while model rankings vary by task and a specialized model ranks first on colorectal MSI prediction. Ensembling improves AUC over one-shot retraining across extractors and tasks.

4.1 ViT-L+ histology encoders stand out

  • Phikon-v2 and UNI rank first on 2 of 7 tasks and second on 3 of 7, and both outperform Phikon by almost 2 AUC points (p<1e-4).Both use ViT-L and DINOv2; Phikon-v2 uses 400M tiles and UNI 100M tiles.Source: §4.1 · pp. 7–8
  • GigaPath ranks first on 7 of 10 external cohorts, while H-Optimus-0 ranks first on 4 of 7.The pairwise comparison explains the absence of a statistical difference between H-Optimus-0 and GigaPath.Source: §4.1 · p. 7, Table 2 · p. 8, Table 3 · p. 8

4 Results

  • Pairwise Fisher-combined corrected p-values assess overall model superiority while reducing penalties for strong models that underperform on only a few tasks.Table 3 reports these comparisons; grey cells denote no statistical significance at the 95% confidence level.Source: §4 · p. 7, Table 3 · p. 8

4.1 ViT-L+ histology encoders stand out

  • Performance varies across evaluation datasets and tasks: Phikon-v2 leads on Herohe and MSI but performs poorly on IDH1, while GigaPath and UNI decline on Herohe after strong Bcnb results.The authors argue that robust evaluation needs multiple real-world datasets per task and candidate models, and note that pre-training samples' influence needs further study.Source: §4.1 · pp. 8–9

4.3 Ensembling yields consistent improvements over one-shot retraining

  • +1.75 AUC: ensembling improves over one-shot retraining across feature extractors and tasks (p<0.001).The authors report statistically significant gains for individual extractors except Lunit-DINO (B/8) and REMEDIS.Source: Figure 2 · p. 9, §4.3 · p. 10

4.2 Don’t set aside specialized models

  • The iBOT (B/16) Coad model ranks first on three external colorectal MSI cohorts despite being 13 times smaller and pre-trained on 350 times fewer histology images than Virchow2.The authors conclude that scaling alone is not a systematic solution; DINOv2 slightly decreased performance versus iBOT for this lighter model.Source: §4.2 · p. 9, Table 4 · p. 10

5 Conclusion

Phikon-v2 performs competitively across eight slide-level biomarker tasks on external cohorts, while the conclusion highlights benefits and limits of scaling and general-purpose models. The authors also call for evaluation that better reflects clinical domain shifts.

5 Conclusion

  • Phikon-v2 demonstrates competitive performance across eight slide-level biomarker tasks on five external datasets unseen during pre-training.The model is publicly released under a non-commercial license.Source: §5 · p. 10
  • DINOv2 scaling improved over iBOT, but iBOT-only scalability on PANCAN-XL was not assessed, and other masked-image-modeling methods may also improve results.Source: §5 · p. 10
  • In-domain self-supervised pre-training benefits feature extractors over out-of-domain pre-training, even for models 30 times smaller.Source: §5 · p. 10
  • GigaPath, H-Optimus-0, Phikon-v2, UNI, and Virchow2 support scaling benefits, but remain general-purpose across downstream tasks and target organs.The authors identify task- and organ-specific fine-tuning or distillation as a focus for clinical deployment; feature-extractor ensembling brings a significant computational bottleneck.Source: §5 · p. 10
  • The authors note that robustness to slide-preparation and acquisition shifts remains under-assessed, and call for real-world open evaluations across external validations, domain shifts, and population bias.Source: §5 · p. 11

Computing resources

The acknowledgments identify the HPC allocation supporting the work and the sources and support for TCGA, PAIP, and GTEx data.

Computing resources

Data sources

  • The data acknowledgments cover TCGA, PAIP pathology images and annotations prepared by Seoul National University Hospital, and GTEx data retrieved on 07/01/2023.PAIP preparation was supported by a Korea Health Technology R&D Project grant; GTEx support came from NIH offices and institutes.Source: Data sources · p. 11

Supplementary data

The supplementary material lists pre-training hardware and cohort details for PANCAN-XL, alongside its reported total WSI count and availability.

Supplementary data

  • Phikon-v2 pre-training used 32 nodes, each with four 32GB NVIDIA V100 GPUs.Source: Supplementary data · p. 16
  • The supplementary cohort listings name datasets and slide counts across multiple tissue sites, including kidney, lung, breast, and colorectal cohorts.Source: Supplementary data · pp. 17–18
  • PANCAN-XL comprises 58,359 WSIs, and the listed datasets are publicly available unless otherwise specified.Source: Supplementary data · p. 19
Loading 2409.09173v1…