PaperlayerView exact revision on arXiv

Source-linked AI summary

Distilling foundation models for robust and efficient models in digital pathology

Alexandre Filiot, Nicolas Dop, Oussama Tchita, Auriane Riou, Rémy Dubois, Thomas Peeters, Daria Valter, Marin Scalbert, Charlie Saillard, Geneviève Robin, Antoine Olivier

arXiv:2501.16239v3cs.CV

TL;DR

Foundation models improve computational pathology performance but bring increased computational cost, and their robustness to preparation and digitization variation has received less attention. This paper distills H-Optimus-0 into the smaller H0-mini. The model is competitive with larger foundation models and shows improved robustness to staining and scanner variation.

  • Problem

    Foundation models improve downstream computational pathology performance, but their increased computational cost and comparatively understudied robustness to preparation and digitization variation remain concerns.

    Source: §1 · p. 2

  • Method

    The study distills the H-Optimus-0 foundation model into H0-mini, a smaller student model, using class-token and patch-token supervision.

    Source: §1 · p. 2, Distillation setup · p. 3

  • Results

    H0-mini is competitive with much larger models across benchmarks and improves robustness to staining and scanner variation, significantly outperforming H-Optimus-0 on BreastBm performance and robustness metrics.

    Source: §3.1 · p. 6, §3.2 · p. 8, §5 · p. 9

  • Takeaways & Limitations

    The authors conclude that distillation can produce a competitive, smaller pathology model with improved robustness, supporting future research toward clinical adoption.

    Source: §5 · p. 9

    Distillation requires the teacher model and takes 1.7 times longer to train than a similar ViT-Base trained from scratch.

    Source: §4 · p. 8

Abstract

from arXiv · show

In recent years, the advent of foundation models (FM) for digital pathology has relied heavily on scaling the pre-training datasets and the model size, yielding large and powerful models. While it resulted in improving the performance on diverse downstream tasks, it also introduced increased computational cost and inference time. In this work, we explore the distillation of a large foundation model into a smaller one, reducing the number of parameters by several orders of magnitude. Leveraging distillation techniques, our distilled model, H0-mini, achieves nearly comparable performance to large FMs at a significantly reduced inference cost. It is evaluated on several public benchmarks, achieving 3rd place on the HEST benchmark and 5th place on the EVA benchmark. Additionally, a robustness analysis conducted on the PLISM dataset demonstrates that our distilled model reaches excellent robustness to variations in staining and scanning conditions, significantly outperforming other state-of-the art models. This opens new perspectives to design lightweight and robust models for digital pathology, without compromising on performance.

1 Introduction

Foundation models improve downstream computational pathology performance, but their scale increases computational cost, while robustness to preparation and digitization variation remains less studied. This work distills H-Optimus-0 into H0-mini, reporting competitive benchmark performance and improved robustness to staining and scanning variation.

1 Introduction

  • Scaling foundation models improves downstream performance but increases computational cost, and robustness to sample-preparation or digitization variation is critical for clinical deployment.The introduction notes that robustness has received less attention than performance.Source: §1 · p. 2
  • H0-mini distills the billion-parameter H-Optimus-0 into an 86-million-parameter ViT-Base, with competitive benchmark performance and improved robustness to staining and scanning variation.The authors evaluate it on public benchmarks and robustness datasets.Source: §1 · p. 2
  • Distillation is established in computer vision but remains relatively scarce for foundation models in digital pathology.Prior work includes simultaneous distillation of three models in GPFM and a distilled ViT-Small from Virchow2G.Source: Related work · p. 2

2 Material and methods

The methods distill H-Optimus-0 into H0-mini using class-token and patch-token objectives, then evaluate performance and robustness on public benchmarks and pathology datasets. The evaluation includes cross-staining and cross-scanner comparisons, as well as breast biomarker prediction across cohorts.

2.1 Pre-training setup

  • H0-mini is trained from H-Optimus-0 using DINO class-token and iBOT patch-token distillation objectives on augmented image views.The iBOT loss is applied to all patch scores without masking; training uses a spare exponential moving average of the student.Source: Distillation setup · p. 3
  • Distillation uses 43M tiles (224x224 at 20x magnification) from 6,093 TCGA slides covering 16 cancer sites, enabling fair comparison with Phikon, a ViT-B feature extractor pretrained from scratch on the same data with iBOT.The dataset is described as relatively small compared with typical pre-training datasets; the cited setup performs distillation on a 1B-tile dataset.Source: Pre-training datasets · p. 3

2.2 Evaluation setup

  • Performance is evaluated on EVA’s eight patch- and slide-level classification and segmentation tasks and HEST gene-expression prediction tasks across nine indications.HEST evaluation uses 50 highly variable genes per task and ridge regression after PCA to 256 components.Source: Performance benchmarks · p. 4
  • PLISM robustness evaluation uses 91 registered whole-slide images from 46 tissue types, 13 H&E staining conditions, and seven scanners.Tiles are compared across matching slide pairs using cosine similarity and top-k accuracy, aggregated across slide pairs.Source: PLISM robustness dataset · p. 4
  • Breast biomarker evaluation covers ER, PR, HER2, and germline BRCA status across private cohorts with varied stains and scanners.Mean-pooled tile features feed logistic regression; performance uses AUC and robustness uses CCC between predictions for the same tumor blocks.Source: BreastBiomarker (BreastBm) performance… · pp. 4–5

3 Results

H0-mini performs competitively across EVA and HEST, ranks fourth across their 17 combined tasks, and shows strong robustness on PLISM and BreastBm.

3.1 Performance evaluation

  • On EVA, H0-mini is competitive with larger state-of-the-art models across patch classification, slide classification, and segmentation.Six models average between 0.78 and 0.79; Kaiko-B/8 reaches equivalent performance with a smaller patch size that entails more computation.Source: §3.1 · pp. 5–6, Table 1 · p. 6
  • H0-mini ranks 4th across 17 combined HEST and EVA tasks, statistically outperforming UNI and Phikon models but not UNI2-h or H-Optimus-0.Virchow2 was not significantly different (p = 0.29).Source: §3.1 · p. 6
  • On HEST, H0-mini significantly outperforms same-data Phikon and surpasses larger Virchow2 and Gigapath on average.Table 2 reports Pearson correlations.Source: §3.1 · p. 6, Table 2 · p. 7

3.2 Robustness evaluation

  • On PLISM, H0-mini has the highest top-10 accuracy for scanning and joint scanning-and-staining variation, while staining variation is harder than scanning variation.Cosine similarity alone does not capture retrieval robustness: PLIP and Phikon score highly on cosine similarity but fail at retrieval.Source: §3.2 · pp. 6–7, Table 3 · p. 7
  • On BreastBm, H0-mini significantly outperforms H-Optimus-0 on every performance and robustness metric, and simple-average pooling results suggest feature-space robustness translates downstream.Performance is measured by AUC and robustness by CCC; the reported comparison has p < 10^-4.Source: Table 4 · p. 8, §3.2 · p. 8

4 Discussion

The discussion identifies teacher dependence and training cost as limitations of distillation, while noting open questions about tuning and robustness evaluation.

4 Discussion

  • The final projection head of the teacher is required for distillation; whether it can be omitted without changing downstream performance or robustness remains open.The authors also suggest learning projection heads during pre-training as a possibility to explore.Source: §4 · p. 8
  • Distillation takes 1.7x longer to train than a similar ViT-Base from scratch because it requires the teacher, despite improved resulting performance and robustness.Source: §4 · p. 8
  • The study did not perform extensive hyperparameter optimization, leaving distillation-specific settings such as more global crops or smaller student patches to explore.Source: §4 · p. 8
  • A more comprehensive understanding of model behavior may require high-quality datasets scanned and stained in multiple ways.Source: §4 · p. 8

5 Conclusion

The authors conclude that distillation produces a competitive smaller model and improves robustness to scanner and staining changes. They see public release as a way to broaden access for researchers with limited computational resources.

5 Conclusion

  • Distillation yields a competitive smaller model that closes the performance gap with larger foundation models on downstream applications.Source: §5 · p. 9
  • The authors hope public release will democratize pathology foundation-model use for researchers with limited computational resources.Source: §5 · p. 9
  • Distilled models have greater feature invariance to scanner and staining changes; the authors describe this as a first demonstration that distillation further improves robustness.Source: §5 · p. 9

A.1 Distillation hyperparameters

The appendix points to Table 5 for the main distillation hyperparameters.

A.1 Distillation hyperparameters

  • Table 5 lists the main hyperparameters used in distillation.Source: §A.1 · p. 11

A.2 Mathematical details of robustness evaluation metrics

The appendix defines cosine similarity for tile feature vectors and describes slide-pair metrics that summarize matched-tile similarity and retrieval rank.

A.2 Mathematical details of robustness evaluation metrics

Related papers

Loading 2501.16239v3…