PaperlayerView exact revision on arXiv

Source-linked AI summary

Benchmarking Self-Supervised Learning on Diverse Pathology Datasets

Mingu Kang, Heon Song, Seonwook Park, Donggeun Yoo, Sérgio Pereira

arXiv:2212.04690v2cs.CVcs.LG

TL;DR

Pathology models require scarce, costly expert annotations, while large unlabeled datasets are available and SSL methods had not been systematically compared or adapted for this domain. The paper benchmarks four SSL methods using large-scale pathology pre-training and pathology-specific techniques, finding consistent advantages over ImageNet pre-training across tasks and label regimes. It also extends SSL to nuclei instance segmentation and reports improvements under diverse settings.

  • Problem

    Pathology annotation is scarce and expensive, despite abundant unlabeled data, and principled comparisons of SSL methods and pathology adaptations were lacking.

  • Method

    The study evaluates four representative SSL methods with large-scale pathology pre-training across two downstream tasks and five datasets, incorporating pathology-specific techniques.

  • Results

    Domain-aligned pathology pre-training consistently out-performs ImageNet pre-training across standard evaluations and low-label regimes, including nuclei instance segmentation.

  • Takeaways & Limitations

    Practitioners should prioritize large, domain-aligned pathology datasets and incorporate pathology-specific knowledge into SSL pre-training rather than focusing on a single SSL method.

  • Takeaways & Limitations

    The study notes that stain augmentation can clash with color distortion because both alter color statistics.

Abstract

from arXiv · show

Computational pathology can lead to saving human lives, but models are annotation hungry and pathology images are notoriously expensive to annotate. Self-supervised learning has shown to be an effective method for utilizing unlabeled data, and its application to pathology could greatly benefit its downstream tasks. Yet, there are no principled studies that compare SSL methods and discuss how to adapt them for pathology. To address this need, we execute the largest-scale study of SSL pre-training on pathology image data, to date. Our study is conducted using 4 representative SSL methods on diverse downstream tasks. We establish that large-scale domain-aligned pre-training in pathology consistently out-performs ImageNet pre-training in standard SSL settings such as linear and fine-tuning evaluations, as well as in low-label regimes. Moreover, we propose a set of domain-specific techniques that we experimentally show leads to a performance boost. Lastly, for the first time, we apply SSL to the challenging task of nuclei instance segmentation and show large and consistent performance improvements under diverse settings.

1. Introduction

Computational pathology can improve cancer diagnosis and treatment but is constrained by scarce, expensive annotations. This study benchmarks four SSL methods on diverse pathology tasks and develops pathology-specific adaptations.

  • Computational pathology supports objective, large-scale analysis and triaging, but pathology annotation requires specialist expertise and remains scarce.
  • Self-supervised learning can exploit abundant unlabeled pathology data to benefit downstream tasks with limited annotations.
  • The authors propose data curation and augmentation techniques tailored to pathology-specific differences in orientation, color, and field-of-view.
  • The study evaluates MoCo v2, SwAV, Barlow Twins, and DINO using large-scale pathology pre-training across classification and nuclei instance segmentation.
  • Domain-aligned SSL pre-training improves over ImageNet pre-training across diverse downstream tasks and remains beneficial in limited-label settings.

2. Related Work

Prior SSL work spans contrastive, non-contrastive, clustering, and Vision Transformer paradigms, but pathology lacked a common comparison across methods and tasks. This study establishes such a benchmark and adds pathology-specific augmentation techniques.

  • Contrastive Learning: Contrastive learning brings augmented views together while separating instances, using large batches or memory banks to obtain diverse negative pairs.
  • Non-contrastive Learning: Non-contrastive methods learn from different augmented views without negative pairs, allowing operation with smaller batch sizes.
  • Clustering: Clustering-based SSL discriminates among clusters of image representations rather than explicit image pairs.
  • SSL with Vision Transformer: The benchmark includes DINO as a representative Vision Transformer-based SSL method alongside methods from the other major paradigms.
  • Unlike earlier studies centered on one approach or one architecture family, this work compares representative methods across diverse tasks, including nuclei instance segmentation.

3. Self-supervised Pre-training for Pathology

Pathology images differ from natural images in orientation, color, and field-of-view, motivating domain-specific SSL adaptations. The proposed approach uses realistic stain augmentation, multiple magnifications, and large unlabeled datasets.

  • 3.1. Differences to Natural Images: Pathology images lack canonical orientation, have limited color variation, and require interpretation across different field-of-views.
  • 3.2. Techniques to Adapt SSL for Pathology: Random vertical flips are introduced because vertically flipped pathology images remain plausible.
  • 3.2. Techniques to Adapt SSL for Pathology: RandStainNA models channel-wise statistics across HSV, Lab, and HED color spaces to produce pathology-relevant stain variations.
  • 3.2. Techniques to Adapt SSL for Pathology: RandStainNAGMM fits a 10-component Gaussian Mixture Model to channel-wise color statistics, modeling covariances and multimodality.
  • 3.2. Techniques to Adapt SSL for Pathology: The main augmentation recipe combines RandStainNAGMM with random grayscale and weaker color jittering because stain augmentation can clash with color distortion.
  • 3.2. Techniques to Adapt SSL for Pathology: The pre-training dataset combines patches from 20× and 40× objective magnifications to support tasks requiring different fields-of-view.

4. Experiment Setup

The study pre-trains SSL models on large-scale pathology images and evaluates them across classification and nuclei instance segmentation, using standard downstream protocols and metrics.

  • Pre-training Dataset: 20,994 TCGA WSIs and 15,672 TULIP WSIs provide 32.6M unlabeled 512 × 512 patches for pre-training.The data covers 20× and 40× objective magnifications; standard experiments use TCGA only.
  • Downstream Tasks: Four classification datasets cover breast and colorectal cancer tasks, while nuclei instance segmentation is evaluated separately.Classification uses BACH, CRC, MHIST, and PCam; segmentation uses the Hover-Net architecture with a pre-trained backbone.
  • Pre-training Details: Four SSL methods use ResNet-50 backbones, while DINO uses ViT-Small with 16×16 or 8×8 patches.Pre-training follows each method’s recipe and is distributed across 64 NVIDIA V100 GPUs.
  • Downstream Evaluation: Classification datasets are split into training, validation, and test sets, with hyperparameters selected on validation data and performance reported on test data.Segmentation follows Hover-Net preprocessing and training schemes for reproducibility and fair comparison.
  • Evaluation Metrics: Top-1 accuracy evaluates image classification under linear and fine-tuning protocols, while panoptic quality evaluates nuclei instance segmentation.PQ accounts for both detection and segmentation quality for each instance.
  • Evaluation Metrics: PQ matches predicted and ground-truth nuclei instances using IoU above 0.5, then incorporates matched overlap, false positives, and false negatives.The study also uses multi-class PQ to measure instance segmentation and classification simultaneously.

5. Experimental Results

Experiments across classification and nuclei instance segmentation show that domain-aligned SSL pre-training generally outperforms ImageNet pre-training, while pathology-specific data and augmentation improve downstream performance.

  • Image classification: Linear evaluation typically favors self-supervised TCGA pre-training over supervised ImageNet pre-training, with Barlow Twins strong on BACH, PCam, and MHIST.
  • Image classification: Fine-tuning narrows the ImageNet-versus-TCGA gap, while MoCo v2 performs consistently well among CNN methods.
  • Nuclei instance segmentation: SSL pre-trained models outperform supervised ImageNet pre-training by a large margin on nuclei instance segmentation, especially given the difficulty of increasing mPQ.
  • Nuclei instance segmentation: DINO trained on TCGA outperforms ImageNet-trained DINO p=16 weights, while DINO p=8 exceeds CNN-based methods in linear evaluation.
  • Pre-training data: TCGA pre-training outperforms ImageNet pre-training across BACH, CRC, PCAM, and CoNSeP, while combining TCGA with TULIP yields the overall best performance.
  • Label efficiency and schedules: SSL retains its advantage with increasing labeled-data quantities and longer training schedules, including on CoNSeP nuclei segmentation.
  • Ablation studies: Pathology-specific augmentations, including stain augmentation and multiple FoVs, improve performance over natural-image augmentation schemes.

6. Discussion

The discussion concludes that pathology data should be used for SSL pre-training and that domain knowledge, rather than a single SSL method, is central to achieving strong performance.

  • Pathology pre-training consistently outperforms supervised ImageNet pre-training across comprehensive experiments and maintains an advantage on CoNSeP with longer schedules.
  • No SSL method is a clear winner; practitioners should prioritize curating large-scale, domain-aligned datasets.
  • Domain-specific techniques such as stain augmentation and patches from multiple FoVs improve performance beyond naively applying SSL to pathology data.

7. Conclusion and Future Work

This study presents a comprehensive evaluation of pathology-domain SSL, showing its value with limited labeled data and proposing pathology-informed techniques that improve performance. It also identifies opportunities for future gains through domain-specific augmentation strategies.

  • The study evaluates 4 representative SSL methods using up to 33 million pathology image patches across 2 downstream tasks and 5 datasets.
  • Large-scale domain-aligned pre-training is helpful for pathology when labeled data is limited, fine-tuning schedules are longer, or pre-training data is larger and more diverse.
  • Pathology-specific data curation and augmentation techniques are integrated into SSL pre-training and result in performance improvements.
  • The supplementary material documents downstream datasets, pre-training and evaluation procedures, limited-label experiments, and additional analyses of training duration and magnification.

C. Pre-training for more epochs (Section 5)

Longer SSL pre-training generally provides marginal downstream gains, although SwAV benefits consistently and certain settings show method-specific improvements. The study therefore treats 200 ImageNet epochs as generally adequate for evaluating pathology SSL.

  • SwAV is the only evaluated method that clearly benefits from longer pre-training in fine-tuning, especially on BACH, MHIST, and CoNSeP.The benefit persists across different learning schedules.
  • Table B.1 reports Top-1 accuracy for image classification under linear and fine-tuning protocols across different pre-training durations.
  • Longer pre-training yields marginal improvements or performance on par with 200-epoch models for most other methods and tasks.DINO shows slightly improved image classification, while nuclei instance segmentation remains on par.
  • 200 ImageNet epochs can be sufficient for satisfactory downstream performance, particularly for MoCo v2, Barlow Twins, and DINO.
  • Table C.1 reports mPQ for nuclei instance segmentation under linear and fine-tuning protocols across different pre-training durations.

D. Pre-training Stability with Different Magnifications (Section 5.6)

Pre-training stability depends on objective magnification: combining 20× and 40× pathology images produces faster convergence and more stable Barlow Twins training than using a single magnification.

  • Using a single magnification produces unstable losses before convergence during Barlow Twins pre-training.The loss begins converging after approximately 4,000 and 7,000 iterations for the reported magnifications.
  • Combining 20× and 40× images results in quick convergence and stable Barlow Twins pre-training.
  • ViT input resolution is increased to 272 × 272 through positional embedding interpolation for nuclei instance segmentation.The chosen resolution is divisible by both 16 and 8.

F. Further Data Augmentation Ablation Study (Section 5.6)

Pathology-informed augmentation techniques consistently improve downstream performance over augmentations designed for natural images. The ablation study demonstrates this effect in challenging nuclei instance segmentation evaluations.

  • The proposed augmentation techniques consistently and significantly improve linear evaluation performance compared with the baseline natural-image augmentations.
  • The augmentation study uses nuclei instance segmentation as a practical and challenging fine-tuning setting.
  • Table E.1 reports mPQ for nuclei instance segmentation under different input resolutions and both linear and fine-tuning protocols.
  • Attention maps show that self-supervised ViTs identify cells and morphological phenotypes, with different attention heads focusing on epithelial, inflammatory, and fibroblast cells.

H. Qualitative Results of Nuclei Instance Segmentation (Section 5.2)

The qualitative comparison evaluates linear-evaluation models using supervised ImageNet or self-supervised TCGA pre-training for nuclei instance segmentation. Domain-aligned pre-training captures foreground cells and classifies them more accurately than ImageNet pre-training.

  • The experiment compares supervised ImageNet pre-training with self-supervised TCGA pre-training under linear evaluation for nuclei instance segmentation.The comparison is qualitative and focuses on the effect of domain-aligned pre-training.
  • Domain-aligned pre-training captures foreground cells and accurately classifies them, unlike the model using ImageNet-pre-trained weights.

I. Slide-level Evaluation

The paper includes a preliminary slide-level breast-cancer-metastasis classification experiment on Camelyon16, while explicitly placing slide-level classification outside the main work's scope. It uses CLAM's configuration and compares Supervised (IN) with MoCo v2 (TC+TU) features, but the supplied passage does not state the outcome.

  • Slide-level classification is outside the scope of the main work, but a preliminary breast-cancer-metastasis experiment is conducted on Camelyon16 WSIs.Models are trained and tested using the same configuration as CLAM.
  • The preliminary experiment extracts WSI features using Supervised (IN) and MoCo v2 (TC+TU) pre-trained weights.
  • The supplied passage does not report the experiment's result.
Loading 2212.04690v2…