PaperlayerView exact revision on arXiv

Source-linked AI summary

Self supervised contrastive learning for digital histopathology

Ozan Ciga, Tony Xu, Anne L. Martel

arXiv:2011.13971v2eess.IVcs.CV

TL;DR

Medical image analysis needs methods that learn from unlabeled data because expert annotations are scarce and expensive. The paper applies SimCLR contrastive self-supervision to 57 unlabeled histopathology datasets, finding that histopathology pretraining outperforms ImageNet pretraining across several downstream settings and benefits from diverse data.

  • Problem

    Medical image analysis is constrained by scarce, expensive expert annotations, motivating learning methods that can use unlabeled data.

  • Method

    The paper pretrains residual networks with SimCLR by contrasting augmented views of the same histopathology image and separating other batch images using NT-Xent.

  • Results

    Histopathology pretraining is superior to ImageNet on classification and regression across all settings, comparable for segmentation with ResNet 18 and 34, and worse for segmentation with larger networks.

  • Takeaways & Limitations

    The success of contrastive pretraining depends heavily on unlabeled training-set diversity, while training-site choice has no substantial effect on learned-representation quality.

  • Takeaways & Limitations

    Contrastive learning can produce noisy representations when histopathology images look similar but differ in small, task-important nuances.

Abstract

from arXiv · show

Unsupervised learning has been a long-standing goal of machine learning and is especially important for medical image analysis, where the learning can compensate for the scarcity of labeled datasets. A promising subclass of unsupervised learning is self-supervised learning, which aims to learn salient features using the raw input as the learning signal. In this paper, we use a contrastive self-supervised learning method called SimCLR that achieved state-of-the-art results on natural-scene images and apply this method to digital histopathology by collecting and pretraining on 57 histopathology datasets without any labels. We find that combining multiple multi-organ datasets with different types of staining and resolution properties improves the quality of the learned features. Furthermore, we find using more images for pretraining leads to a better performance in multiple downstream tasks. Linear classifiers trained on top of the learned features show that networks pretrained on digital histopathology datasets perform better than ImageNet pretrained networks, boosting task performances by more than 28% in F1 scores on average. These findings may also be useful when applying newer contrastive techniques to histopathology data. Pretrained PyTorch models are made publicly available at https://github.com/ozanciga/self-supervised-histopathology.

1. Introduction

The paper applies contrastive self-supervised learning to digital histopathology to learn generalizable features from unlabeled images and reduce reliance on scarce annotations.

  • Medical image analysis faces scarce, expensive expert annotations, motivating methods that use unlabeled data.
  • The study applies SimCLR with residual networks to multiple unlabeled, multi-organ histopathology datasets and evaluates learned representations across downstream tasks.
  • Each image is augmented twice, with same-source views brought together and different-source patches pushed apart using NT-Xent.
  • Pretraining with unlabeled histopathology images improves classification, regression, and segmentation performance over ImageNet pretraining or training from scratch.

2. Related work

Earlier self-supervised methods often rely on natural-image structure, whereas contrastive learning uses instance-level consistency and is applicable to histopathology images.

  • Earlier unsupervised approaches often target cell-level structures or specific resolution and staining properties, limiting their broader applicability.
  • Self-supervised learning generates its learning signal from raw input data and can use standard architectures such as residual networks.
  • Context-based tasks such as tile rearrangement or rotation prediction are not directly applicable to histology because its structures can form multiple valid arrangements.
  • Contrastive methods compare representations of transformed views from the same image and do not require spatial regularity between or within instances.
  • SimCLR avoids custom networks and memory banks by using large minibatches to provide more negative samples during training.

3. The method

The method learns histopathology representations by contrasting augmented views of the same image while separating other batch images in a projected embedding space.

  • SimCLR maximizes agreement between two stochastically augmented views of each image through an encoder and auxiliary projection layer.
  • The NT-Xent loss makes representations of other images in the batch dissimilar from the reference image.
  • The temperature parameter weighs examples to support hard negative mining, while similarity is computed between normalized vectors.
  • The projection layer maps pre-activation features into a lower embedding space, and comparing projected outputs was more effective for representation learning than comparing pre-activation outputs directly.
  • The approach exploits variability in staining, resolution, and tissue type to construct visually diverse histopathology pretraining data.

4. Experiments

The experiments use diverse histopathology data spanning many organs and datasets, emphasizing diversity across datasets as a basis for pretraining.

  • The datasets span numerous organs and tissue types, including blood, breast, lymph, colon, bone, prostate, liver, lung, skin, brain, and ovary.
  • Images evenly sampled across all 60 datasets appear more diverse than images sampled from one dataset containing multiple whole-slide images.
  • The pretraining collection includes 57 datasets, comprising 22 image-patch datasets and 35 whole-slide-image datasets.
  • Most datasets use hematoxylin and eosin staining and the highest resolution of 0.25µm/pixel, commonly called 40×.

Validation datasets.

The study evaluates pretrained networks across classification, segmentation, and regression datasets, comparing initialization and training strategies under a standardized validation setup.

  • Validation datasets.: The evaluation covers five classification, two segmentation, and one regression dataset.Classification includes breast, lymph-node, colorectal, and prostate tasks; segmentation includes breast and colon-tumor whole-slide tasks; regression predicts cancer cellularity.
  • Validation datasets.: Pretrained networks are compared with randomly initialized and ImageNet-pretrained ResNet-18, ResNet-34, ResNet-50, and ResNet-101 models.Segmentation uses a UNet with a pretrained encoder and randomly initialized decoder.
  • Validation datasets.: The study assesses learned representations by freezing each residual network and training only a linear classifier or regressor.This last-layer setting is omitted for segmentation because the UNet-like decoder has a comparable number of trainable parameters to the encoder.
  • Validation datasets.: Each experiment runs for 100 epochs using Adam, with dataset splits of 50% training, 25% validation, and 25% testing.Weight decay is 0.00001 for ResNet-18 and 0.0001 for other models.
  • Validation datasets.: Macro F1 score is used as the validation metric for classification and segmentation tasks.

5. Results

The results evaluate self-supervised, ImageNet, and random initialization across classification, regression, segmentation, data-efficiency, representation, clustering, resolution, and organ-transfer settings. Self-supervised pretraining generally improves downstream performance, particularly with limited labeled data, while broader and higher-resolution pretraining further helps.

  • Overall comparison: Self-supervised pretraining outperforms ImageNet initialization on classification and regression across all settings, while segmentation performance depends on network size.It is comparable to ImageNet for ResNet 18 and 34 segmentation, whereas ImageNet performs better for larger networks.
  • Limited labeled data: Over 40% higher performance than random initialization is observed on the NCT dataset when self-supervision uses only 5% of labeled images.Self-supervision remains superior to ImageNet and random initialization when more labeled data is used.
  • Unsupervised clustering: Learned representations support unsupervised clustering of whole-slide-image patches and can highlight regions of interest without labels.Clustering uses visual or morphological similarity in feature space to select representative patches and address class imbalance from random sampling.
  • Pretraining-set size: Larger unlabeled pretraining sets outperform smaller sets, but gains diminish with each order-of-magnitude increase.Average classification F1 differs by 7.4% between 0.01% and 0.1% of available images, versus around 2.1% between 1% and 10%.
  • Resolution: Higher-resolution pretraining tends to improve downstream tasks, and combining multiple resolutions further improves task performance.The evaluated resolutions are 10×, 20×, and 40×, corresponding to 1µm/pixel, 0.50µm/pixel, and 0.25µm/pixel.
  • Transferability: Diverse pretraining datasets perform better across tasks than site-specific datasets with comparable numbers of pretraining images.Site-specific pretraining does not show a strong overall correlation with validation performance, although some datasets favor the matching tissue source.

6. Discussion

Segmentation benefits from self-supervised pretraining for smaller residual networks, whereas ImageNet initialization performs better for larger ones. Tissue type alone does not predict downstream performance, and diverse pretraining data may be important for richer representations.

  • Segmentation: Self-supervision outperforms ImageNet initialization for segmentation with ResNet 18 and 34, but ImageNet performs better with ResNet 50 and 101.The authors hypothesize that larger decoder parameter counts diminish the effect of encoder pretraining.
  • Pretraining data diversity: Tissue type used for pretraining was not correlated with downstream performance on the validation datasets.Performance degraded compared with using all tissue types.
  • Pretraining data diversity: The authors attribute this degradation to limitations of contrastive representations, sensitivity to staining and resolution, and insufficient diversity for encoding richer features.These are presented as possible factors rather than established mechanisms.

7. Conclusion

The paper demonstrates that simple contrastive self-supervision can improve learned features across histopathology classification, regression, and segmentation tasks. Its results emphasize diverse unlabeled data and domain-specific augmentation choices, while identifying representation noise for visually similar but diagnostically distinct patches.

  • Conclusion: Self-supervised pretraining improves performance on multiple downstream tasks, including classification, regression, and segmentation.The method is reported to achieve comparable performance to ImageNet pretraining without additional complexity.
  • Conclusion: The study uses 23 image datasets with over 206 thousand patches and approximately 25 thousand gigaresolution images in 35 whole-slide-image datasets.The authors describe this as the first study using such a large number of digital histopathology images for pretraining.
  • Conclusion: Contrastive pretraining relies heavily on diversity in the unlabeled training set rather than merely the number of images.The site from which training images were extracted did not have a substantial effect, according to the supplied passage.
  • Conclusion: The authors focused on a simple contrastive method and tested multiple settings to characterize contrastive training's capabilities and limitations for histopathology images.They suggest the insights may remain useful for future self-supervised work while the framework fundamentals remain unchanged.
  • Conclusion: Visually distinguishable image pairs help learn salient representations, whereas visually similar patches with small histopathological nuances can produce noisy representations.The authors highlight small cell tumors occupying only a small portion of a patch as an example.

Appendix C. Hyperparameter and suitable augmentation selection

The appendix evaluates large-batch optimization and augmentation choices for SimCLR pretraining on histopathology images. Lars with batch size 512 and temperature 0.1 performed best, while stronger color jittering and aggressive cropping generally improved representations.

  • Hyperparameter selection: Approximately 40 thousand images from 57 unlabeled datasets were used to select hyperparameters using average macro F1 across five validation datasets.The experiments compared temperatures, batch sizes, and optimizers.
  • Augmentation selection: The augmentation ablations modify natural-scene settings by comparing randomly resized crop minima of 1%, 5%, and 25%, with 100% as the maximum.Each experiment was trained for 1000 epochs and evaluated on independent validation datasets.
  • Hyperparameter selection: Lars was selected over Lamb after outperforming it in most settings, while Adam was discarded because it failed to converge at batch sizes ≥256.Batch size 512 performed best for both remaining optimizers, and temperature 0.1 performed best for both.
  • Augmentation selection: A 1% crop outperformed 5% and 25% crops under medium and heavy color jittering, while medium jittering performed best overall by over 0.6% in F1 over the next-best setting.The authors associate stronger augmentations with better representations and propose a regularization explanation.
  • Evaluation: The appendix reports macro F1 averaged over five validation datasets for classification and describes validation datasets spanning classification, segmentation, and regression.The supplied captions identify the relevant datasets and metrics but do not provide the detailed table values.

Appendix D. Detailed results

The detailed-results appendix organizes downstream comparisons across self-supervised, ImageNet, and random initialization, while also examining linear classifiers, image resolution, pretraining-set size, and augmentation choices. Reported metrics include classification and segmentation F1 scores and regression mean L1 error.

  • Initialization comparisons: Table D.7 compares downstream networks initialized by self-supervised pretraining, ImageNet, or random initialization.It reports averaged classification F1, regression mean L1 difference, and averaged segmentation F1.
  • Initialization comparisons: Table D.8 evaluates linear classifiers trained on features from self-supervised, ImageNet-initialized, and randomly initialized networks.The same classification, regression, and segmentation metrics are reported.
  • Resolution: Table D.9 examines whether pretraining at 10×, 20×, or 40× resolution affects performance on all available validation datasets.These resolutions correspond to 1, 0.50, and 0.25 µm/pixel, respectively.
  • Pretraining-set size: Table D.10 studies how the number of pretraining images affects classification, regression, and segmentation performance.The reported metrics are averaged classification F1, regression mean L1 difference, and averaged segmentation F1.
  • Augmentations: Table D.11 reports macro F1 scores for selecting suitable augmentations in digital histopathology self-supervised learning.The baseline includes rotations, axis flips, Gaussian blurring, and light color jittering.

Appendix E. Comparison to other self-supervised methods

This section compares SimCLR with earlier self-supervised techniques using a common ResNet50-based setup and 1% of available pretraining data.

  • Each compared technique uses 1% of all available pretraining data.
  • The comparison repurposes the experimental segmentation network with a ResNet50 encoder for the autoencoder baseline.The autoencoder is trained with an image-reconstruction objective using L1 loss.
  • The evaluated alternatives include autoencoding, colorization, and CPCv2 alongside SimCLR.Colorization predicts RGB channels from grayscale images, while CPCv2 encodes small patches from larger images using a sliding window.

Appendix F.1. Pretraining datasets

This appendix table identifies the datasets used to pretrain the models.

  • The table lists the datasets used for model pretraining.
Loading 2011.13971v2…