Self-Path: Self-supervision for Classification of Pathology Images with Limited Annotations

Navid Alemi Koohbanani; Balagopal Unnikrishnan; Syed Ali Khurram; Pavitra Krishnaswamy; Nasir Rajpoot

Source-linked AI summary

Self-Path: Self-supervision for Classification of Pathology Images with Limited Annotations

Navid Alemi Koohbanani, Balagopal Unnikrishnan, Syed Ali Khurram, Pavitra Krishnaswamy, Nasir Rajpoot

arXiv:2008.05571v1Aug 12, 2020cs.CVeess.IV

TL;DR

Pathology images are plentiful but expensive to annotate exhaustively, limiting supervised learning. Self-Path uses a multi-task CNN with domain-specific self-supervised pretext tasks to learn useful representations from unlabeled data. Across three datasets, it achieves state-of-the-art semi-supervised performance with little labeled data and improves domain adaptation without target labels.

Problem
Exhaustive pathology-image annotation is difficult, while DNN performance depends on substantial annotated data.
Method
Self-Path is a multi-task framework combining tissue classification with self-supervised tasks based on contextual, multi-resolution, and semantic image features.
Results
Across three datasets, Self-Path achieves state-of-the-art semi-supervised performance with 1-2% annotation and competitive performance when the target domain has no annotations.
Takeaways & Limitations
Domain-specific self-supervision improves pathology classification in limited-annotation and domain-adaptation settings.

Abstract

from arXiv · show

While high-resolution pathology images lend themselves well to `data hungry' deep learning algorithms, obtaining exhaustive annotations on these images is a major challenge. In this paper, we propose a self-supervised CNN approach to leverage unlabeled data for learning generalizable and domain invariant representations in pathology images. The proposed approach, which we term as Self-Path, is a multi-task learning approach where the main task is tissue classification and pretext tasks are a variety of self-supervised tasks with labels inherent to the input data. We introduce novel domain specific self-supervision tasks that leverage contextual, multi-resolution and semantic features in pathology images for semi-supervised learning and domain adaptation. We investigate the effectiveness of Self-Path on 3 different pathology datasets. Our results show that Self-Path with the domain-specific pretext tasks achieves state-of-the-art performance for semi-supervised learning when small amounts of labeled data are available. Further, we show that Self-Path improves domain adaptation for classification of histology image patches when there is no labeled data available for the target domain. This approach can potentially be employed for other applications in computational pathology, where annotation budget is often limited or large amount of unlabeled image data is available.

I. INTRODUCTION

Computational pathology has abundant high-resolution image data but costly, incomplete annotations, motivating methods that exploit unlabeled images. Self-Path addresses this through domain-specific self-supervision for semi-supervised learning and domain adaptation.

Motivation: High-resolution pathology images provide extensive raw data, but effective DNN training depends on large, diverse annotated datasets.Digital scanning increases available pixels, not necessarily expert labels.
Motivation: WSI annotation is demanding because images are enormous, require inspection across magnifications, and depend on specialized pathologist expertise.Even labeling tumor regions or isolated cells may require part-by-part annotation.
Motivation: Semi-supervised learning can combine small labeled sets with larger unlabeled pools, while domain adaptation transfers knowledge from related source resources.Pathology-specific differences in tissue, tumors, and stain appearance complicate direct transfer across datasets.
Approach: Self-supervision uses labels inherent to the input data to learn representations for downstream pathology tasks without additional manual annotations.These labels can encode morphological, geometrical, and contextual image content.
Contributions: Self-Path is a flexible framework for pathology image classification with limited or no annotations, using three domain-specific tasks targeting contextual, multi-resolution, and semantic features.The paper investigates these tasks across three histopathology classification datasets.
Results: 1-2% annotation achieves state-of-the-art semi-supervised performance, while Self-Path also supports competitive target-task performance with no target annotations.The framework leverages related annotated resources for domain adaptation.

A. Related Work

Prior work uses pseudo-labeling, consistency regularization, adversarial domain alignment, and self-supervised pretext tasks, but pathology-specific evidence remains limited. The paper formulates semi-supervised learning and domain adaptation around labeled and unlabeled datasets with shared or distinct distributions.

Semi-supervised Learning: Pseudo-labeling and consistency regularization use limited labeled data alongside larger unlabeled pools for semi-supervised learning.MixMatch combines pseudo-labeling with temperature sharpening and mixup augmentation.
Semi-supervised Learning: Semi-supervised learning has been relatively underexplored in histopathology compared with the broader computer vision literature.The passage reports only six histopathology studies at the time of writing.
Domain Adaptation: Domain adaptation commonly uses adversarial training to align source and target feature distributions through a domain discriminator.The discriminator distinguishes domains while the feature generator is trained to confuse it.
Domain Adaptation: Most practical domain adaptation methods require target-domain labels, and unsupervised domain adaptation remains insufficiently established for histopathology.This limits direct application to settings where target annotations are unavailable.
Self-Supervision: Self-supervised learning trains representations with pretext labels inherent to the input, including reconstruction, generation, inpainting, and colorization tasks.The learned representations can support downstream classification, segmentation, and detection.
Problem Formulation: In semi-supervised learning, labeled and unlabeled samples share a distribution; in domain adaptation, labeled source and unlabeled target domains have related but distinct distributions.The source and target label spaces are defined as the same or approximately equivalent.

III. METHODS

Self-Path jointly learns the main classification task and auxiliary pretext tasks through a shared encoder. Its formulation supports both same-domain semi-supervised learning and transfer from labeled source data to unlabeled target data.

Framework: Self-Path uses multi-task learning to learn class-discriminative and domain-invariant features from limited annotated data.The framework supports domain-specific or domain-agnostic pretext tasks and adversarial or non-adversarial training.
Multi-task learning: Auxiliary tasks connect separate task heads to a shared encoder, and all tasks are optimized simultaneously with the main classification task.The shared encoder learns common features used across tasks.
Multi-task learning: The objective combines main-task and pretext-task losses, weighted by task coefficients, over labeled and unlabeled samples.The shared encoder, classifier, and pretext heads have separate parameter sets.
Learning settings: For semi-supervised learning, labeled and unlabeled data come from one domain; for domain adaptation, labeled data come from the source and unlabeled data from the target.This operational distinction matches the paper’s same-distribution versus distinct-distribution settings.

B. Self-Supervision

Self-Path uses domain-specific and domain-agnostic self-supervised pretext tasks with a shared encoder. These tasks derive supervision from image transformations and pathology structure, including magnification and nuclear morphology.

B. Self-Supervision: Self-Path uses multiple self-supervised tasks to learn representations from unlabeled pathology images for downstream classification.The framework supports both domain-specific and domain-agnostic tasks.
B. Self-Supervision: A transformation function g(x, r) applies a predefined transformation to image x, with r serving as its self-supervised label.The transformed image is supplied to the network as input.
B. Self-Supervision: The framework introduces histology-specific pretext tasks based on shape, morphology, arrangement, context, and location of nuclei and other tissue structures.These features are intended to provide semantic representations from labeled and unlabeled domains.
B. Self-Supervision: Magnification prediction classifies images into 4 levels: 40×, 20×, 10× and 5×.Higher magnification provides more detail but less context, while lower magnification provides less detail but more context; missing levels can be created by bilinear resizing.

2) Solving Magnification Puzzle (jigmag):

Jigmag rearranges images from different magnification levels into a 2×2 puzzle and trains the network to infer their order. The task is designed to encourage globally meaningful representations despite histology’s small, unordered objects.

2) Solving Magnification Puzzle (jigmag):: Unlike natural-image jigsaw tasks, histology jigmag is challenging because objects are smaller and lack consistent relative ordering.The proposed arrangement uses multiple magnifications to provide structured contextual information.
2) Solving Magnification Puzzle (jigmag):: Jigmag uses a 2×2 grid whose tiles correspond to images at different magnifications and whose arrangement is represented by an order vector v.For example, v = [0, 1, 2, 3] specifies the magnification assigned to each grid position.
2) Solving Magnification Puzzle (jigmag):: Jigmag is intended to force representations that can improve semi-supervised learning or domain adaptation.Its supervision emphasizes differences between tiles and their positions rather than low-level image statistics.
2) Solving Magnification Puzzle (jigmag):: The task considers 12 different orders of magnification and uses cross entropy for classification tasks.Pixel-wise tasks use L1 loss.

1) Predicting image rotation:

The rotation pretext task predicts which of four rotations was applied to an input image, while related auxiliary tasks include horizontal flipping and image reconstruction.

1) Predicting image rotation:: Rotation prediction applies 0°, 90°, 180° or 270° rotations and assigns labels 0, 1, 2 or 3, respectively.The label directly identifies the applied rotation.
1) Predicting image rotation:: Horizontal flipping uses label 1 when the image is flipped and label 0 otherwise.This provides a binary self-supervised classification task.
1) Predicting image rotation:: Image reconstruction uses a convolutional decoder placed on top of the feature extractor.The setup is similar to predicting the hematoxylin channel.
1) Predicting image rotation:: In the generative auxiliary task, a network generates samples resembling original unlabeled images while the shared encoder supports classification and real-versus-fake prediction.The generator learns Dgen to align with Dreal, and a simpler encoder/discriminator was used for convergence.

5) Domain prediction:

Domain prediction supports domain adaptation by training a discriminator to distinguish source from target features while the feature extractor learns to confuse it. A gradient reversal layer implements this domain confusion.

5) Domain prediction:: DANN trains a domain discriminator to distinguish source and target domains while the feature extractor learns domain-invariant features.The approach uses a minimax game between domain discrimination and feature confusion.
5) Domain prediction:: The domain loss is weighted by coefficient α_d and uses domain labels d_i for each input x_i.These terms appear in the domain-prediction objective.
5) Domain prediction:: A Gradient Reversal Layer reverses gradients from the domain loss during back-propagation with respect to feature-extractor parameters.This trains the extractor to confuse the domain discriminator.
5) Domain prediction:: Camelyon16 contains 399 H&E-stained lymph-node histological images acquired from two medical centers.The centers are Radboud University Medical Center and University Medical Center Utrecht.

2) LNM-OSCC:

LNM-OSCC is an in-house oral squamous cell carcinoma metastasis dataset assembled from WSIs acquired at two hospitals with different scanners. The study uses these data for patch-based experiments with limited annotations.

Dataset: LNM-OSCC contains H&E WSIs of oral squamous cell carcinoma metastasized to cervical lymph nodes, acquired using two scanners at two hospitals.The dataset includes 98 WSIs from UHCW and 119 from Sheffield University.
Annotation setting: Tumor regions in the training and validation sets were not exhaustively annotated.
Patch preparation: The experiments extract 128 × 128 patches at 10× magnification from LNM-OSCC and Camelyon16.
Model: Self-Path uses a ResNet50 feature extractor with classification, image-reconstruction, and hematoxylin-prediction heads.
Training: The shared-encoder experiments train for 200 epochs with batch size 64, Adam, and learning rate 10^-3.Fake/real image-generation experiments use 500 epochs, batch size 32, and learning rate 3 × 10^-4.

C. Semi-Supervised Experiments

Semi-supervised experiments compare Self-Path pretext tasks with established baselines under varying annotation budgets. Domain-specific tasks, especially jigmag, are strongest when labeled data are scarce.

Experimental design: The experiments compare self-supervised tasks with Mean Teacher, VAT, and teacher-student chain under varying annotation budgets.Performance is evaluated using test-set AUCs with a fixed validation set and three random-seed runs.
LNM-OSCC: At 1% and 4% annotation budgets on LNM-OSCC, jigmag achieves the best performance among the evaluated methods.These budgets correspond to 1 and 4 labeled WSIs, containing 134 and 1120 labeled patches.
LNM-OSCC: At 1% and 2% annotation budgets on LNM-OSCC, hematoxylin and magnification tasks outperform domain-agnostic and generative tasks.
LNM-OSCC: At 5% annotation on LNM-OSCC, the generative task reaches an AUC of 95.4%.
Camelyon16: At 1% annotation on Camelyon16, only magnification and jigmag outperform Mean Teacher and the supervised baseline.Jigmag achieves the highest overall performance, with its main boost appearing at very low labeled-data levels.

3) Comparisons of semi-supervised learning on Kather dataset:

Kather experiments evaluate patch-based classification with macro AUC-ROC, while the domain-adaptation experiment transfers from labeled Camelyon16 to unlabeled LNM-OSCC. Self-supervised tasks improve low-annotation performance, and generative modeling performs best for domain adaptation.

Kather dataset: Kather evaluation uses macro AUC-ROC because the dataset contains nine tissue classes.Unlike the other datasets, Kather provides patches without access to WSIs, so annotation budgets are patch-based.
Kather results: At 2% labeled Kather data, the supervised baseline still reaches a Macro AUC of 98%.Therefore, semi-supervised performance is reported at 1% and 0.1% annotation budgets, where supervised performance degrades.
Kather limitation: The Kather experiments could not use jigmag or magnification tasks because WSIs were unavailable for extracting large or multi-magnification patches.
Kather results: At 1% Kather annotation, hematoxylin-channel prediction improves performance by 2.8% over the baseline and 1.2% over Mean Teacher.
Domain adaptation: Domain adaptation uses labeled Camelyon16 as the source and unlabeled LNM-OSCC as the target, testing on the LNM-OSCC test set.
Domain adaptation: Generative modeling achieves the best domain-adaptation performance, 2% above WDGRL and 11% above the Camelyon16-only baseline.Magnification, jigmag, and hematoxylin auxiliary tasks each boost AUC-ROC by 10% relative to the baseline.

1) WSI Analysis:

The WSI-level evaluation aggregates patch predictions into heat maps and then classifies extracted heat-map features. Using unlabeled target-domain images with the generative Self-Path setting improves performance over source-only and WDGRL baselines, with corresponding visual differences in missed and false-positive tumor regions.

Heat-map generation: WSI heat maps are built by aggregating overlapping 128 × 128 patch predictions extracted at 10× magnification.Patches are taken from tissue regions with 50% overlap before aggregation.
Heat-map generation: Heat-map objects are summarized with morphological and geometrical features across three thresholds, then classified using random forest.The post-processing extracts object statistics before final WSI-level evaluation.
Domain adaptation results: 75.2 AUC-ROC is obtained using only Camelyon16 data, while the generative model using unlabeled images gains 15.2%.The comparison evaluates Camelyon16 → LNM-OSCC domain adaptation at the WSI level.
Domain adaptation results: The generative model improves WSI classification by 5% over WDGRL, translating a 2% patch-level AUC-ROC improvement into a WSI-level gain.The reported improvement is also reflected in the visual heat maps.
Scope and future work: The study identifies Self-Path as applicable to computational-pathology problems with limited annotation budgets or abundant unlabeled images.The conclusion also proposes expanding the self-supervision tasks in future work.
Visual comparison: Source-only predictions miss tumor regions and produce false negatives, whereas WDGRL and generative Self-Path use unlabeled data to improve the overlaid heat maps.Figure 5 compares ground truth, source-only predictions, WDGRL predictions, and generative Self-Path predictions.