A Generalized Deep Learning Framework for Whole-Slide Image Segmentation and Analysis

Mahendra Khened; Avinash Kori; Haran Rajkumar; Balaji Srinivasan; Ganapathy Krishnamurthi

Source-linked AI summary

A Generalized Deep Learning Framework for Whole-Slide Image Segmentation and Analysis

Mahendra Khened, Avinash Kori, Haran Rajkumar, Balaji Srinivasan, Ganapathy Krishnamurthi

arXiv:2001.00258v2Jan 1, 2020eess.IVcs.CVcs.LGq-bio.TO

TL;DR

Whole-slide histopathology analysis is difficult to automate because images are enormous and datasets and tasks vary substantially. The paper presents a deep-learning framework combining segmentation, downstream analyses, and uncertainty estimation, achieving strong results across CAMELYON, DigestPath, and PAIP challenges. The framework is validated across multiple cancer sites and released with documentation and software for reuse.

Problem
Automated histopathology analysis is needed because manual pathological analysis is arduous, while whole-slide images, limited annotations, and stain variability complicate integrated analysis.
Method
The framework combines ensemble FCN-based whole-slide segmentation with efficient tissue-guided patch inference, downstream classification and tumour-burden analysis, uncertainty estimation, and class-imbalance handling.
Results
The framework achieved strong results across three public challenges, including a Cohen’s kappa score of 0.9090 on CAMELYON17 and a Dice score of 0.78 on DigestPath.
Takeaways & Limitations
The framework supports segmentation and downstream histopathology analysis across breast lymph-node, liver, and colon cancer datasets and is available as an open-source application.
Takeaways & Limitations
The authors identify efficient multi-resolution FCN design and improved post-processing as directions for further refinement.

Abstract

from arXiv · show

Histopathology tissue analysis is considered the gold standard in cancer diagnosis and prognosis. Given the large size of these images and the increase in the number of potential cancer cases, an automated solution as an aid to histopathologists is highly desirable. In the recent past, deep learning-based techniques have provided state of the art results in a wide variety of image analysis tasks, including analysis of digitized slides. However, the size of images and variability in histopathology tasks makes it a challenge to develop an integrated framework for histopathology image analysis. We propose a deep learning-based framework for histopathology tissue analysis. We demonstrate the generalizability of our framework, including training and inference, on several open-source datasets, which include CAMELYON (breast cancer metastases), DigestPath (colon cancer), and PAIP (liver cancer) datasets. We discuss multiple types of uncertainties pertaining to data and model, namely aleatoric and epistemic, respectively. Simultaneously, we demonstrate our model generalization across different data distribution by evaluating some samples on TCGA data. On CAMELYON16 test data (n=139) for the task of lesion detection, the FROC score achieved was 0.86 and in the CAMELYON17 test-data (n=500) for the task of pN-staging the Cohen's kappa score achieved was 0.9090 (third in the open leaderboard). On DigestPath test data (n=212) for the task of tumor segmentation, a Dice score of 0.782 was achieved (fourth in the challenge). On PAIP test data (n=40) for the task of viable tumor segmentation, a Jaccard Index of 0.75 (third in the challenge) was achieved, and for viable tumor burden, a score of 0.633 was achieved (second in the challenge). Our entire framework and related documentation are freely available at GitHub and PyPi.

1. Introduction

Histopathology is central to cancer diagnosis, but manual analysis is difficult, time-consuming, and subject to disagreement. Digital whole-slide imaging and automated analysis offer a promising response, while large images, limited annotations, stain variability, and feature extraction remain key challenges.

Histopathology supports cancer diagnosis and identification of prognostic and therapeutic targets.
Pathological analysis is arduous, time-consuming, and requires in-depth knowledge, with pathologists disagreeing on breast-biopsy diagnoses 24.7% of the time on average.
Whole-slide images can reach gigapixel dimensions, creating substantial computational challenges for automated analysis.
Deep-learning development is constrained by insufficient curated, representative, expert-labeled training data and restricted data access.
Generalized whole-slide analysis must address stain variability and extraction of clinically relevant information across datasets.

2. Related work

Prior histopathology methods span engineered-feature pipelines and patch-based deep learning, while the proposed framework integrates segmentation with downstream analysis across multiple cancer sites. Its contributions include ensemble models, efficient inference, uncertainty estimation, and open-source implementation.

Whole-slide imaging produces gigapixel, multi-resolution images suitable for machine-learning-based computer-aided diagnosis.
Engineered-feature methods are limited by their dependence on predetermined features that may not fully distinguish tissue classes.
Patch-based deep-learning approaches crop slides into smaller regions and aggregate patch predictions into slide-level outputs or heatmaps.
The proposed framework uses a core segmentation network for downstream metastasis classification and viable tumour burden estimation across breast, liver, and colon cancer datasets.
Its ensemble segmentation model trains multiple FCN architectures on different training subsets and generates tumour probability maps during inference.
The inference pipeline samples tissue-derived patch coordinates, averages overlapping predictions, uses large inference patches, and parallelizes ensemble computation.
Uncertainty estimation addresses both data-specific and model-parameter-specific uncertainty through an efficient patch-based framework.

3. Materials and methods

The framework was evaluated across CAMELYON, PAIP, and DigestPath datasets covering breast, liver, and colon cancer analysis tasks. These datasets contain whole-slide or tissue images with expert annotations and task-specific labels or metrics.

Datasets: The study uses CAMELYON, PAIP, and DigestPath datasets for whole-slide image segmentation and downstream analysis across multiple cancer sites.The framework overview includes breast cancer metastases, liver tumour analysis, and colon tumour lesion segmentation.
CAMELYON16: CAMELYON16 contains 399 whole slide images, with pathologist pixel-level metastasis annotations and separate training and testing splits.The dataset includes 160 negative and 110 metastasis slides for training, plus 80 negative and 49 metastasis slides for testing.
CAMELYON17: CAMELYON17 contains 1000 whole slide images from five medical centres, with slide-level metastasis-type labels and pixel-level annotations for 50 slides.The challenge allocated 500 slides for training and 500 for testing.
PAIP: PAIP 2019 contains 100 liver-tissue whole slide images averaging 50,000x50,000 pixels, with pixel-level viable-tumour and whole-tumour annotations.The dataset also provides a viable tumour burden metric; 50 images were used for training, 10 for validation, and 40 for testing.
PAIP: Tumour burden is defined as the ratio of the viable tumour region to the whole tumour region.The viable region is cancerous tissue, while the whole-tumour boundary includes dispersed viable tumour nests, necrosis, and capsule.
DigestPath: DigestPath provides pathologist-annotated colon tissue images, with 660 training images and 212 confidential testing images from 152 patients.The tissue images average 5000x5000 pixels and were H&E stained and scanned at 20x magnification.

3.3. Data pre-processing

Pre-processing first isolates tissue from the glass background, then samples balanced tumour and non-tumour patches. Augmentation and coordinate perturbation increase training diversity while reducing unnecessary computation.

Tissue mask generation: Tissue masks are generated at low resolution after transforming RGB images to HSV and applying Otsu thresholding.Processing at low resolution reduces computational cost because only an approximate tissue boundary is required.
Tissue mask generation: Black background regions are replaced with white pixels and median blurring is applied before thresholding in some CAMELYON17 cases.A 7x7 median blur smooths tissue regions and removes border noise while preserving tissue edges.
Patch extraction: Training patches are randomly extracted from the tissue mask with equal numbers of tumorous and non-tumorous patches.A patch is tumorous when at least one pixel is classified as tumour, and patch dimensions are treated as a hyperparameter.
Data augmentation: Augmentation uses flips, 90-degree rotations, Gaussian blurring, colour changes, and random coordinate perturbation.Brightness, contrast, hue, and saturation changes are bounded, while coordinate offsets diversify patches sampled during each training epoch.

3.4. Network architecture

The segmentation system uses an ensemble of encoder-decoder FCN architectures and a hybrid Dice–cross-entropy objective. The design addresses pixel-level class imbalance while combining complementary model architectures.

FCN design: Tumour segmentation uses fully convolutional networks composed of encoder, decoder, and pixel-wise classification layers.The encoder produces low-resolution feature maps, while the decoder reconstructs segmentation outputs.
Ensemble model: The ensemble averages tumour posterior probability maps from three independently trained encoder-decoder FCN models during inference.Its members include U-Net variants and DeepLabV3Plus with different pretrained backbones.
Ensemble model: The ensemble combines U-Net with DenseNet-121, U-Net with Inception-ResNet-V2, and DeepLabV3Plus with Xception.The backbones use ImageNet or PASCAL VOC pretraining, depending on the architecture.
Loss function: Tumour pixels form a minuscule image proportion, so training minimizes a hybrid loss combining cross-entropy and Dice-based loss.The Dice component is designed for overlap-based segmentation assessment and helps address class imbalance.
Loss function: The hybrid loss is optimized with ADAM using empirically assigned weights α = 0.5, β = 0.25, and γ = 0.25.The weights define the relative contributions of the individual loss components.

3.6. Uncertainty analysis

The framework models uncertainty from both data transformations and model variation. Test-time augmentations estimate aleatoric uncertainty, while diversity across ensemble architectures estimates epistemic uncertainty.

Uncertainty types: The paper distinguishes aleatoric uncertainty from epistemic uncertainty in deep-learning diagnostic predictions.Aleatoric uncertainty arises from the data-generation process, whereas epistemic uncertainty reflects uncertain model parameters or architectures.
Clinical use: Uncertainty estimation is intended to help pathologists focus on unclear or uncertain regions during analysis.The paper frames this as support for assessing ambiguous diagnostic cases predicted by deep-learning models.
Aleatoric uncertainty: Aleatoric uncertainty is estimated for each model using test-time augmentations.The allowed transformations include rotation, vertical flip, and horizontal flip.
Epistemic uncertainty: Epistemic uncertainty is estimated from diversity among the model architectures in the ensemble.The associated probabilistic formulation represents outputs for inputs under different trained models.

3.8. Inference pipeline

The inference pipeline reduces whole-slide computation by sampling high-resolution patches within a low-resolution tissue mask, then generates tumour heatmaps and downstream pN-stage predictions.

Pre-processing: A low-resolution uniform sampling grid is rescaled to whole-slide coordinates for extracting fixed-size high-resolution patches.The grid restricts patch extraction to tissue-mask regions.
Heatmap generation: The pN-staging pipeline generates down-scaled tumour probability heatmaps from extracted patches before feature extraction.The complete pipeline is organized into preprocessing, heatmap generation, feature extraction, data balancing, and classification blocks.
Feature extraction: Heatmaps are thresholded at 0.5 and 0.9, after which connected components yield 32 geometric and morphological metastasis features.Region properties are measured using scikit-image.
Data balancing: SMOTETomek balances the training data by combining synthetic minority oversampling with Tomek-link noise removal.The two operations are performed consecutively.
Classification: An ensemble of Random Forest classifiers assigns metastasis types from extracted features, supporting patient-level pN-stage prediction across lymph-node slides.The pN-stage uses all available lymph-node whole-slide images and their individual metastasis types.

3.10. Tumour burden estimation for PAIP dataset

The tumour burden algorithm segments viable tumour, approximates the whole tumour region geometrically, and computes burden as their area ratio.

Method: The viable tumour region is segmented with the proposed deep learning network, while the whole tumour region is approximated heuristically.Training the same network for whole-tumour segmentation produced sub-optimal results.
Method: The algorithm removes false positives and fills holes using morphological operations after viable-tumour segmentation.
Method: A smallest convex hull is constructed around the entire viable tumour region.
Method: The whole tumour region is approximated as the intersection of the convex hull and the tissue mask.
Output: Tumour burden is calculated as the ratio between viable-tumour area and whole-tumour area.

4. Experimental analysis

The experiments evaluate the framework’s training and inference choices across CAMELYON datasets, including tissue detection, FCN ensembles, patch configurations, and Random Forest pN-staging.

CAMELYON16 lesion detection: CAMELYON16 lesion detection is evaluated on 139 test slides using FROC, which plots sensitivity against average false positives per image.The final score averages sensitivity at six predefined false-positive levels.
Training data: 628 whole-slide images were used for training, with tumour and non-tumour patch coordinates sampled across three cross-validation folds.The training data combined CAMELYON16 and CAMELYON17 slides.
Ensemble configurations: Ensemble-A combines three FCN architectures using non-overlapping 256x256 patches, whereas Ensemble-B uses replicated FCNs with 1024x1024 patches and 50% overlap.
CAMELYON16 lesion detection: DenseNet-121 showed higher sensitivity and fewer false positives than other FCNs, while Ensemble-A significantly improved FROC over its constituent models.The authors attribute the boost to averaging heatmaps, which lowers uncertain-region probabilities and eliminates false positives.
CAMELYON17 inference: Ensemble-A was selected for CAMELYON17 inference because ensemble configurations had only marginal FROC differences and the smaller non-overlapping patches reduced computation time.
pN-staging classification: Five-fold cross-validation of the Random Forest classifier achieved an average accuracy of 90%, with performance similar to the train-set model.The models were trained with partial and complete training data under imbalanced class conditions.

5. Challenge results

The framework achieved competitive challenge results for CAMELYON17 pN-staging and DigestPath tumour segmentation, while its convex-hull method performed comparably for PAIP tumour-burden analysis.

CAMELYON17: 0.9090 Cohen’s kappa was achieved by the CAMELYON17 RF-Ensemble on 500 testing slides.The proposed approach ranked third on the open leaderboard.
DigestPath-2019: 0.78 Dice score was obtained on the DigestPath-2019 test set of 212 slides.The result was compared with the top four challenge entries.
PAIP-2019: PAIP Task 1 evaluates viable-tumour segmentation with average Jaccard index, while Task 2 evaluates viable-tumour burden estimation using accuracy weighted by Task 1 performance.
PAIP-2019: The convex-hull approximation method showed comparable performance with deep learning methods for PAIP viable-tumour burden estimation.

6. Open source contribution

The authors released an open-source application built around their segmentation pipeline, supporting whole-slide analysis, uncertainty estimation, and extensibility. It also provides an API and GUI-based slide viewer for integrating and testing segmentation workflows.

The application loads whole slide images, runs segmentation, and calculates uncertainty maps.
Its modular design lets researchers add segmentation pipelines or extend the application’s functionality.
An API supports using the segmentation pipeline within other applications, while the GUI enables researchers to test pipelines interactively.

7. Discussion and conclusions

The framework combines patch-based whole-slide processing, FCN ensembles, and downstream analysis methods across multiple cancer-related tasks. The authors identify future improvements in multi-resolution architectures, post-processing, and learning-based whole-tumour estimation.

Framework and segmentation: The framework divides gigapixel slides into feasible patches, segments them with independently trained FCN ensembles, and stitches the outputs into whole-slide tumour maps.The FCNs use encoder-decoder architectures for dense pixel-level classification.
Future directions: Future work includes efficient multi-resolution FCN architectures and post-processing methods such as patch-based conditional random fields instead of hardcoded thresholds.
Downstream analysis: Segmentation supports downstream lymph-node metastases classification and pN-staging using Random Forest ensembles trained on tumour-probability-map features.
Downstream analysis: Synthetic training samples demonstrated efficacy for addressing class imbalance in metastases classification tasks.
Downstream analysis: For liver cancer, viable tumour burden estimation approximated the whole-tumour region with a convex hull around the predicted viable tumour region.The approximation performed on par with other deep learning-based segmentation approaches and was computationally inexpensive; learning-based refinement remains possible.