Source-linked AI summary
Deep neural network models for computational histopathology: A survey
Chetan L. Srinidhi, Ozan Ciga, Anne L. Martel
TL;DR
Computational histopathology needs methods that can handle very large whole-slide images, multiscale information, weak labels, stain variation, and difficult annotations. This survey reviews more than 130 papers across major deep-learning strategies and histology tasks, finding broad methodological activity alongside challenges for reliable clinical translation.
Problem
Whole-slide histopathology analysis involves extremely large images, multiscale information, weak slide-level labels, stain variation, and costly expert annotations.
Method
The paper surveys more than 130 computational histopathology studies covering supervised, weakly supervised, unsupervised, transfer-learning, survival-model, dataset, and clinical-translation approaches.
Results
The surveyed literature applies deep learning to diverse tasks including segmentation, classification, tumour detection, disease prediction, and prognosis across multiple cancer types.
Takeaways & Limitations
Transfer-learning approaches are increasingly used because of their reported generalization adaptability across datasets and scanning protocols.
Takeaways & Limitations
Clinical translation remains constrained by regulatory and privacy concerns over patient data and by difficulty objectively comparing models developed for narrow populations.
Abstract
from arXiv · showhide
Histopathological images contain rich phenotypic information that can be used to monitor underlying mechanisms contributing to diseases progression and patient survival outcomes. Recently, deep learning has become the mainstream methodological choice for analyzing and interpreting cancer histology images. In this paper, we present a comprehensive review of state-of-the-art deep learning approaches that have been used in the context of histopathological image analysis. From the survey of over 130 papers, we review the fields progress based on the methodological aspect of different machine learning strategies such as supervised, weakly supervised, unsupervised, transfer learning and various other sub-variants of these methods. We also provide an overview of deep learning based survival models that are applicable for disease-specific prognosis tasks. Finally, we summarize several existing open datasets and highlight critical challenges and limitations with current deep learning approaches, along with possible avenues for future research.
1. Introduction
Histopathology supports diagnosis, grading, prognosis, and biological discovery, but digital whole-slide analysis introduces scale, multiscale, labeling, variability, and annotation challenges that motivate specialized deep-learning methods.
- H&E tissue interpretation provides diagnostic information and phenotypic features associated with tumour aggressiveness and recurrence risk.Examples include nuclear atypia, gland formation, mitosis, and inflammation.
- Tumour grading systems such as the Gleason and Nottingham scores guide treatment decisions but exhibit inter-pathologist variability.Quantitative systems such as residual cancer burden can also be too time-consuming for routine clinical use.
- Digital pathology features can reveal biological aspects of cancer tissue and provide information complementary to genomic data.These developments increased interest in image-derived biomarkers for precision oncology.
- Whole-slide images must be divided into hundreds or thousands of tiles, while both low-magnification context and high-magnification detail may matter.Survival prediction additionally requires locating salient regions without knowing them a priori.
- Weak slide-level labels, high within-subtype variability, and the need for extensive cell detection and annotation require adapted or novel deep-learning approaches.The survey covers more than 130 papers spanning detection, diagnosis, prediction, and prognosis tasks.
2. Overview of learning schemas
The survey organizes computational histopathology around multiple learning schemas and deep-network families, ranging from labeled patch prediction to weakly labeled, unlabeled, and cross-domain learning.
- The survey uses figures to summarize publication counts, learning-schema categories, architectures, and histopathological image-analysis tasks.
- Deep-learning models in computational histopathology include CNNs, RNNs, GANs, auto-encoders, and related variants.
- Supervised learning trains a model from image-label pairs to predict labels for previously unseen test images.In histopathology, patches may be annotated as cancerous or normal by a pathologist.
- Weakly supervised learning uses coarse image-level annotations to infer fine-grained pixel- or patch-level labels.Such weak labels are easier to obtain than expensive pixel-wise annotations.
- Unsupervised learning identifies image patterns without mapping inputs to predefined labels, often using lower-dimensional representations and clustering.
- Transfer learning transfers knowledge from a source domain to a target domain while relaxing the assumption that training and test data are identically distributed.
3. Methodological approaches
The methodological review is structured by learning scheme, covering supervised, weakly supervised, unsupervised, and transfer-learning approaches as general references for diverse histopathological tasks.
- The survey organizes its methodological discussion into supervised, weakly supervised, unsupervised, and transfer-learning approaches.Models originally developed for specific applications may apply across a wide variety of histopathological tasks.
3.1. Supervised learning
Supervised deep learning in computational histopathology spans classification, regression, and segmentation, with models operating locally on cells or globally on whole-slide images. CNNs remain central, while attention, multiscale, and transfer-learning approaches address contextual, computational, and generalization challenges.
- Supervised learning: Supervised models are organized into classification, regression, and segmentation categories, covering local object analysis and global whole-slide prediction.Local tasks identify or localize cells, nuclei, and other regions; global tasks perform image-level disease grading or prediction.
- Local-level tasks: CNNs applied through sliding windows are widely used for cell and nuclei detection, but gigapixel whole-slide images make direct processing impractical.Patch-based analysis reduces input size, yet requires many patches and can involve redundant computation.
- Model robustness: Multi-scale and transfer-learning approaches are increasingly used because they provide adaptability across datasets and scanning protocols, while stain variation still limits traditional model generalization.UNet models were also reported to achieve better generalization ability and robustness than FCN models in some comparisons.
- Global-level tasks: 5% improvement in average sensitivity and specificity was reported by a three-layer CNN over previous handcrafted methods for invasive ductal carcinoma identification.The model performed breast cancer image analysis using patch-based classification.
- Global-level tasks: Patch-based methods lose surrounding visual context, perform redundant computations, and scale linearly with the number of input patches.Multi-magnification networks and attention mechanisms were proposed to better capture context and more closely reflect pathologists’ slide examination.
- Global-level tasks: Attention-based models select diagnostically relevant regions, reduce parameter counts and inference time, and keep model complexity independent of whole-slide image size.These models emulate sequential visual attention used by pathologists and avoid repeatedly processing all locations.
3.2. Weakly supervised learning
Weakly supervised learning uses coarse image-level annotations to infer fine-grained information, reducing annotation demands in histopathology. The survey emphasizes multiple-instance learning, including global and local detection strategies that can support both slide-level prediction and localization.
- Weakly supervised learning: Weakly supervised learning infers pixel- or patch-level information from readily available image-level labels, reducing pathologist annotation burden.In histopathology, a slide may be labeled cancerous when only a small region contains cancer, without specifying its location.
- Multiple-instance learning: The survey focuses on multiple-instance learning, representing each labeled histology image as a bag of instances such as pixels or patches.Bags receive positive or negative labels, while instance labels are predicted or remain unknown.
- Multiple-instance learning: Standard voting or pooling may produce inaccurate image-level predictions because instance-level labels can be misclassified.Global-detection MIL methods therefore use alternative aggregation strategies suited to specific histology tasks.
- Global and local detection: Local-detection MIL uses fully convolutional networks to compute features jointly across instances and is generally applied to delineating cancerous regions.Bag labels may be propagated to instances, and predicted instance labels can subsequently train an FCN end to end.
- Global and local detection: Global and local detection methods can simultaneously classify whole slides and localize diagnostically significant cancerous tissue or cells.These approaches capture instance-wise dependencies and their effect on the final image-level decision score.
- Limitations and interpretability: MIL methods disagree about the accuracy of instance-level predictions trained using only bag-level labels, motivating attention-based approaches.The survey describes attention and deep supervision as ways to localize discriminative evidence while retaining competitive slide-level accuracy.
3.3. Unsupervised learning
Unsupervised learning seeks useful structure in histopathology images without expert labels, using representations and clustering to organize the data. The survey also covers unsupervised transfer and self-supervised approaches, while noting that histopathology-specific structure and staining make these methods challenging.
- Unsupervised learning: Fully unsupervised learning identifies useful structure in images without labels, but such methods remain rare in digital pathology.The survey distinguishes fully unsupervised methods from approaches that use domains or other identifiers as implicit labels.
- Unsupervised learning: Unsupervised methods commonly reduce images to lower-dimensional representations and group them into mutually exclusive or hierarchical clusters.Autoencoders are typically used for dimensionality reduction before feature aggregation and clustering.
- Unsupervised transfer learning: Unsupervised transfer learning uses learned mapping functions to initialize target tasks, potentially with very few labeled training images.The survey describes training on unlabeled images from a specific modality using a reconstruction-like objective.
- Self-supervised learning: Self-supervised methods can handle larger images and offer an alternative to clustering approaches that usually require context and a larger field of view.Examples include predicting spatial ordering or rotations and mapping grayscale images to RGB counterparts for network initialization.
- Challenges and opportunities: Histopathology-specific cell structure, color, and staining rules make unsupervised learning more difficult than in natural-scene settings.The survey presents these domain differences as an opportunity for developing techniques applicable to medical images.
- Motivation: Unsupervised methods are desirable because they can train models with little or no labeled data and may improve interpretability through data grouping.The survey notes that fully unsupervised methods for arbitrary tasks are still uncommon.
3.4. Transfer learning
Transfer learning adapts knowledge across domains and is widely used in digital pathology, while domain adaptation and stain-focused preprocessing address staining and other distribution shifts. Comparative evidence indicates that preprocessing choices can substantially affect performance, and stain-transfer benefits remain uncertain.
- 3.4. Transfer learning: Transfer learning applies knowledge from a source domain to a target domain, commonly using ImageNet-pretrained CNNs such as VGGNet, ResNet, InceptionNet, MobileNet, and DenseNet.These pretrained models have been widely applied to digital pathology tasks.
- 3.4.1. Domain adaptation: Domain adaptation seeks similar target-task performance with little or no labeled target data by learning features that are discriminative yet insensitive to domain shift.Domain-adversarial networks implement this objective, and a multi-level variant achieved state-of-the-art performance on the BACH challenge task.
- 3.4.2. Stain normalization: Stain normalization, augmentation, and stain transfer modify staining properties or input images to reduce the impact of staining variability on generalization.Color augmentation can vary pixel values in HSV or HED spaces, whereas stain normalization directly modifies input images.
- 3.4.2. Stain normalization: 21%: the CAMELYON16 winning entry exceeded Shaban et al. with respect to AUC for binary WSI classification, despite using traditional normalization rather than stain transfer.This comparison contributes to uncertainty about the use cases of aesthetically pleasing stain-transfer methods.
- 3.4.2. Stain normalization: Augmentation and normalization strategies drastically alter automated medical-imaging performance, while scale, resolution, image quality, and scanner imperfections also contribute to domain shift.A broader comparison found color augmentation in HSV or HED consistently advisable, with slight additional gains possible from network-based augmentation.
4. Survival models for disease prognosis
Survival models use whole-slide images to estimate event probabilities or time to events, but weak patient-level labels, heterogeneous tumors, and limited datasets complicate training. End-to-end models can associate image regions with outcomes and reveal morphology linked to survival.
- 4. Survival models for disease prognosis: Survival models estimate the probability of an event within a predefined period or predict time to an event from a whole-slide image.Endpoints vary by disease, including survival time for glioblastoma and postsurgical recurrence for breast cancer.
- 4. Survival models for disease prognosis: Weak patient-level labels provide one outcome per patient, while whole-slide images require hundreds or thousands of patches and only a subset may be salient.Tumor heterogeneity makes identifying outcome-relevant patches computationally challenging.
- 4. Survival models for disease prognosis: Survival datasets usually contain fewer than 1000 slides, limiting the data available for training compared with diagnostic settings where more than 10,000 slides enabled accurate MIL results.The passage presents dataset size as a constraint specific to survival analysis.
- 4. Survival models for disease prognosis: Patch predictions can be aggregated by means, top-ranked patches, or low-dimensional feature representations to produce a patient-level survival prediction.Averaging all patches may fail when salient patches represent only a small fraction of the slide.
- 4. Survival models for disease prognosis: End-to-end models associate probabilities with individual patches and can reveal morphology related to outcome, including stroma, inflammation, cellular diversity, vacuolization, microvascular proliferation, and cellularity.Prediction heatmaps may also expose tumor heterogeneity and guide tissue extraction for genomics and proteomics assays.
5. Discussion and future trends
The survey emphasizes that performance depends less on selecting a particular deep-learning architecture than on task-specific preprocessing, training practices, diverse data, and clinically relevant validation. It also identifies annotation quality, interpretability, privacy, generalization, and workflow integration as barriers to clinical translation.
- 5.1. Effect of deep learning architectures on task performance: Many tasks did not show a particular architecture determining the outcome, while appropriate training practices enabled simpler networks to prevail.Challenge winners often used architectures introduced before 2015, despite challenge dates ranging from 2016 to 2019.
- 5.2. Input data processing: Preprocessing must balance spatial context, resolution, color representation, memory limits, and information loss when tiling or resizing whole-slide images.The optimal field-of-view and resolution trade-off depends on the application.
- 5.3. Quality of training and validation data: High-quality annotations remain costly and variable across pathologists, making label-generation procedures, consensus, and label accuracy important for supervised learning.The survey reports that most methods are fully supervised and that annotation requires substantial manual expertise.
- 5.3. Quality of training and validation data: Training-data diversity can improve robustness and generalization, with one model trained on more than 15,000 patients from 45 countries achieving AUC greater than 0.98 for three histology tasks.The survey also calls for larger, clinically relevant datasets that better capture real clinical complexity.
- 5.4. Interpretability: Survival models may be more acceptable to clinicians when they use a small number of image features familiar to pathologists rather than operating as end-to-end black boxes.Interpretability is considered less important for tasks that pathologists can readily validate by visual inspection.
- 5.5. Clinical translation: Clinical translation is constrained by patient-data ownership and privacy, inconsistent comparisons, out-of-domain performance, algorithmic bias, adversarial vulnerability, interpretability, and limited digital workflow investment.The survey recommends independent testing on comparable target populations and human-centred, interpretable systems with confidence measures.
6. Conclusions
The survey organizes deep-learning approaches for computational histopathology across learning paradigms, tasks, and cancer types, while critically discussing architecture performance, training data, and interpretability. It serves as a reference guide to current techniques and their challenges.
- 6. Conclusions: The survey covers more than 130 papers spanning supervised, weakly supervised, unsupervised, and transfer-learning approaches across diverse histology tasks and cancer types.Covered tasks include segmentation, tissue classification, tumour detection, disease prediction, and prognosis.
- 6. Conclusions: Its methodological categorization provides a reference guide to techniques in computational histopathology and examines the importance of training data and model interpretability.
Conflict of interest
The authors report a conflict-of-interest disclosure for the paper.
- Conflict of interest: One author is co-founder and chief scientific officer of Pathcore, while the other listed authors have no conflicts.