PaperlayerView exact revision on arXiv

Source-linked AI summary

A Comprehensive Review of Computer-aided Whole-slide Image Analysis: from Datasets to Feature Extraction, Segmentation, Classification, and Detection Approaches

Chen Li, Xintong Li, Md Rahaman, Xiaoyan Li, Hongzan Sun, Hong Zhang, Yong Zhang, Xiaoqi Li, Jian Wu, Yudong Yao, Marcin Grzegorzek

arXiv:2102.10553v1cs.CVcs.AI

TL;DR

WSI analysis seeks more quantitative and reproducible computer-aided support for pathology than traditional manual examination. This review synthesizes datasets, metrics, feature extraction, segmentation, classification, and detection methods, finding broad progress alongside dataset and computational limitations.

  • Problem

    Traditional pathological image analysis lacks quantitative indicators and reproducibility, while workload and time pressure can contribute to incomplete diagnosis and misdiagnosis.

  • Method

    The paper reviews machine-learning-based WSI analysis, covering datasets, evaluation methods, feature extraction, segmentation, classification, and detection.

  • Results

    The review identifies common datasets and features, U-Net as a recent segmentation mainstream, classification as the most studied task, and CNN methods as performing excellently across tasks.

  • Takeaways & Limitations

    Machine learning combined with WSI is presented as promising for assisting pathologists, with research opportunities extending across additional anatomical fields.

  • Takeaways & Limitations

    Large-scale, comprehensive, fully annotated WSI datasets remain lacking, and more computationally efficient, lower-hardware, interpretable networks are needed.

Abstract

from arXiv · show

With the development of computer-aided diagnosis (CAD) and image scanning technology, Whole-slide Image (WSI) scanners are widely used in the field of pathological diagnosis. Therefore, WSI analysis has become the key to modern digital pathology. Since 2004, WSI has been used more and more in CAD. Since machine vision methods are usually based on semi-automatic or fully automatic computers, they are highly efficient and labor-saving. The combination of WSI and CAD technologies for segmentation, classification, and detection helps histopathologists obtain more stable and quantitative analysis results, save labor costs and improve diagnosis objectivity. This paper reviews the methods of WSI analysis based on machine learning. Firstly, the development status of WSI and CAD methods are introduced. Secondly, we discuss publicly available WSI datasets and evaluation metrics for segmentation, classification, and detection tasks. Then, the latest development of machine learning in WSI segmentation, classification, and detection are reviewed continuously. Finally, the existing methods are studied, the applicabilities of the analysis methods are analyzed, and the application prospects of the analysis methods in this field are forecasted.

1 Introduction

Whole-slide imaging converts glass slides into large digital images that can be viewed and analyzed computationally. Combined with CAD, WSI addresses limitations of manual pathology and supports increasingly broad, structured analysis workflows.

  • Whole-slide imaging: WSI systems digitize glass slides through scanning hardware and provide software for viewing, managing, and analyzing the resulting digital slides.A typical system includes a scanner, networked computers, storage, display, and compatible image software.
  • Whole-slide imaging: WSI acquisition uses area or line scanning, with line scanners providing smooth continuous movement and fast scanning.Z-stacking can scan multiple focal planes and combine them into composite multiplanar images for areas of interest.
  • Applications and benefits: Digital pathology reduces slide transport, breakage, fading, and deterioration while improving image quality, resolution, annotation, and storage efficiency.These properties support broader pathology use, including remote consultation, education, and quantitative biomarker assessment.
  • Development of WSI analysis: Manual pathological analysis is time-consuming and vulnerable to non-reproducibility, workload-related cognitive disturbance, incomplete diagnosis, and misdiagnosis.CAD is presented as a more efficient, accurate, intuitive, and quantitative alternative to manual slide reading.
  • Development of WSI analysis: Computer-aided WSI analysis has grown over time, especially in classification, segmentation, and detection, with increasing use reported since 2014.Other applications, including retrieval and localization, are also described as growing.
  • Analysis workflow: The review organizes histopathology WSI analysis into data acquisition, image presentation, preprocessing, feature extraction, post-processing, classifier design, and system evaluation.Preprocessing can include augmentation and segmentation; evaluation includes accuracy, error rate, sensitivity, and specificity.
  • Motivation of this review: This review addresses a gap in prior surveys by analyzing CAD combined with WSI and summarizing more than 210 related works from 2004 to 2020.The intended audience includes medical-imaging researchers and medical professionals.

2 Datasets and Evaluation Methods

The review surveys public WSI datasets and evaluation measures used for classification, segmentation, and detection in digital pathology.

  • Public datasets: TCGA and Camelyon are prominent public datasets used for WSI classification and detection.TCGA combines WSI data with extensive cancer clinical and molecular information, while Camelyon contains H&E-stained breast-cancer lymph-node WSIs.
  • Public datasets: TUPAC16 supports mitosis detection, tumor-proliferation prediction, automatic scoring, and region-of-interest analysis.Its main challenge dataset contains 821 TCGA WSIs, with separate auxiliary datasets for areas of interest and mitotic detection.
  • Public datasets: Kimia Path24 contains 24 manually selected WSIs emphasizing diverse texture patterns rather than anatomy or malignancy.It is mainly used for classification and retrieval of histopathological images.
  • Evaluation methods: Dice coefficient and Jaccard index are popular measures for evaluating segmentation overlap.Dice compares the intersection with the combined areas, while Jaccard represents the intersection over the union.
  • Evaluation methods: Classification and detection commonly use accuracy, precision, recall, sensitivity, specificity, and F-measure, with AUC also used for classification.Detection additionally uses FROC for positioning and quantitative analysis of multiple lesions.

3 Feature Extraction

Feature extraction converts WSI data into informative representations for CAD, using preprocessing followed by color, texture, shape, or learned feature construction.

  • Feature-extraction pipeline: Traditional WSI feature extraction uses preprocessing, feature extraction, feature processing, and subsequent machine-learning segmentation or classification.Preprocessing standardizes or normalizes images to reduce interference and emphasize characteristic information.
  • Feature-extraction pipeline: Feature extraction aims to retain relevant information while representing the original data in a lower-dimensional space.The review groups commonly extracted WSI features into color, texture, and shape categories.
  • Color features: Color features are commonly computed in RGB, HSV, or LAB spaces using histograms, moments, or coherence descriptors.The reviewed studies include RGB-channel statistics, HSV-channel statistics, and LAB histograms.
  • Color features: WSIs are often divided into patches before color features such as LAB-channel histograms and LBP histograms are computed.The illustrated workflow includes stain-related channels, LAB channels, and channel-wise color histograms.

Others Color Features

The review describes additional color and texture descriptors, emphasizing their varied use across WSI analysis studies and their differing representational limitations.

  • Other color features: Six reviewed studies use color features outside the main RGB, HSV, and LAB groups, including HSI, HSD, and rough color descriptors.Rough features characterize broad shape, color, and texture variation, while fine features describe more comprehensive image properties.
  • Other color features: RGB features were used most frequently, followed by HSV and LAB features across the reviewed literature.The reported study periods were 2009–2018 for RGB, 2012–2014 for HSV, and 2009–2017 for LAB.
  • Texture limitations: Usual texture features may not accurately reflect differences between textures that differ in human visual perception.This limitation is stated for texture comparisons involving visual differences.
  • Texture features: Texture descriptors include statistical, geometric, model-based, and signal-processing methods, with 57 reviewed papers using texture features.Examples include GLCM, Voronoi-based descriptors, random fields, and wavelet transforms.
  • Texture features: LBP, Haralick, and GLCM descriptors are repeatedly used to represent local patterns, intensity texture, and co-occurrence structure.The review identifies 14 LBP papers, five Haralick papers, and eight GLCM papers.
  • Texture features: Filter- and SIFT-based methods provide alternative texture representations, including Gabor, Gaussian-like, MSER, and SIFT features.These methods are reported across five related papers.

Others Texture Features

Shape descriptors complement color and texture features, while CNNs and other engineered representations extend feature extraction toward richer WSI analysis.

  • Shape features: Shape features represent image contours or regions using boundaries, geometric parameters, Hough transforms, moments, area, circumference, and Fourier descriptors.The review identifies 15 papers using shape features.
  • Shape features: Basic geometric shape features are commonly extracted before classification, segmentation, or detection to represent images more effectively.One example uses the major-axis to minor-axis ratio of a best-fit ellipse to eliminate false regions.
  • Integrated analysis: Image analysis can characterize tissue through image normalization, segmentation, and feature extraction.The reviewed workflow presents these operations as connected stages for objective tissue characterization.
  • Shape features: Other shape studies extract sharpness, morphometric, topological, and core-group features, including 461 quantitative image features in one study.These feature sets capture combinations of texture, color, shape, and topological properties.
  • Deep features: CNNs are widely used for learned feature extraction, with convolutional, pooling, and fully connected layers producing feature maps.The review reports 53 papers using CNN-based deep feature extraction and discusses VGGNet as a deeper architecture.
  • Deep features: VGG-16 produces a high-dimensional feature vector for each individual WSI tile.This tile-level representation is illustrated as the network output for an input image.

Deep Learning Features of the ResNet Series

The review describes traditional and deep-learning feature extraction approaches used in WSI analysis, highlighting common handcrafted features and the growing use of deep architectures.

  • ResNet uses residual modules to enable deeper, lower-complexity networks, easier optimization, and mitigation of vanishing gradients.
  • U-net adapts fully convolutional networks for medical segmentation by adding upsampling and feature channels that preserve high-resolution texture information.
  • GoogLeNet approximates sparse structures with dense blocks to improve performance without increasing computational cost.
  • RNN and LSTM architectures are used for deep feature extraction, with RNNs suited to variable-length and time-series data.
  • From 2004 to 2019, texture features were most common, appearing in 51 reviewed papers, followed by color features, while shape features were least used.
  • Deep-learning features began appearing progressively from 2016, and the review summarizes their WSI CAD applications in Table 4.

4 Segmentation Methods

WSI segmentation divides images into meaningful regions to locate structures and abnormalities, using thresholding, region-based, graph-based, clustering-based, deep-learning, and other approaches.

  • Segmentation divides digital images into pixel regions with similar color, intensity, or texture to identify objects and boundaries.
  • Clinical requirements determine the segmentation detail, while image noise introduces uncertainty that can degrade the process.
  • From 2010 to 2020, papers using WSI segmentation to assist diagnosis increased from 2 to 28 across five major technique categories.
  • Threshold-based segmentation: Threshold segmentation separates foreground and background by grayscale differences and threshold selection, using methods such as Otsu and adaptive thresholds.
  • Threshold-based segmentation: Reviewed threshold-based studies applied background normalization, multiresolution analysis, MapReduce parallelization, or tumor-attribute modeling to WSI segmentation.
  • Region-based segmentation: Region-based methods locate regions through similarity, commonly using watershed or region-growing techniques.

Watershed Segmentation

Watershed segmentation treats an image as a topographic surface and separates regions through basins and watershed boundaries, supporting nuclei analysis in WSI.

  • The watershed algorithm maps grayscale values to terrain height, with low-valued basins separated by ridges called watersheds.
  • Watershed segmentation separates overlapping nuclei in several reviewed methods, including approaches using preprocessing, multiscale markers, post-processing, and result merging.
  • One automatic nuclei method uses watershed segmentation to extract size-related morphometric features and assess their prognostic value in male breast cancer.

Region Growing

The review covers region growing, graph, clustering, and deep-learning approaches for WSI segmentation, emphasizing multiresolution processing and increasingly prevalent neural architectures.

  • Region Growing: Region growing expands neighboring pixels from a seed according to growth criteria and termination conditions.
  • Region Growing: Region growing can handle complex images without prior knowledge, but its iterative process has relatively high space and time costs.
  • Graph-based Segmentation: Graph-based segmentation represents pixels as graph vertices and neighborhood relationships as weighted edges expressing similarity.
  • Graph-based Segmentation: Because WSIs are stored as tiled pyramids, graph-based methods can combine low-resolution area selection with high-resolution classification and top-down segmentation.
  • Clustering-based Segmentation: K-means clustering assigns samples to user-specified groups using converged cluster centers, and reviewed WSI studies applied it to segmentation and tumor separation.
  • Clustering-based Segmentation: One reviewed unsupervised k-means WSI segmentation method reported 95.5% accuracy, described as equivalent to supervised learning.
  • Deep-learning Segmentation: Ten reviewed papers used deep learning for WSI segmentation, including FCN, U-net, RAZN, boundary-aware networks, and multiscale architectures.
  • Deep-learning Segmentation: A U-net-based DCNN achieved DSC = 0.771 and F1 = 0.601 after five training runs, with wider contextual information benefiting performance.

5 Classification Methods

Image classification is a central WSI computer-aided diagnosis task that maps whole-slide images to disease categories. The reviewed literature includes traditional and deep learning approaches, with classification applications increasing over time.

  • Image classification forms a basis for related computer-vision tasks, including positioning, detection, and segmentation.
  • Classification maps one or more WSIs to a disease category to support pathological diagnosis.
  • Approximately 54 WSI classification studies published from 2004 to 2020 assisted pathologists in diagnosis.
  • The reviewed WSI classification methods include traditional machine learning, deep learning, and other approaches.
  • Eighteen reviewed papers used traditional machine learning algorithms for WSI computer-aided diagnosis.

SVM-based Classification Method

SVM-based WSI classification methods use handcrafted features, domain adaptation, and multi-resolution tissue analysis across several pathological applications.

  • SVM fundamentals: SVM classification separates WSI samples using learned hyperplanes in a high-dimensional feature space.SVM is a supervised machine-learning technique based on optimization and statistical learning theory.
  • WSI classification applications: Tile-based prostate cancer analysis combines texture extraction and spatial filtering with RF or SVM classifiers to generate probability maps.Cancer probabilities are calculated from classifier responses in an ensemble.
  • WSI classification applications: A heterogeneous GBM method classifies tissue sections with black tumor, pink necrosis, and green transition-to-necrosis labels.The classification outputs are compared with the original tissue images.
  • WSI classification applications: Colon cancer classification using intensity and GLCM texture features achieved mean accuracy of 96.67%, recall of 83.33%, and F-measure of 89.51%.The reported values use the complete feature set with an SVM classifier.
  • Adaptation and interpretability: Domain-adaptive kidney WSI classification trains on a source domain and uses a small number of user-labeled target samples, with interactive adaptation improving classification.Efficient linear SVM is used to reduce interaction waiting time.
  • WSI classification applications: SVM-based skin analysis combines epidermal and dermal segmentation with morphology, spatial-distribution, structural, and cytological features.Multi-class SVM classifiers are applied to the extracted epidermal and dermal features.

Random Forest-based Classification Method

Random Forest methods support WSI classification through ensemble tree models, feature-based pipelines, and combinations with deep-learning representations.

  • RF fundamentals: Random Forest is an ensemble of tree classifiers that assigns labels by voting and is described as fast, noise-robust, and capable of identifying nonlinear patterns.Each tree uses a randomly selected vector distributed across the forest.
  • RF classification applications: Necrotic-tissue quantification combines thresholding, color and texture features, classification, and spatial-context post-processing to correct isolated misclassified tiles.The processed tiles are merged into tissue sections.
  • RF classification applications: Traditional stomach-image classification compares RF-based image analysis with deep-learning methods after data augmentation.The cited method uses an RF classifier for the traditional image-analysis branch.
  • RF classification applications: Combining GoogLeNet-generated tumor probability heatmaps with RF classification identified metastatic breast cancer with an AUC of 0.925.The RF classifier distinguishes metastatic from negative breast-cancer WSIs.
  • RF classification applications: RF classification of lymph-node tissue after CNN-based tumor segmentation achieved an AUC score of 0.94.Features are extracted from the segmented tumor area before RF classification.
  • RF classification applications: RF models use normalized, downsampled, and texture-based DCIS images to predict high- and low-risk recurrence groups.The study reports training accuracy of 0.87 for predicting 10-year recurrence risk.

Others Traditional Machine Learning Classification Method

Traditional and deep learning methods classify WSI content using handcrafted features, classifiers, CNNs, multi-resolution strategies, and patch-level aggregation. Reported studies address diverse cancers and tasks, with results including high accuracy, AUC, sensitivity, and computational efficiency.

  • Traditional Machine Learning: 95% overall accuracy and 60% lower computational cost were reported for multi-resolution neuroblastoma classification using texture features, SFFS, and kNN confidence switching.Non-overlapping WSI blocks were processed in parallel, and higher resolution was used when confidence fell below a threshold.
  • Traditional Machine Learning: Traditional neuroblastoma grading combined segmented-tissue texture features with multiple classifiers and a two-step classifier-combination strategy.The classifier group included kNN, LDA, nearest mean, CORRLDA, Bayesian classification, and linear-kernel SVM.
  • Traditional Machine Learning: Bayesian localization and AdaBoost feature selection identified prostate cancer regions from multi-resolution WSIs using 10 features selected from more than 900 handcrafted features.The feature groups included first-order statistics, second-order co-occurrence features, and Gabor-filter features.
  • Deep Learning: Reported deep-learning results included AUC 0.962 for binary breast-slide classification, 93.0% accuracy for colorectal polyp classification, and 91.3% average accuracy for modified ResNet classification.The cited colorectal study also reported 88.3% recall and an 88.8% F1 score.
  • Deep Learning: Deep learning studies classified WSIs with CNNs, ResNet variants, AlexNet, autoencoder-derived representations, multiple-instance learning, and multi-resolution prediction aggregation.Approaches included patch tiling, background removal, feature selection, hard-example mining, and slide-level pooling or voting.
  • Deep Learning: Other studies reported 94.67 ± 14.60% accuracy with multi-resolution CNN voting, 98.1% AUC for unsupervised basal-cell-carcinoma analysis, and 87% accuracy for breast-cancer multiclass classification.The multi-resolution CNN study also reported 96.00±8.94% sensitivity, 92.00±17.85% specificity, and 96.24± 5.29% F-score; another study reported 78% average accuracy for five-class slide-level classification.

6 Detection Methods

WSI detection methods include traditional machine learning, ensemble learning, deep learning, and non-machine-learning approaches. Reported studies apply these methods to diverse pathological targets with varied performance measures.

  • Detection research has expanded steadily, with detection cases increasing from 2009 to 2019 as a major CAD-WSI application.
  • Detection approaches are grouped into traditional methods, ensemble learning, and deep learning.Traditional examples include SVM and image enhancement.
  • Traditional Detection Methods: SVM-based studies use handcrafted intensity, texture, LBP, and tissue-composition features for detecting cancers, necrosis, glomeruli, and abnormal regions.Reported examples include 96.67% precision, 83.33% recall, and 89.51% F-measure for colon cancer detection.
  • Deep Learning Detection Methods: Deep-learning detection studies include autofocus quality assessment and CNN-based invasive ductal carcinoma detection.ConvFocus achieved Spearman rank coefficients of 0.81 and 0.94 on two scanners; a CNN obtained 71.80% F-measure and 84.23% balanced accuracy.
  • Method Distribution: The review identifies six SVM studies, five studies using other traditional or ensemble algorithms, two non-machine-learning studies, and fifteen additional detection papers.

7 Methodology Analysis

The review analyzes prominent WSI approaches across segmentation, classification, detection, feature extraction, and related methods. It highlights multi-resolution segmentation, classifier trade-offs, CNN and MIL classification, and candidate methods for future WSI applications.

  • Segmentation: Segmentation studies predominantly use deep learning, especially multi-resolution U-Net architectures, but differing datasets prevent vertical effectiveness comparisons.
  • Segmentation: U-Net combines downsampling, upsampling, and skip connections for end-to-end segmentation, yet single-scale prediction limits handling of scale changes and deeper networks may generalize poorly.
  • Classification: Traditional classification frequently uses SVM, but basic SVM reports weaker results; ensemble combinations improve AUC while reducing efficiency and increasing computational demands.Examples include 84% accuracy for multi-class regularized SVM, 0.87 AUC for RBF SVM, and 0.95 AUC for an ensemble method.
  • Classification: Gaussian pyramids combined with kNN can improve multi-resolution classification, although kNN is vulnerable to minority-class bias in imbalanced datasets.An improved kNN with a Gaussian pyramid achieved 95% classification accuracy for neuroblastoma.
  • Classification: Deep-learning classification is dominated by CNNs, while MIL reduces noise by treating multiple examples as training bags and reports AUC values of 0.98 in cited studies.CNN-based models automatically extract features but are described as weak in interpretability.
  • Potential Approaches: The review considers transferring feature extraction, image-quality assessment, CRF, hierarchical segmentation, region-fitting, region-growing, and spectral-spatial methods from other fields to WSI CAD.

8 Conclusions and Future Work

The review identifies dominant datasets, features, and machine-learning approaches across WSI segmentation, classification, and detection. It concludes that deep learning has increasingly replaced traditional methods while highlighting dataset and deployment limitations for future work.

  • Cross-task methods: Deep learning algorithms have gradually replaced traditional machine-learning algorithms, and CNN-based methods perform well across segmentation, classification, and detection.
  • Datasets: TCGA and Camelyon are the two commonly used WSI datasets identified by the review.
  • Feature extraction: Color, texture, shape, and deep learning features are the most commonly used feature types.
  • Segmentation: Traditional segmentation methods are simple to calculate but sensitive to noise, while U-net-based segmentation has become mainstream in recent years.
  • Classification: Classification is the most studied task, with ensemble learning, multiple-instance learning, and neural networks providing better recognition ability.
  • Future work: Future work needs larger, comprehensive, fully annotated WSI datasets and networks requiring less computation and hardware while remaining interpretable.
Loading 2102.10553v1…