Source-linked AI summary
Patch-based Convolutional Neural Network for Whole Slide Tissue Image Classification
Le Hou, Dimitris Samaras, Tahsin M. Kurc, Yi Gao, James E. Davis, Joel H. Saltz
TL;DR
Direct CNN training on gigapixel WSIs is computationally impractical, even though subtype information appears in high-resolution patches and not every patch is discriminative. The paper combines EM-based discriminative-patch identification with learned decision fusion of patch predictions, achieving WSI subtype accuracy similar or close to pathologist agreement and favorable patch-based results on smaller images.
Problem
CNNs cannot be applied directly to gigapixel WSIs, while subtype evidence is concentrated in high-resolution patches with unknown and potentially inconsistent patch labels.
Method
An EM-based patch-level CNN identifies discriminative patches using spatial smoothing, then logistic regression or SVM fuses patch-prediction histograms into image-level labels.
Results
WSI subtype accuracy is similar or close to inter-observer agreement between pathologists, and combining patch-level classifiers can outperform max-pooling or voting.
Takeaways & Limitations
Patch-based CNNs with supervised decision fusion are effective for WSI cancer-subtype classification and compare favorably with image-based CNNs on a comparable smaller-image dataset.
Takeaways & Limitations
The method assumes non-discriminative instances follow a uniform generative model, and future work proposes incorporating them into the EM data likelihood instead.
Abstract
from arXiv · showhide
Convolutional Neural Networks (CNN) are state-of-the-art models for many image classification tasks. However, to recognize cancer subtypes automatically, training a CNN on gigapixel resolution Whole Slide Tissue Images (WSI) is currently computationally impossible. The differentiation of cancer subtypes is based on cellular-level visual features observed on image patch scale. Therefore, we argue that in this situation, training a patch-level classifier on image patches will perform better than or similar to an image-level classifier. The challenge becomes how to intelligently combine patch-level classification results and model the fact that not all patches will be discriminative. We propose to train a decision fusion model to aggregate patch-level predictions given by patch-level CNNs, which to the best of our knowledge has not been shown before. Furthermore, we formulate a novel Expectation-Maximization (EM) based method that automatically locates discriminative patches robustly by utilizing the spatial relationships of patches. We apply our method to the classification of glioma and non-small-cell lung carcinoma cases into subtypes. The classification accuracy of our method is similar to the inter-observer agreement between pathologists. Although it is impossible to train CNNs on WSIs, we experimentally demonstrate using a comparable non-cancer dataset of smaller images that a patch-based CNN can outperform an image-based CNN.
1. Introduction
WSI subtype classification requires preserving cellular-scale details while handling gigapixel computational constraints and heterogeneous, non-discriminative patches. The paper addresses this with EM-based patch selection and learned decision fusion, reporting strong pathology-relevant results.
- Motivation: Gigapixel WSIs make direct CNN classification computationally impractical, while downsampling can lose discriminative cellular-scale details.The paper also notes that direct CNNs may learn from only one of multiple discriminative patterns.
- Motivation: Patch-level labels are unavailable, and simple voting or max-pooling can fail when patch predictions do not match the WSI-level label.Multiple discriminative regions and mixed tumor structures make single-patch or naive aggregation unreliable.
- Method: The method trains patch-level CNNs and uses an EM procedure with spatial smoothing to identify discriminative patches iteratively.The hidden patch variable indicates whether a patch’s true label matches the image label.
- Method: An image-level decision-fusion model aggregates patch-level predictions using prediction histograms rather than relying on max-pooling or voting.Multiclass logistic regression or SVM is used for image-level prediction under a Count-based Multiple Instance formulation.
- Results: The approach achieves cancer-subtype classification accuracy similar or close to inter-observer agreement between pathologists, with larger improvements on harder cases.The paper also reports multiple state-of-the-art results on TCGA WSIs.
- Results: Experiments on a comparable non-cancer dataset show that combining multiple patch-level classifiers can be advantageous relative to whole-image classification.This provides experimental support for the patch-based strategy beyond WSI classification.
2. EM-based method with CNN
The method models each high-resolution image as a bag of patches with unobserved discriminative-status variables, then uses EM to estimate those variables and train a CNN.
- Each image is represented as a bag of instances, with each patch paired with the image-level label.
- A hidden binary variable indicates whether each patch is discriminative for its image label.
- EM alternates between estimating hidden discriminative statuses and updating CNN parameters to maximize data likelihood.The initial E step treats all instances as discriminative, while the M step updates θ.
- The model assumes non-discriminative instances follow a uniform generative model, simplifying the optimization to a discriminative CNN model.
- Under the SMI interpretation, the instance with maximum P(H_i,j | X) is selected as the discriminative instance for a positive bag.
3. Discriminative patch selection
Discriminative patches are selected from spatially smoothed CNN predictions using thresholds defined at both image and class levels.
- Patches whose P(H_i,j | X) exceeds threshold T_i,j are retained for continued CNN training.
- Two CNNs trained at different scales are averaged, and Gaussian smoothing produces a more robust estimate of each patch’s discriminative probability.Smoothing preserves potentially informative hard-to-classify or boundary patches that raw confidence might reject.
- T_i,j is the minimum of an image-level percentile threshold and a class-level percentile threshold.The image threshold guarantees a minimum retained proportion, while the class threshold adapts to different class priors.
4. Image-level decision fusion model
The image-level model aggregates patch predictions through learned decision fusion rather than relying on a single patch or simple pooling.
- A multi-class logistic regression or RBF-kernel SVM receives all patch-level predictions and outputs the image-level label.
- Decision fusion is motivated by the need to combine multiple patches when individual predictions are insufficient or jointly informative.The mixed glioma example requires recognizing two jointly present subtype patterns.
- Class histograms formed by summing patch-level probabilities are concatenated across four CNN models for robust fusion.The four models use two patch scales and two iteration counts.
5. Experiments
The experiments evaluate patch-based CNN methods for glioma and NSCLC subtype classification on TCGA WSIs, comparing fusion strategies and EM-based discriminative-patch selection. Results approach pathologist agreement, improve on difficult subtypes, and show patch-based CNNs outperform image-based CNNs on a comparable smaller-image dataset.
- Experiment scope: The experiments classify glioma and NSCLC cases into subtypes using TCGA whole-slide images.The WSI dataset typically contains images of approximately 100K by 50K pixels.
- Compared methods: The evaluated baselines include voting, max-pooling, feature fusion, logistic regression, SVM, pretrained CNN features, and a non-CNN baseline.CNN-Vote averages patch predictions, whereas CNN-SMI uses the maximum patch probability; second-level models learn image-level predictions from patch outputs.
- Compared methods: The EM-based variants train patch-level CNNs on discriminative patches identified by the E-step and aggregate their predictions with alternative fusion models.The method uses spatial smoothing and predefined image- and class-level discriminative thresholds.
- Glioma results: 77% accuracy on glioma classification is similar to pathologist agreement of approximately 70% and around 80%.The reported pathologist agreement came from a similar dataset, with agreement after joint review reaching only around 80%.
- Glioma results: 97% accuracy distinguishes GBM from LGG, while five LGG subtypes achieve 57.1% accuracy against 36.7% chance.Most confusions involve oligoastrocytoma, whose mixed morphology is challenging for pathologists as well.
- Subtype results: The proposed method improves recognition on difficult subtypes, reaching an OA F-score of 0.544 versus 0.482 for CNN-LR and 0.426 for PreCNN-Fea-SVM.For NSCLC ADC-mix, EM-Finetune-CNN-SVM reaches an F-score of 0.472 versus 0.412 for Pretrained CNN-Fea-SVM.
- Patch versus image classification: On a comparable non-cancer smaller-image dataset, EM-CNN-SVM and EM-CNN-Fea-SVM outperform the conventional image-based CNN-Image method.Patch-extracted pretrained CNN features also outperform image-extracted pretrained CNN features.
6. Conclusions
The paper presents a patch-based CNN with supervised decision fusion for WSI classification and reports performance comparable to pathologists. Experiments also favor patch-based over image-based methods on smaller non-cancer images, while scalability remains future work.
- The patch-based CNN with supervised decision fusion successfully classifies Whole Slide Tissue Images.
- The method achieves cancer-subtype accuracy similar or close to inter-observer agreement between pathologists.
- Future work will incorporate non-discriminative patches into the EM likelihood instead of assuming they are uniformly distributed.
- The patch-based methods EM-CNN-SVM and EM-CNN-Fea-SVM significantly outperform image-based methods on rail surface defect severity classification.
- Future work will optimize CNN training to scale to large-scale pathology datasets.
Appendix A. Description of cancer subtypes
The appendix describes glioma and NSCLC subtypes using diagnostic grades, cellular classifications, and mixed-subtype labels. It also lists the abbreviations used for these cancer categories.
- Glioblastoma (GBM) is a WHO grade IV glioma, while oligodendroglioma (OD) and diffuse astrocytoma (DA) are WHO grade II gliomas.
- Oligoastrocytoma (OA) is a WHO grade II mixed glioma, whereas anaplastic oligoastrocytoma is a WHO grade III mixed glioma.
- Anaplastic astrocytoma (AA) and anaplastic oligodendroglioma (AO) are WHO grade III glioma subtypes.
- Low-Grade-Glioma (LGG) includes OD, OA, DA, AA, and AO.
- The NSCLC labels include squamous cell carcinoma (SCC), adenocarcinoma (ADC), and mixed-subtype adenocarcinoma (ADC-mix).