Source-linked AI summary
A Multi-resolution Model for Histopathology Image Classification and Localization with Multiple Instance Learning
Jiayun Li, Wenyuan Li, Anthony Sisk, Huihui Ye, W. Dean Wallace, William Speier, Corey W. Arnold
TL;DR
Whole-slide image analysis faces challenges from image size, tissue variability, and artifacts. The paper proposes a weakly supervised, multi-resolution MIL model for detection and grade classification, reporting strong classification and cancer-detection results across internal and external datasets.
Problem
Whole-slide image analysis must handle enormous images, variable tissue appearances, and staining and scanning artifacts.
Source: §I · p. 1Method
A two-stage attention-based MIL model uses slide-level labels to localize suspicious regions and classify selected tiles at higher magnification.
Source: B. Attention-based MIL with Instance… · p. 3, §VII · p. 8Results
MRMIL achieved 92.7% accuracy and 81.8% κ for benign, low-grade, and high-grade classification, 98.2% AUROC for malignant-slide detection, and external detection AUROC/AP of 99.4%/99.8%.
Source: §IV · pp. 6–7, §V · p. 8Takeaways & Limitations
Attention maps were consistent with cancerous regions identified by pathologists and could potentially help them localize relevant areas more quickly.
Source: §V · p. 7Attention maps were evaluated qualitatively; manual region-level annotations could provide a better metric for the attention module.
Source: §VI · p. 8
Abstract
from arXiv · showhide
Histopathological images provide rich information for disease diagnosis. Large numbers of histopathological images have been digitized into high resolution whole slide images, opening opportunities in developing computational image analysis tools to reduce pathologists' workload and potentially improve inter- and intra- observer agreement. Most previous work on whole slide image analysis has focused on classification or segmentation of small pre-selected regions-of-interest, which requires fine-grained annotation and is non-trivial to extend for large-scale whole slide analysis. In this paper, we proposed a multi-resolution multiple instance learning model that leverages saliency maps to detect suspicious regions for fine-grained grade prediction. Instead of relying on expensive region- or pixel-level annotations, our model can be trained end-to-end with only slide-level labels. The model is developed on a large-scale prostate biopsy dataset containing 20,229 slides from 830 patients. The model achieved 92.7% accuracy, 81.8% Cohen's Kappa for benign, low grade (i.e. Grade group 1) and high grade (i.e. Grade group >= 2) prediction, an area under the receiver operating characteristic curve (AUROC) of 98.2% and an average precision (AP) of 97.4% for differentiating malignant and benign slides. The model obtained an AUROC of 99.4% and an AP of 99.8% for cancer detection on an external dataset.
I. INTRODUCTION
Prostate biopsy review is time-consuming and variable, while whole-slide analysis must contend with large images, tissue variability, and artifacts. The paper proposes weakly supervised, multi-resolution MIL for slide classification and tumor-region localization.
I. INTRODUCTION
- Prostate biopsy review requires examining multiple levels and suspicious regions, and grading patterns such as ill-defined G4 versus tangentially sectioned G3 can vary between and within observers.The paper describes computer-aided tools as potential primary screening and second-reader support.Source: §I · p. 1
A. Related Work
- Earlier ROI methods relied on manually selected same-grade tiles, while slide-level segmentation approaches still required pixel-level annotations or pre-trained preprocessing networks.The cited Bulten framework used three pre-trained modules, each requiring pixel-wise annotations.Source: A. Related Work · pp. 1–2
- Attention-based MIL offers instance-importance visualization, but prior work did not address larger heterogeneous WSIs or use attention maps beyond visualization.Earlier prostate WSI work by Campanella et al. addressed malignant-versus-benign classification, not Gleason grading.Source: A. Related Work · p. 2
- 92.7% accuracy and 81.8% Cohen’s κ were achieved for benign, low-grade, and high-grade classification on the independent test set.The test set comprised 7,114 biopsy slides from 169 patients.Source: A. Related Work · p. 2
A. Problem Definition
The paper formulates whole-slide classification as weakly supervised MIL, aggregating tile features into slide-level predictions. It uses attention maps and instance dropout to support informative tile selection while reducing reliance on only a few instances.
A. Problem Definition
- A slide is a bag of tiles, but training provides only the bag-level label; under the stated binary MIL assumption, a positive bag contains at least one positive instance.The model uses attention-based aggregation rather than a predefined mean or maximum pooling function.Source: A. Problem Definition · p. 2
B. Attention-based MIL with Instance Dropout
- An attention module aggregates tile features into a bag representation and assigns attention values that indicate each instance’s relative contribution.The CNN feature extractor and attention aggregation are differentiable and trained end-to-end.Source: B. Attention-based MIL with Instance… · pp. 2–3
- The two-stage workflow detects benign versus malignant slides at 5x, then uses the attention map to select tiles for 10x cancer-grade classification.The overview describes clustering tiles by features and setting cluster tile counts from mean attention values.Source: Figure 1 · p. 3
- Sparse attention maps can select too few informative tiles for the later classification stage, so training randomly drops instances while evaluation uses all instances.Dropped-instance pixels are set to the dataset mean RGB value.Source: B. Attention-based MIL with Instance… · p. 3
C. Attention-based Tile Selection
Attention-weighted clustering selects suspicious tiles for the higher-resolution classification stage. This multi-resolution workflow mirrors low-magnification review followed by closer examination of selected regions.
C. Attention-based Tile Selection
- PCA-compressed instance representations are clustered by semantic similarity, and each cluster’s mean attention determines its relevance and tile allocation.The selection method avoids a fixed top-percentage threshold, which could over- or under-select when suspicious-region size varies across slides.Source: C. Attention-based Tile Selection · p. 3
D. Multi-resolution WSI classification
- Because whole slides contain billions of pixels and many irrelevant regions, the model first analyzes all tiles at 5x to detect benign versus malignant slides and generate attention maps.Pathologists typically scan slides at lower resolution before switching to higher magnification for suspicious areas.Source: D. Multi-resolution WSI classification · pp. 3–4
- Tiles selected by attention-based clustering are extracted from the same locations at 10x and passed to an MIL network for cancer-grade prediction.Source: D. Multi-resolution WSI classification · p. 4
III. EXPERIMENT
The experiments used patient-stratified prostate biopsy slide splits, tiles normalized for stain variability, and VGG11bn feature extractors with attention-based MIL in both model stages.
A. Dataset and data preprocessing
- Tiles were extracted at 10x as 256 × 256 pixels with 12.5% overlap, and those with less than 80% tissue were discarded.The internal slides were scanned at 20x and downsampled to 5x for detection; stain normalization was applied to both datasets.Source: A. Dataset and data preprocessing · p. 4
B. Implementation Details
- Both stages used VGG11bn feature extractors with a 1 × 1 convolution for dimensionality reduction and a two-layer attention module.Instance dropout probability was 0.5 for both stages.Source: B. Implementation Details · p. 4
A. Dataset and data preprocessing
- 20,229 slides from 830 patients were split by patient into 70% training, 10% validation, and 20% testing, yielding 7,114 test slides from 169 patients.The validation set contained 3,477 slides from 86 patients; labels came from pathology reports, with no pixel- or region-level annotations or extensive artifact curation.Source: 1) Dataset · p. 4
1) Dataset:
The experiments use a public prostate biopsy dataset and an external dataset, with tissue masking and staged training and clustering procedures. Class-imbalance weighting is used for grade-group prediction.
1) Dataset:
- SICAPV1 provides 10x tiles from 79 prostate biopsy slides; 19 slides are benign and the remainder malignant.The tiles are 512 × 512 pixels with 50% overlap.Source: 2) External dataset · p. 4
2) External dataset:
- Tissue masks are refined from low-magnification slides after HSV hue thresholding, using dilation and erosion.The passage describes these steps as removing isolated points and filling small gaps.Source: 3) Data preprocessing · p. 4
3) Data preprocessing:
- Training freezes the ImageNet-initialized feature extractor for three epochs, then trains its last three VGG blocks with attention and classification for 97 epochs.The detection and classification stages are trained separately using the same hyperparameters.Source: 3) Data preprocessing · p. 4
- PCA reduces tile embeddings from 256 to 32 dimensions before three-cluster K-means, while reversed-class-frequency weighting addresses grade-group imbalance.The three clusters are intended to group low-grade, high-grade, and benign tiles.Source: 3) Data preprocessing · pp. 4–5
C. Evaluation Metrics
Evaluation uses metrics suited to class imbalance and grade agreement, while visualizations examine learned representations and localized regions. Model comparisons include alternative feature, aggregation, and tile-selection approaches.
C. Evaluation Metrics
- AUROC and average precision supplement accuracy because more than 75% of test slides are benign; grade prediction is evaluated with Cohen’s Kappa.Kappa adjusts observer agreement for chance agreement.Source: C. Evaluation Metrics · p. 5
D. Model Visualization
- t-SNE maps flattened slide-level features into two dimensions to visualize representations from both model stages.The reported settings are learning rate 1.5×10^2 and perplexity 30.Source: D. Model Visualization · p. 5
- Grad-CAM uses activations and gradients from the top-attention tiles to localize discriminative regions within tiles for a target class.The attention map itself indicates only relative importance across tiles.Source: D. Model Visualization · p. 5
E. Model Comparison
- Comparisons include handcrafted tile features with XGBoost or random forest, Campanella et al.’s MIL model, and mean- or max-pooling alternatives to attention.Handcrafted features are aggregated by maximum pooling; classifier hyperparameters are selected by three-fold cross-validation.Source: E. Model Comparison · p. 5
- The single-stage baseline uses all tiles at 5x, while blue-ratio selection replaces the detection stage and selects the top 25% of tiles.Blue-ratio conversion is described as accentuating blue-channel signal to highlight proliferating nuclei.Source: E. Model Comparison · pp. 5–6
- Ablations compare models without instance dropout and with attention-only selection of the top 25% of tiles.Figure 2 shows ROC and PR curves for detection models trained with all tiles at 5x on internal and external data.Source: E. Model Comparison · p. 6, Figure 2 · p. 6
IV. RESULTS
The proposed model performs strongly on grade-group classification and malignant-versus-benign detection, including on an external dataset. Visual analyses examine attention-map localization, tile-level predictions, and learned slide representations.
IV. RESULTS
- 97.7% AUROC and 96.7% AP are achieved by the detection-stage model on the internal test set, rising to 99.4% AUROC and 99.8% AP externally.The model without instance dropout has slightly lower AUROC and AP on both datasets.Source: §IV · p. 6
- Attention maps learned from slide-level labels are reported as consistent with pathologists’ cancer-region markers, supporting their use to identify suspicious regions.The markers were masked out and not used during training.Source: §IV · p. 6
- Grad-CAM visualizations show that true-positive predictions capture relevant tile regions but can miss cancerous glands near tile boundaries; false positives can resemble cancer to pathologists.The visualization uses the two highest-attention tiles from each slide.Source: §IV · p. 6
- The detection-stage t-SNE features separate benign from malignant slides, while classification-stage features form distinct low- and high-grade clusters, with low-grade representations nearer benign slides.The visualization uses slide-level features from the detection model and selected high-resolution tiles in the classification model.Source: §IV · p. 7
- 92.7% accuracy and 81.8% Cohen’s Kappa are reported for benign, low-grade, and high-grade classification, outperforming the baselines; cancer detection reaches 98.2% AUROC and 97.4% AP.For Grade Group prediction, accuracy is 87.9%, quadratic κ is 86.8%, and κ is 71.1%.Source: §IV · p. 7
V. DISCUSSION
MRMIL achieved strong grade-classification and cancer-detection results, including on an external dataset. The discussion highlights attention maps for localization and interpretation, component contributions, and learned representations.
V. DISCUSSION
- The detection model performed on internal and external test sets; the external test comprised 7,114 slides from 830 patients collected without data curation.The authors suggest the external set's size and lack of curation may help explain its slightly better performance.Source: §V · p. 7
- Campanella et al. achieved 98.3% AUC and 97.3% AP for cancer detection, but performed worse than MRMIL on LG, HG, and BN classification.The authors contrast their attention aggregation, which selects discriminative tiles and combines tile features, with the prior model's top-tile selection and RNN aggregation.Source: §V · p. 7
- Attention maps from weak slide-level labels aligned with pathologist-identified cancerous regions; without instance dropout, attention focused on fewer tiles and κ was 77.3%, about 4% lower.The authors also report that attention-based models outperformed the Br-image model, whose most blue areas may not be diagnostically relevant.Source: §V · p. 7
- Grad-CAM highlighted relevant regions in true-positive tiles but sometimes missed boundary cancer; false positives included tangentially sectioned benign glands and seminal vesicle tissue.Although trained for acinar adenocarcinoma, the model also identified IDC-P in an example.Source: §V · p. 7
- Classification-stage embeddings formed distinct LG and HG clusters, with LG representations closer to benign; detection-stage embeddings did not distinctly separate LG from HG.The detection stage was trained to classify cancerous versus benign slides.Source: §V · p. 7
- Attention-map selection of higher-resolution tiles improved κ by 1% over Br selection, while instance dropout added over 3%.The authors summarize both as component contributions evaluated in Table II.Source: §V · p. 7
- 86.8% quadratic κ for GG prediction and 98.2% AUROC for detecting malignant slides; MRMIL achieved the highest κ for BN, LG, and HG classification.The GG-prediction result was comparable to state-of-the-art models requiring pre-trained segmentation networks.Source: §V · p. 8
VI. LIMITATIONS AND FUTURE WORK
The model has limitations in how it selects tiles, the cancer types it covers, the evaluation of its attention maps, and evidence of clinical utility.
VI. LIMITATIONS AND FUTURE WORK
- Tile selection is fixed by the detection stage, with no mechanism to update selected tiles from classification-stage loss.Future work could use recurrent networks or reinforcement learning to dynamically resample suspicious regions during training.Source: §VI · p. 8
- The model was developed only for acinar adenocarcinoma detection and classification in prostate biopsies; ductal adenocarcinoma and IDC-P remain to be investigated.Source: §VI · p. 8
- Attention maps were evaluated qualitatively by visual inspection against slides with pen markers, not quantitatively with manual region-level annotations.The authors state that region-level annotations could provide a better metric for the attention module.Source: §VI · p. 8
- The study has not yet evaluated whether the model facilitates clinical diagnosis, which the authors identify as a requirement for a successful CAD tool.They plan to investigate approaches for evaluating its effectiveness as a CAD tool.Source: §VI · p. 8
VII. CONCLUSION
The conclusion presents MRMIL as a weakly supervised, two-stage model evaluated on a large prostate-slide dataset and an external dataset, with potential clinical applications.
VII. CONCLUSION
- MRMIL combines detection and grade-classification stages, uses slide-level labels, and localizes cancerous regions.Source: §VII · p. 8
- The study visualizes saliency maps at slide and tile levels and learned representations to enhance interpretability.Source: §VII · p. 8
- The model was developed and evaluated on over 20,000 prostate slides from 830 patients and an external dataset; proposed clinical uses include efficiency, expert-level support for general pathologists, and second reads.Source: §VII · p. 8