Source-linked AI summary
Whole Slide Images based Cancer Survival Prediction using Attention Guided Deep Multiple Instance Learning Networks
Jiawen Yao, Xinliang Zhu, Jitendra Jonnagaddala, Nicholas Hawkins, Junzhou Huang
TL;DR
Existing image-based survival models face scalability and annotation challenges when analyzing large, heterogeneous WSIs. DeepAttnMISL uses Siamese MI-FCN feature learning and attention-based MIL pooling to aggregate WSI information at the patient level. Evaluations on two large cancer WSI datasets support its effectiveness, interpretability, and suitability for large-scale survival prediction.
Problem
Existing WSI survival models face computational limitations, dependence on selected patches or clusters, and a shortage of weakly supervised approaches that learn patient-level representations without annotations.
Method
DeepAttnMISL uses phenotype clusters as multiple-instance representations, Siamese MI-FCNs for feature learning, and attention-based MIL pooling for trainable patient-level aggregation.
Results
The proposed method achieves the highest C-index and AUC among the compared survival models and produces the most significant reported log-rank result, with p-value = 4.527 × 10−3.
Takeaways & Limitations
The framework can highlight prognosis-related patterns, estimate individual patient risk, and support treatment-group analysis for personalized medicine.
Takeaways & Limitations
Comparisons with WSISA are constrained because WSISA becomes computationally impractical beyond approximately 100K–200K patches, so the authors evaluate their method alone at around one million patches.
Abstract
from arXiv · showhide
Traditional image-based survival prediction models rely on discriminative patch labeling which make those methods not scalable to extend to large datasets. Recent studies have shown Multiple Instance Learning (MIL) framework is useful for histopathological images when no annotations are available in classification task. Different to the current image-based survival models that limit to key patches or clusters derived from Whole Slide Images (WSIs), we propose Deep Attention Multiple Instance Survival Learning (DeepAttnMISL) by introducing both siamese MI-FCN and attention-based MIL pooling to efficiently learn imaging features from the WSI and then aggregate WSI-level information to patient-level. Attention-based aggregation is more flexible and adaptive than aggregation techniques in recent survival models. We evaluated our methods on two large cancer whole slide images datasets and our results suggest that the proposed approach is more effective and suitable for large datasets and has better interpretability in locating important patterns and features that contribute to accurate cancer survival predictions. The proposed framework can also be used to assess individual patient's risk and thus assisting in delivering personalized medicine. Codes are available at https://github.com/uta-smile/DeepAttnMISL_MEDIA.
1 Introduction
Whole-slide survival prediction is difficult because WSIs are enormous, heterogeneous, and costly to annotate, while ROI-based models may miss tumor morphology. DeepAttnMISL addresses this with weakly supervised multiple instance learning and attention-based patient-level aggregation.
- 1 Introduction: WSI survival prediction must handle terabyte-scale images, laborious expert inspection, and limited scalability of conventional computational approaches.These constraints make direct analysis of complete WSIs computationally challenging and clinically burdensome.
- Related work: ROI-based methods depend on pathologist-selected patches or handcrafted features, which may incompletely represent a patient’s heterogeneous tumor morphology.Deep convolutional survival models also commonly average patch-level predictions, limiting adaptive aggregation.
- Related work: Weakly supervised WSI classification aggregates tile predictions, but survival risk may depend on diverse histology patterns and inter- and intra-tumor heterogeneity.This makes survival modeling more complex than identifying a single differentiated region or tumor type.
- Contributions: DeepAttnMISL formulates each patient’s WSI data as a multiple instance bag with phenotype clusters as instances and only patient-level survival labels.This matches the multiple instance assumption that not every instance in a bag is relevant to the label.
- Contributions: The model uses a Siamese MIL-based network to extract morphology-specific phenotype features and attention-based pooling to learn trainable patient-level aggregation.Attention weights are parameterized by neural networks rather than fixed pooling operators.
- Contributions: The framework is designed to exploit whole-slide patterns, identify prognosis-related regions, and support patient risk assessment and treatment-group analysis.The authors evaluate it on two large cancer WSI datasets and present it as a possible treatment recommender system.
2 Methodology
DeepAttnMISL formulates patient survival prediction as weakly supervised multiple instance learning over phenotype clusters from whole slide images. A siamese MI-FCN extracts cluster features, attention-based pooling aggregates them into patient-level representations, and a negative log partial likelihood trains the risk model.
- Problem formulation: Each patient is represented by multiple WSIs and patient-level survival labels, without pixel-level annotations, motivating a multiple instance learning formulation.In this formulation, each patient is a bag whose instances are phenotype clusters rather than individually labeled patches.
- Sampling and clustering: Phenotype clusters replace raw patches as bag instances, reducing complexity while preserving morphology-specific information from heterogeneous tissue patterns.Patches from all patient WSIs are clustered into phenotypes before feature learning.
- Siamese MI-FCN: A siamese MI-FCN uses shared-weight subnetworks to learn representations for different phenotype clusters and accommodates variable numbers of patches per phenotype.Each MI-FCN applies convolutional and ReLU layers followed by global pooling; fully convolutional design avoids dependence on a fixed spatial resolution.
- Aggregation via attention-based MIL pooling layer: Attention-based MIL pooling assigns trainable, phenotype-specific weights to local representations instead of using fixed maximum or mean pooling.The resulting patient-level representation can emphasize informative clusters and provide potential regions of interest.
- Loss function: The model outputs a patient hazard risk and is trained by minimizing the negative log partial likelihood over event times and risk sets.The risk set contains individuals still under study at each event time, and the loss penalizes discordant risk ordering.
3.1 Dataset Description
The study evaluates DeepAttnMISL on two large, high-resolution cancer WSI datasets: NLST for lung cancer and MCO for colorectal cancer. MCO includes more than 1,500 WSIs from over 1,500 Australian colorectal cancer patients, while experiments differ because only NLST has tumor-region annotations.
- Datasets: The datasets comprise NLST lung cancer WSIs and the MCO colorectal cancer imaging study.MCO contains imaging, specimen, clinical, and genetic data from over 1,500 individuals who underwent curative resection.
- MCO dataset: MCO includes more than 1,500 WSIs, with at least one H&E-stained 40x section representing each tumor case.The WSIs were scanned using a 40x objective, and follow-up clinical data were collected for all cases.
- Experimental scope: The experiments use different comparison settings because tumor-region annotations are available only for NLST.Both datasets support WSI-based models without ROI labeling, but ROI-based comparisons are possible only on NLST.
- Patch scale: Baseline WSI models use 130K MCO patches and 275K NLST patches, while an expanded MCO setting contains 915K patches.The expanded MCO dataset assigns more than 500 patches to each WSI and includes over 1,000 patients.
- Dataset statistics: Table 2 reports the numbers of WSIs, patients, patches, and average patches per WSI for each dataset.These quantities characterize the scale of the datasets used for evaluation.
3.2 Implementation details
The model is trained with Adam and evaluated using C-index and AUC. C-index measures the ranking quality of predicted risks using comparable patient pairs.
- Optimization: Training uses Adam optimization with weight decay 5 × 10^-4 and learning rate 10^-4.Training monitors validation loss and early stops when the loss increases substantially.
- Evaluation metrics: Performance is evaluated with the concordance index (C-index) and area under curve (AUC).These are the reported survival-prediction evaluation metrics.
- C-index: C-index quantifies risk-ranking quality over comparable patient pairs using observed times and predicted risks.Its value ranges from 0 to 1, with larger values indicating better prediction.
3.3 MCO results
MCO experiments examine phenotype-cluster counts, network components, attention mechanisms, and comparisons with WSISA. DeepAttnMISL consistently performs best in the reported settings, while attention weights provide patient-specific visual interpretation.
- Settings and Parameters: DeepAttnMISL achieves the best results across phenotype-cluster settings, outperforming fixed Max and Mean pooling baselines.The authors report that attention MIL pooling is more flexible than fixed pooling, while increasing clusters beyond suitable values can worsen results.
- Settings and Parameters: A one-layer convolutional-ReLU MI-FCN with Global Average Pooling is selected for the final network configuration.The study compares one-, two-, and three-layer configurations with different filter counts.
- Settings and Parameters: Removing the Siamese network produces worse overall performance than DeepAttnMISL on MCO-130K.The comparison uses five-fold cross-validation with six phenotype clusters, supporting the contribution of phenotype clusters in the Siamese network.
- Settings and Parameters: InceptionV3 with K-means clustering obtains a C-index of 0.598±0.054 in five-fold cross-validation.The study also evaluates spectral clustering while keeping other settings and architectures unchanged.
- Settings and Parameters: Plain and gated attention perform similarly across phenotype-cluster settings, with plain attention slightly better on MCO-130K.The gated variant adds a tanh(.) non-linearity to the attention mechanism.
- Comparisons: DeepAttnMISL outperforms WSISA-based models across reported MCO-130K cluster settings in both C-index and AUC.The comparison includes WSISA features combined with Lasso-Cox, En-Cox, and MTLSA survival models.
- Comparisons: Results on MCO-1M and MCO-130K show no significant differences, suggesting sampling strategy does not affect the proposed method in cross-validation.The method consistently performs better than WSISA models across the displayed settings.
- Interpretability: DeepAttnMISL heatmaps identify high-risk regions mainly within tumor areas, whereas WSISA selects many non-tumor patches.The authors attribute WSISA’s less reliable patient-specific selection to clustering patches across the whole database and inter-patient heterogeneity.
3.4 Lung Cancer dataset results
The lung cancer experiments compare the proposed survival model with ROI-based, classical, and other deep learning approaches. Results indicate that phenotype-based modeling and attention-guided aggregation improve prediction and risk stratification.
- Phenotype number: Using 10 phenotypes produces slightly better predictions than using 6, 8, or 12, while fewer clusters perform poorly.The authors attribute this pattern to heterogeneity in lung cancer patches and the difficulty of learning survival-related representations from fewer phenotypes.
- Prediction performance: The proposed method achieves the highest C-index and AUC among the compared survival models on 5-fold cross-validation.Hand-crafted baselines perform poorly because they provide limited local information and use ineffective aggregation for heterogeneous tumors and patients.
- Representation learning: The model learns complex deep bag representations from phenotype patterns instead of relying on a small set of ROI patches and hand-crafted features.This design addresses limitations in local patch information and patch-level aggregation of tumor and patient heterogeneity.
- Risk stratification: The proposed model more successfully separates testing death patients into low- and high-risk groups than other methods across all datasets.Risk groups are formed using the median predicted risk score, with longer-surviving patients expected in the low-risk group.
- Risk stratification: Across five testing folds, the proposed model achieves four significant log-rank results and one marginal result with p = 0.09.DeepMISL achieves two significant and one marginal result, while the other listed baselines achieve fewer significant results.
3.5 Ensemble Models
The study tests whether averaging predictions from five models per fold improves survival prediction. The reported results show no additional predictive benefit from ensembling.
- Ensemble comparison: Ensemble models do not provide additional prediction power over single models on either MCO-1M or NLST.The averaged C-index is 0.606 for MCO single versus 0.600 for MCO ensemble, and 0.696 for NLST single versus 0.695 for NLST ensemble.
- Ensemble procedure: The ensemble procedure averages prediction scores from five models trained during each cross-validation fold.The maximum cluster number is set to 6 and 10, respectively, for the reported comparisons.
4 Conclusion
The paper concludes that DeepAttnMISL learns survival patterns directly from gigapixel WSIs without annotations and supports flexible, interpretable patient-level risk prediction. The framework is presented as applicable to large cancer datasets and potentially other tumor types.
- Conclusion: DeepAttnMISL directly learns survival patterns from gigapixel images without annotations, making it more suitable for large-scale cancer datasets.The framework handles whole slide images with varying numbers and sizes across patients.
- Conclusion: Attention-based MIL pooling provides flexible and interpretable aggregation while presenting reasonable regions of interest associated with clinical outcome prediction.The paper contrasts this trainable aggregation with fixed aggregation techniques in existing survival models.
- Conclusion: The framework can assess individual patient risk and may assist doctors with personalized treatment decisions.The paper states that future research could extend the approach to other tumor types.