Source-linked AI summary
Interpretable Survival Prediction for Colorectal Cancer using Deep Learning
Ellery Wulczyn, David F. Steiner, Melissa Moran, Markus Plass, Robert Reihs, Fraser Tan, Isabelle Flament-Auvigne, Trissia Brown, Peter Regitnig, Po-Hsuan Cameron Chen, Narayan Hegde, Apaar Sadhwani, Robert MacDonald, Benny Ayalew, Greg S. Corrado, Lily H. Peng, Daniel Tse, Heimo Müller, Zhaoyang Xu, Yun Liu, Martin C. Stumpe, Kurt Zatloukal, Craig H. Mermel
TL;DR
Deep-learning prognostic histopathology models can be difficult to interpret systematically. This study combines weakly supervised survival prediction with clustering-derived histologic features, finding that these features explain most DLS-score variance and reveal a reproducibly identifiable, independently prognostic tumor-adipose pattern.
Problem
Deep-learning prognostic histopathology models have limited systematically characterized, human-interpretable features explaining their predictions.
Method
The study developed a weakly supervised DLS for disease-specific survival prediction and clustered image-similarity embeddings to generate human-interpretable histologic features.
Results
Clustering-derived features explained the majority of DLS-score variance, whereas nine clinicopathologic features explained less than 20%.
Takeaways & Limitations
The approach identified a tumor-adipose feature that was strongly associated with high DLS risk, independently prognostic, and reproducibly identifiable by pathologists.
Takeaways & Limitations
Retrospective treatment confounding and the absence of geographically diverse validation limit conclusions about generalization and treatment effects.
Abstract
from arXiv · showhide
Deriving interpretable prognostic features from deep-learning-based prognostic histopathology models remains a challenge. In this study, we developed a deep learning system (DLS) for predicting disease specific survival for stage II and III colorectal cancer using 3,652 cases (27,300 slides). When evaluated on two validation datasets containing 1,239 cases (9,340 slides) and 738 cases (7,140 slides) respectively, the DLS achieved a 5-year disease-specific survival AUC of 0.70 (95%CI 0.66-0.73) and 0.69 (95%CI 0.64-0.72), and added significant predictive value to a set of 9 clinicopathologic features. To interpret the DLS, we explored the ability of different human-interpretable features to explain the variance in DLS scores. We observed that clinicopathologic features such as T-category, N-category, and grade explained a small fraction of the variance in DLS scores (R2=18% in both validation sets). Next, we generated human-interpretable histologic features by clustering embeddings from a deep-learning based image-similarity model and showed that they explain the majority of the variance (R2 of 73% to 80%). Furthermore, the clustering-derived feature most strongly associated with high DLS scores was also highly prognostic in isolation. With a distinct visual appearance (poorly differentiated tumor cell clusters adjacent to adipose tissue), this feature was identified by annotators with 87.0-95.5% accuracy. Our approach can be used to explain predictions from a prognostic deep learning model and uncover potentially-novel prognostic features that can be reliably identified by people for future validation studies.
Introduction
The study develops a weakly supervised deep-learning system for colorectal cancer survival prediction and a clustering-based approach to explain its histologic risk signals. The system generalized across validation cohorts, while clustering-derived features explained substantially more score variance than clinicopathologic features and highlighted a reproducibly identifiable prognostic tumor-adipose pattern.
- Results: 0.698 and 0.686 were the combined stage II/III 5-year AUCs in validation sets 1 and 2, respectively.The DLS also significantly stratified high- and low-risk groups in both validation sets.
- Results: Clinicopathologic features explained 18% of DLS-score variance in both validation sets, leaving most variance unexplained.Higher T and N categories were the features most significantly associated with higher DLS risk scores.
- Introduction: The DLS predicted disease-specific survival from routine, unannotated histopathology slides in stage II and III colorectal cancer.It was evaluated in two held-out validation sets from different time periods.
- Results: The DLS added significant prognostic value to nine clinicopathologic baseline features across two validation datasets.Its associations with disease-specific survival remained significant after adjustment for clinicopathologic features.
Competing interests
The paper reports employment, consulting, and stock-ownership relationships involving Google LLC and Medical University of Graz personnel. The TensorFlow framework and model architecture are publicly referenced.
- Several authors are current or former Google LLC employees and own Alphabet stock.
- Some authors are Google LLC consultants, while others are Medical University of Graz employees.
- The study uses TensorFlow and provides the deep learning architecture through pseudocode and standard Keras components.
Figures
The figures and tables describe cohort construction, survival discrimination, risk stratification, and interpretable histologic features associated with DLS predictions. Clustering-derived features and their patch-level scores are presented alongside clinicopathologic comparisons.
- Figure 1: Kaplan-Meier curves stratify stage II, stage III, and combined patients by DLS risk quartiles, using low risk as the reference group.
- Table 1: The cohort table separates development data from validation set 1 and a later validation set 2, with DSS inferred from cause-of-death ICD codes.
- Table 2: Table 2 reports 5-year AUC values for DLS-based disease-specific survival prediction.
- Table 4: Table 4 compares clinicopathologic and clustering-derived features as multivariable predictors of case-level DLS scores.
- Clustering-derived features: The highest-risk described feature contains small clusters of moderate-to-high-grade tumor cells intermixed with substantial adipose tissue.
- Table 5: Table 5 reports average and interquartile-range DLS scores across clustering-derived and known histologic features using block-bootstrap confidence intervals.
Tumor Segmentation Model Development
The tumor ROI model was developed from pathologist pixel-level annotations to distinguish colorectal adenocarcinoma from other tissue classes. Annotated slides were partitioned into training, tuning, and testing subsets for convolutional neural network development.
- Pathologists annotated adenocarcinoma, normal epithelium, atypical epithelium, necrosis, and other tumor-containing regions on 265 sampled slides.
- The annotations supported a model distinguishing colon adenocarcinoma from all other classes.
- After 21 slides were dropped for quality or completeness issues, 149 training, 51 tuning, and 44 testing slides remained.
- A convolutional neural network based on Inception-v3 was used for tumor-model development.
Region of Interest Mask Generation
The tumor model produces slide-wide probability maps that are thresholded and denoised into binary tumor ROI masks. These masks support downstream patch sampling, prognostic modeling, and analyses of clustering-derived features and survival.
- Region of Interest Mask Generation: Tumor probability heatmaps are generated across slides and converted into binary ROI masks by thresholding patch-level outputs.
- Region of Interest Mask Generation: Connected-component denoising removes selected positive regions before the ROI masks are used for downstream analysis.
- Supporting analyses: Supplementary figures show the dataset curation, DLS development, slide annotations, and example tumor-segmentation predictions with derived ROI masks.
- Supporting analyses: The study compares whole-tissue training with ROI-based training and evaluates model choices using tune-set performance.
- Supporting analyses: Supplementary analyses examine clustering-feature associations, TAF prevalence, survival curves, loss functions, and prognostic-model hyperparameters.
- Reporting and interpretation: The reporting materials include analyses of clinicopathologic associations, marker effects, multivariable effects, sensitivity checks, and study limitations.