Source-linked AI summary
Interpretable Survival Prediction for Colorectal Cancer using Deep Learning
Ellery Wulczyn, David F. Steiner, Melissa Moran, Markus Plass, Robert Reihs, Fraser Tan, Isabelle Flament-Auvigne, Trissia Brown, Peter Regitnig, Po-Hsuan Cameron Chen, Narayan Hegde, Apaar Sadhwani, Robert MacDonald, Benny Ayalew, Greg S. Corrado, Lily H. Peng, Daniel Tse, Heimo Müller, Zhaoyang Xu, Yun Liu, Martin C. Stumpe, Kurt Zatloukal, Craig H. Mermel
TL;DR
Interpretable prognostic features are difficult to derive from weakly supervised histopathology models, limiting systematic understanding of their predictions. This study clusters image-similarity embeddings to explain a colorectal cancer survival model and identifies a reproducible feature associated with high risk and poor prognosis.
Problem
Systematically characterizing histologic features learned by weakly supervised prognostic models remains limited, despite the need for precise colorectal cancer risk stratification.
Source: Introduction · p. 3, Discussion · p. 7Method
The study developed a weakly supervised survival model and generated interpretable histologic features by clustering embeddings from a deep-learning image-similarity model.
Source: Discussion · p. 6Results
A clustering-derived feature of poorly differentiated tumor clusters adjacent to adipose tissue was strongly associated with high risk, independently associated with poor prognosis, and reproducibly identified by pathologists.
Source: Discussion · p. 7Takeaways & Limitations
Clustering-derived features can explain deep-learning risk scores and reveal histologic features that people can reproducibly identify for future validation studies.
Source: Discussion · pp. 6–7Because the study was retrospective, treatment pathways may confound prognostic associations, and progression-free survival was unavailable at the required scale.
Source: Discussion · p. 8
Abstract
from arXiv · showhide
Deriving interpretable prognostic features from deep-learning-based prognostic histopathology models remains a challenge. In this study, we developed a deep learning system (DLS) for predicting disease specific survival for stage II and III colorectal cancer using 3,652 cases (27,300 slides). When evaluated on two validation datasets containing 1,239 cases (9,340 slides) and 738 cases (7,140 slides) respectively, the DLS achieved a 5-year disease-specific survival AUC of 0.70 (95%CI 0.66-0.73) and 0.69 (95%CI 0.64-0.72), and added significant predictive value to a set of 9 clinicopathologic features. To interpret the DLS, we explored the ability of different human-interpretable features to explain the variance in DLS scores. We observed that clinicopathologic features such as T-category, N-category, and grade explained a small fraction of the variance in DLS scores (R2=18% in both validation sets). Next, we generated human-interpretable histologic features by clustering embeddings from a deep-learning based image-similarity model and showed that they explain the majority of the variance (R2 of 73% to 80%). Furthermore, the clustering-derived feature most strongly associated with high DLS scores was also highly prognostic in isolation. With a distinct visual appearance (poorly differentiated tumor cell clusters adjacent to adipose tissue), this feature was identified by annotators with 87.0-95.5% accuracy. Our approach can be used to explain predictions from a prognostic deep learning model and uncover potentially-novel prognostic features that can be reliably identified by people for future validation studies.
Introduction
The study developed and evaluated a weakly supervised deep-learning system for disease-specific survival prediction in stage II–III colorectal cancer, alongside clustering-derived histologic features to explain its predictions. It examined clinicopathologic and histologic explanations using prespecified metrics and blinded annotation, with reproducible computational inference and ethics-reviewed data access.
Introduction
- The study addresses the need for finer risk stratification within TNM stages, where outcome variability and limitations of existing histoprognostic features complicate colorectal cancer treatment decisions.This motivation is especially relevant to selecting high-risk stage II patients for chemotherapy and tailoring therapy for stage III disease.Source: Introduction · p. 3
Methods
- The DLS sequentially segments tumor regions and predicts case-level DSS from sampled whole-slide-image patches using weakly supervised training and a Cox partial-likelihood survival loss.The tumor segmentation model achieved AUC 0.985, and tumor-based regions improved prognostic performance over a simple tissue detector.Source: Results · p. 4, Methods · p. 10
Results
- 5-year AUCs for combined stage II/III disease were 0.698 and 0.686 in validation sets 1 and 2, respectively, with significant risk stratification in both cohorts.High- versus low-risk groups also showed substantial survival differences across stage II and III cases, with log-rank p<0.001.Source: Results · p. 4
- Adding the DLS to clinicopathologic features increased combined-cohort 5-year AUC by 0.055 and 0.038, yielding final AUCs of 0.733 and 0.721.The DLS remained significantly associated with DSS after adjustment, with p<0.001 in both validation sets.Source: Results · p. 5
- Clustering-derived histologic features explained 73% and 80% of DLS-score variance, compared with 18% explained by clinicopathologic features in both validation sets.The complete 200-feature set captured most variance, while a selected subset of 10 features explained 57% and 61%.Source: Results · p. 5
- TAF patches received the highest average patch-level DLS score at 2.76, exceeding the other high-risk clustering-derived features.Known high-scoring features included lymphovascular invasion at 1.03 and perineural invasion at 0.75.Source: Results · p. 6
- The tumor-adipose feature, defined by poorly differentiated tumor clusters adjacent to adipose tissue, had the strongest association with high DLS scores and was independently prognostic.Its association with survival persisted in T3 cases, suggesting prognostic value beyond T-category.Source: Results · p. 6, Discussion · p. 8
- TAF was identified with 87.0–95.5% accuracy by pathologists and non-pathologists, with 93.5% inter-pathologist concordance.The feature’s reproducible visual appearance supports its use in future biological and prognostic validation studies.Source: Results · p. 6, Discussion · p. 7
Methods
- The prognostic model used randomly sampled, color-normalized and augmented patches from stage II and III cases, with hyperparameters selected across 100 configurations and distributed training for 2 million steps.Five top-tuning-set checkpoints were ensembled, and case-level scores were generated exhaustively across nonoverlapping patches within each ROI mask.Source: Methods · p. 11
- The DLS was assessed independently for stages II, III, and combined using prespecified 5-year survival AUC, hazard ratio, and c-index analyses, with censoring handled according to each metric.Cox models quantified hazard ratios and c-indices for the DLS, clinicopathologic baseline, and combined models; confidence intervals for c-indices used 9,999 paired bootstrap samples.Source: Methods · pp. 11–12
- Exploratory interpretation related standardized DLS scores to clinicopathologic variables and 200 k-means histologic features derived from image-similarity embeddings, selecting 10 features by forward stepwise maximization of R2.Two blinded pathologists reviewed representative patches and scored tumor, stroma, adipose, TILs, grade, fibrosis, and free-text morphology.Source: Methods · pp. 12–13
- Known histoprognostic features were exhaustively annotated on 161 slides, and average patch-level DLS scores were compared across these annotations and clustering-derived features using slide-level blocked bootstrapping.A separate blinded TAF-identification study used representative, randomly sampled, and independent patches reviewed by two anatomic and three non-anatomic pathologists.Source: Methods · pp. 13–14
- 11±7 minutes per case on one 16-core machine, 13±8 seconds across 50 such machines, and 8±5 seconds on a Google Cloud TPU characterized model inference speed.These timings were comparable to or faster than slide preparation and digitization, which can take minutes per slide across approximately 10 slides per case.Source: Methods · p. 14
Data availability
- Archived anonymized pathology slides, clinicopathologic variables, and outcomes are available through the Medical University of Graz Biobank subject to contact, noncommercial academic review, and ethics approval.Requests for access should be directed to K. Z.Source: Data availability · p. 14
Competing interests
Several authors disclosed current or past employment, consulting relationships, and stock ownership involving Google LLC, while other authors were employed by the Medical University of Graz. The study’s TensorFlow framework and architecture implementation details are publicly available.
Competing interests
- Google affiliations included current or past employment and Alphabet stock ownership for 16 authors, plus two Google consultants; five other authors were Medical University of Graz employees.These relationships are disclosed as competing interests.Source: Competing interests · p. 15
Code availability
- The TensorFlow framework, detailed architecture pseudocode, and standard Keras SeparableConv2D implementation are publicly available.The architecture pseudocode is provided in the S1 algorithm, with links to TensorFlow and Keras resources.Source: Code availability · p. 15
Figures
The figures show that DLS-based risk stratification separates survival groups across stage-specific and combined cohorts, while clustering-derived histologic features explain most DLS-score variance and identify a prognostic tumor-adipose pattern.
Figures
- DLS stratification separates high- and low-risk groups in stage II, stage III, and combined stage II/III cohorts across both validation sets using Kaplan–Meier analysis.High- and low-risk groups correspond to the highest and lowest DLS-risk quartiles, with hazard ratios and log-rank p-values reported relative to the low-risk group.Source: Figure 1 · p. 19
- Clustering-derived histologic features explain the majority of case-level DLS-score variance, with four high-risk and six low-risk features illustrated by representative patches.The ten features are ranked by average DLS score, and each is represented by patches closest to its cluster centroid from distinct cases.Source: Figure 2 · p. 20
- The highest-risk clustering-derived feature is the tumor-adipose feature, whose quantitation supports survival analysis across both validation sets.In stage II cases, 50% of validation-set-1 cases and 38% of validation-set-2 cases lacked any tumor-adipose feature, affecting low- and medium-risk at-risk counts.Source: Figure 3 · p. 21
Tables
- The study uses development and two validation sets, with validation set 2 collected after model development and disease-specific survival inferred from cause-of-death ICD codes.Cases came from the Institute of Pathology and BioBank at the Medical University of Graz, and only colorectal-tissue slides were included.Source: Table 1 · p. 22
- Multivariable regression models assess clinicopathologic and clustering-derived features as predictors of case-level DLS scores, with both overall models reported as significant at p <0.001.Coefficients represent the relative increase in DLS score associated with each input variable; statistically significant variables are indicated separately.Source: Table 4 · p. 25
Tumor Segmentation Model Development
A tumor-region segmentation model was developed from pathologist-annotated colorectal histology slides, using an Inception-v3-based convolutional neural network to distinguish adenocarcinoma from other tissue classes.
Tumor ROI annotation
- Pathologists annotated 265 sampled slides for adenocarcinoma, normal epithelium, atypical epithelium, necrosis, and tumor-associated “other” regions.The “other” category included fibrosis, ulceration, extensive tumor stroma, and treatment-effect regions; normal non-epithelial tissue was excluded.Source: Tumor Segmentation Model Development · p. 28
Data splitting
- After quality review, 149 slides were used for training, 51 for tuning, and 44 for testing.The annotated slides were initially split 3:1:1, but 21 were dropped for quality issues or incomplete annotations.Source: Tumor Segmentation Model Development · p. 28
Model training
- A reduced-parameter Inception-v3 convolutional neural network was trained patch-wise to distinguish adenocarcinoma from all other classes.The model used depth_multipler=0.1, with architecture and hyper-parameter details reported in Supplementary Table S8.Source: Tumor Segmentation Model Development · p. 28
Region of Interest Mask Generation
Binary ROI masks were generated from tumor-probability heatmaps by thresholding, denoising, and dilation to include tumor-proximal regions. Threshold and dilation parameters were tuned for prognostic performance, and only patches substantially overlapping the resulting masks were analyzed.
Region of Interest Mask Generation
- Tumor-probability heatmaps were thresholded, denoised by removing components smaller than 8 superpixels, and dilated to include tumor-proximal regions.Heatmaps used 32μm superpixels, with dilation implemented using a circular filter.Source: Region of Interest Mask Generation · p. 28
- Three recall-based thresholds and three dilation radii were evaluated, with the combination maximizing tune-split DLS performance selected.Thresholds corresponded to 95%, 90%, and 75% recall; dilation radii were 0, 4, and 16 superpixels.Source: Region of Interest Mask Generation · p. 28
- During inference, ROI masks were aligned to 256-pixel patches at 5X magnification, and only patches with at least half their area inside the mask were used.At 5X, each patch covered 512μm.Source: Region of Interest Mask Generation · p. 28
Supplementary Figures
- The tumor model produced ROI heatmaps that were binarized into masks for prognostic patch sampling, while prognostic inference aggregated ROI patches into case-level risk scores.The tumor model was trained from pixel-level pathologist annotations to identify colorectal adenocarcinoma.Source: Figure S2 · p. 30, Figure S4 · p. 32
Supplementary Tables
- Tumor segmentation performance was evaluated at three thresholds selected according to recall on the tuning split, with AUC 98.50 reported for the test split.The thresholds supported the ROI-mask generation procedure used downstream.Source: Table S9 · p. 48