Modeling Dense Multimodal Interactions Between Biological Pathways and Histology for Survival Prediction

Guillaume Jaume; Anurag Vaidya; Richard Chen; Drew Williamson; Paul Liang; Faisal Mahmood

Source-linked AI summary

Modeling Dense Multimodal Interactions Between Biological Pathways and Histology for Survival Prediction

Guillaume Jaume, Anurag Vaidya, Richard Chen, Drew Williamson, Paul Liang, Faisal Mahmood

arXiv:2304.06819v2Apr 13, 2023cs.CVcs.AIq-bio.GNq-bio.QMq-bio.TO

TL;DR

Multimodal survival prediction must reconcile spatially detailed histology with globally summarized transcriptomics, while providing meaningful transcriptomic tokens and scalable cross-modal interactions. SURVPATH learns biological pathway and histology patch tokens, fuses them with a memory-efficient multimodal Transformer, and achieves state-of-the-art survival performance across five TCGA datasets compared with unimodal and multimodal baselines.

Problem
Combining histology and bulk transcriptomics for survival prediction requires meaningful transcriptomics tokenization and scalable modeling of dense interactions between spatial and global molecular information.
Method
SURVPATH learns pathway tokens from transcriptomics and combines them with histology patch tokens using a memory-efficient multimodal Transformer.
Results
SURVPATH achieves state-of-the-art survival performance on five TCGA datasets and outperforms unimodal and multimodal baselines.
Takeaways & Limitations
The framework identifies known and candidate prognostic features and supports multimodal interpretation of interactions between biological pathways and histological patterns.

Abstract

from arXiv · show

Integrating whole-slide images (WSIs) and bulk transcriptomics for predicting patient survival can improve our understanding of patient prognosis. However, this multimodal task is particularly challenging due to the different nature of these data: WSIs represent a very high-dimensional spatial description of a tumor, while bulk transcriptomics represent a global description of gene expression levels within that tumor. In this context, our work aims to address two key challenges: (1) how can we tokenize transcriptomics in a semantically meaningful and interpretable way?, and (2) how can we capture dense multimodal interactions between these two modalities? Specifically, we propose to learn biological pathway tokens from transcriptomics that can encode specific cellular functions. Together with histology patch tokens that encode the different morphological patterns in the WSI, we argue that they form appropriate reasoning units for downstream interpretability analyses. We propose fusing both modalities using a memory-efficient multimodal Transformer that can model interactions between pathway and histology patch tokens. Our proposed model, SURVPATH, achieves state-of-the-art performance when evaluated against both unimodal and multimodal baselines on five datasets from The Cancer Genome Atlas. Our interpretability framework identifies key multimodal prognostic factors, and, as such, can provide valuable insights into the interaction between genotype and phenotype, enabling a deeper understanding of the underlying biological mechanisms at play. We make our code public at: https://github.com/ajv012/SurvPath.

1. Introduction

SURVPATH addresses multimodal survival prediction by learning interpretable biological pathway tokens and modeling dense interactions with histology patch tokens. It targets the complementary global transcriptomic and spatial histologic information while reducing the computational burden of multimodal fusion.

Whole-slide images provide spatial histologic information, whereas bulk transcriptomics captures global gene-expression information and complementary prognostic signals.
Transcriptomics tokenization is challenging because gene measurements are naturally represented as feature vectors, leading prior methods to concatenate entire features or use coarse functional sets.
Dense early fusion becomes difficult when Transformers model interactions between approximately 15,000 histology patches and 331 pathway tokens because attention has quadratic complexity.
SURVPATH learns biological pathway tokens and combines them with histology patch tokens through a memory-efficient multimodal Transformer for survival prediction.
The paper evaluates SURVPATH on five TCGA datasets against unimodal and multimodal baselines and introduces a multi-level interpretability framework for unimodal and cross-modal insights.

2. Related Work

Prior work has explored MIL for histology survival prediction and several Transformer-based fusion strategies, but multimodal cancer prognosis has predominantly used late fusion. SURVPATH builds on early cross-modal fusion while addressing limitations in interaction directionality and transcriptomics tokenization.

Histology-based survival models commonly use MIL to represent tumor heterogeneity and the tumor microenvironment from whole-slide image patches.
Multimodal Transformers concatenate tokens or use hierarchical and cross-attention designs, while low-complexity reformulations address growing sequence length and dimensionality.
Cancer prognosis studies commonly combine histology and omics data through late fusion methods such as concatenation, modality-level alignment, and bilinear pooling.
Early fusion models cross-modal interactions between individual inputs, unlike late fusion methods that combine modality-level representations.
SURVPATH extends MCAT by modeling bidirectional interactions, using biologically grounded pathway tokens, and reducing redundancy from overlapping gene sets.

3. Method

SURVPATH builds interpretable pathway and histology patch tokens, fuses them with memory-efficient multimodal attention, predicts survival, and supports unimodal and cross-modal interpretation.

Pathway tokenizer: Pathways provide biologically meaningful transcriptomics tokens by grouping genes into cellular processes and encoding each pathway's measurements with a sparse multilayer perceptron.The gene-to-pathway connectivity controls sparsity, and each resulting token represents a deep pathway-level transcriptomics representation.
Histology tokenizer: WSIs are converted into low-dimensional patch tokens by identifying tissue regions, extracting non-overlapping patches, and applying a pretrained feature extractor followed by a dimension-matching transform.The pipeline uses 20× patches and pre-extracts embeddings to reduce training-time memory requirements.
Multimodal fusion: SURVPATH concatenates pathway and patch tokens and replaces expensive full self-attention with sparse pathway-to-pathway, pathway-to-patch, and patch-to-pathway interactions.Patch-to-patch interactions are omitted because patch tokens greatly outnumber pathway tokens; shared projections reduce parameter requirements.
Survival prediction: The fused multimodal embedding supports survival prediction by converting time-to-event modeling into classification over non-overlapping survival-time intervals and deriving patient-level risk.The classifier estimates interval death probabilities, whose cumulative products represent survival through each interval.
Multi-level interpretability: A multi-level interpretability framework uses Integrated Gradients for pathway, gene, and histology-patch influence and Transformer attention for pathway–patch interactions.Attention-based heatmaps connect cellular functions represented by pathways with corresponding morphological features.

4. Experiments

SURVPATH was evaluated across five TCGA datasets against unimodal and multimodal baselines, with experiments examining performance, fusion, tokenization, and interpretability. It achieved the best overall performance and showed that dense early fusion and semantically granular pathway tokens were useful design choices.

Dataset and baselines: SURVPATH was evaluated on five TCGA datasets using disease-specific survival and c-index comparisons against histology, transcriptomics, and multimodal baselines.The datasets were BLCA, BRCA, STAD, COADREAD, and HNSC.
Survival prediction results: SURVPATH achieved the best overall performance at both 20× and 10× magnification, exceeding TransMIL by +7.3%, MLP by +3.0%, and MCAT by +3.5% at 20×.These comparisons were reported for c-index-based survival prediction.
Survival prediction results: Multimodal baselines outperformed histology baselines, while a transcriptomics MLP surpassed several multimodal methods, highlighting feature-selection and heterogeneous-data integration challenges.The authors also note that the relatively small dataset size complicates learning complex models and increases over-fitting risk.
Survival prediction results: Early fusion methods outperformed late fusion methods, supporting a joint feature space for fine-grained interactions between transcriptomics and histology tokens.The authors connect this result to modeling dense pathway–patch interactions within a unified Transformer attention mechanism.
Ablation study: Increasing transcriptomics-token granularity improved overall performance, which the authors associate with tokens encoding more specific biological functions.The tokenizer ablation compared Reactome, Hallmarks, a single token, and gene-family alternatives.
Ablation study: Removing either pathway-to-patch or patch-to-pathway interactions reduced c-index by −5.6% and −7.5%, while Nyström attention reduced performance by −6.9%.These ablations support retaining both cross-modal interaction directions and the full attention formulation used by SURVPATH.
Interpretability: The interpretability analysis linked high-importance pathways, including EMT and COX Reactions, with morphologies and risk patterns in breast cancer cases.The analysis also associated estrogen-response pathways with lower-grade or in situ morphologies in the low-risk case.
Interpretability: Unimodal and cross-modal interpretability can identify candidate multimodal prognostic biomarkers connecting pathways with histological morphologies.The authors suggest these findings may inform research on combinations of morphologies and pathways, while the evidence remains interpretive.

5. Conclusion

The paper addresses transcriptomics tokenization and efficient multimodal fusion by combining biological pathway tokens with sparse modality-specific Transformer attention. SURVPATH achieves state-of-the-art survival performance across five TCGA datasets, while its qualitative interpretability framework identifies known and candidate prognostic features.

Conclusion: SURVPATH defines biological pathway tokens and uses sparse modality-specific attention to integrate long transcriptomics and histology sequences.These components address semantic transcriptomics tokenization and computationally efficient multimodal fusion.
Conclusion: SURVPATH achieves state-of-the-art survival performance on five TCGA datasets and reveals known and candidate prognostic features.The conclusion presents these as the paper’s principal performance and interpretability outcomes.
Conclusion: The interpretability findings remain qualitative, motivating future dataset-level metrics such as quantitative morphological characterization of specific pathways.The conclusion also cautions that no performance improvement from patch-to-patch interactions does not establish that such interactions are unnecessary.

1. Survival prediction

SURVPATH predicts patient survival from a multimodal embedding by discretizing survival time into intervals and modeling censored outcomes with survival and hazard functions. A patient-level risk score is then used to stratify patients into risk groups.

Survival prediction: SURVPATH predicts survival from a multimodal embedding x̄Att ∈ R2d by assigning patients to discrete time intervals with censorship status.Observed deaths and last known follow-ups are represented using c, while quartile-based intervals approximate time-to-event values.
Survival prediction: The classifier outputs one logit per time interval, which defines the discrete hazard as the sigmoid probability of death during that interval.The hazard function is fhazard(yj|x̄Att) = S(ŷj).
Survival prediction: The discrete survival function represents the probability that a patient survives up to each time interval.Together with the hazard function, it supports likelihood-based training for censored survival data.
Survival prediction: The censorship-aware negative log-likelihood enforces appropriate survival probabilities for censored patients and correct death timing for observed deaths.Separate terms handle survival beyond follow-up, survival until observed death, and prediction of the observed death interval.
Survival prediction: The negative sum of all logits defines a patient-level risk used to identify and stratify different risk groups.

2. Implementation

The implementation combines pretrained patch embeddings with a transcriptomics encoder and evaluates survival models using concordance and Kaplan–Meier analyses. Training uses fixed optimization settings and modality-specific embedding dimensions.

Implementation: SURVPATH and its baselines use RAdam with batch size 1, learning rate 5 × 10^-4, and weight decay 10^-3.Models were implemented in PyTorch, with interpretability based on Captum.
Implementation: The patch encoder produces 768-dimensional CTransPath embeddings that are projected to token dimension d = 256.
Implementation: Performance is evaluated with c-index, which measures correct survival ordering, and Kaplan–Meier curves, which visualize survival probabilities across risk groups.The c-index ranges from 0.5 for random prediction to 1.0 for perfect prediction.

3. Additional interpretability

SURVPATH’s interpretability framework attributes predicted risk across slides, genes, and pathways while examining cross-modal pathway–patch correspondences. A bladder-cancer analysis links important histology and pathways to low- and high-risk cases.

Additional interpretability: In a bladder-cancer analysis, healthy bladder muscle was associated with reduced risk, while pleomorphic tumor cells with foamy cytoplasm contributed to increased risk.
Additional interpretability: Important pathways in the bladder-cancer cases involved cell-cycle control, metabolism, and immune-related functions.Examples included the G2M checkpoint, fatty acid metabolism, allograft rejection, and IL2 STAT5 signaling.
Additional interpretability: The framework attributes predicted risk at slide, gene, and biological-pathway levels from WSI and transcriptomic inputs.It also examines pathway-to-patch and patch-to-pathway interactions.
Additional interpretability: Cross-modal analysis linked the allograft rejection pathway to tumor-infiltrating lymphocytes and lymphocyte collections near the bladder’s muscular wall in the low-risk case.

4. Additional results

Additional analyses show that SURVPATH maintains strong performance across magnifications, separates risk groups across five diseases, and attributes most validation-fold contribution to histology. Its visualizations also encode risk direction and feature importance.

Additional results: 62.9% performance was identical for SURVPATH at 10× and 20× across the five cohorts.At 10×, SURVPATH remained the best overall model, multimodal models generally outperformed unimodal models, and transcriptomic baselines remained strong competitors.
Additional results: SURVPATH produced statistically better discrimination between high- and low-risk groups than the best histology, transcriptomics, and multimodal baselines across all five diseases.Groups were defined using the cohort-median predicted risk.
Additional results: Histology accounted for 77.2% of modality attribution across cohorts, and SURVPATH outperformed survival prediction from the evaluated clinical covariates.Attributions were computed by summing Integrated Gradients over modality-specific tokens before co-attention.
Additional results: Bladder-cancer visualizations use red and blue to indicate increased and decreased risk, while heatmap colors encode feature importance from high to low.The displayed pathways and morphologies generally corresponded to previously described urothelial-carcinoma patterns, including the G2M checkpoint.