PaperlayerView exact revision on arXiv

Source-linked AI summary

Pathomic Fusion: An Integrated Framework for Fusing Histopathology and Genomic Features for Cancer Diagnosis and Prognosis

Richard J. Chen, Ming Y. Lu, Jingwen Wang, Drew F. K. Williamson, Scott J. Rodig, Neal I. Lindeman, Faisal Mahmood

arXiv:1912.08937v3cs.CVq-bio.GNq-bio.TO

TL;DR

Cancer outcome prediction often separates histology from genomics despite their complementary information. Pathomic Fusion combines gated unimodal representations through Kronecker-product interactions and improves multimodal prognostic and grading performance over unimodal and established comparison approaches.

  • Problem

    Most deep learning outcome-prediction and grading paradigms use histology or genomics alone rather than integrating their complementary information.

  • Method

    Pathomic Fusion combines histology image, cell-graph, and genomic features using gating-based attention, Kronecker-product interactions, and multimodal attribution methods.

  • Results

    Pathomic Fusion outperforms unimodal networks, Cox models, prior fusion methods, and the WHO paradigm on image-omic survival prediction; glioma performance reaches a c-Index of 0.826.

  • Takeaways & Limitations

    The framework provides an interpretable multimodal strategy for integrating imaging and multi-omic data in cancer diagnosis, prognosis, and related treatment-response prediction tasks.

Abstract

from arXiv · show

Cancer diagnosis, prognosis, and therapeutic response predictions are based on morphological information from histology slides and molecular profiles from genomic data. However, most deep learning-based objective outcome prediction and grading paradigms are based on histology or genomics alone and do not make use of the complementary information in an intuitive manner. In this work, we propose Pathomic Fusion, an interpretable strategy for end-to-end multimodal fusion of histology image and genomic (mutations, CNV, RNA-Seq) features for survival outcome prediction. Our approach models pairwise feature interactions across modalities by taking the Kronecker product of unimodal feature representations and controls the expressiveness of each representation via a gating-based attention mechanism. Following supervised learning, we are able to interpret and saliently localize features across each modality, and understand how feature importance shifts when conditioning on multimodal input. We validate our approach using glioma and clear cell renal cell carcinoma datasets from the Cancer Genome Atlas (TCGA), which contains paired whole-slide image, genotype, and transcriptome data with ground truth survival and histologic grade labels. In a 15-fold cross-validation, our results demonstrate that the proposed multimodal fusion paradigm improves prognostic determinations from ground truth grading and molecular subtyping, as well as unimodal deep networks trained on histology and genomic data alone. The proposed method establishes insight and theory on how to train deep networks on multimodal biomedical data in an intuitive manner, which will be useful for other problems in medicine that seek to combine heterogeneous data streams for understanding diseases and predicting response and resistance to treatment.

I. INTRODUCTION

The paper argues that cancer outcome prediction should integrate complementary histologic and genomic information, then introduces Pathomic Fusion for multimodal learning and interpretation.

  • Histology provides spatial and morphological information, while genomic assays provide quantitative molecular information; combining them could support more integrated cancer analysis.
  • Pathomic Fusion models cross-modal feature interactions with a Kronecker product of gated histology and genomic representations.Gating-based attention controls each modality’s expressiveness before fusion.
  • The framework can extract histology features using CNNs, graph convolutional networks, or their combination, alongside genomic features.Its graph-based component is presented as an approach for learning cell-level histopathology features.
  • The study evaluates multimodal fusion in a rigorous 15-fold cross-validation analysis across two disease models.The analysis compares the approach with subjective grading and subtyping determinations and prior deep learning patient-stratification results.

II. RELATED WORK

Related work covers survival modeling, multimodal fusion, and graph-based histology analysis before positioning Pathomic Fusion as a joint image, cell-graph, and genomic representation.

  • Survival Analysis for Cancer Outcome Prediction: Cancer survival analysis supports biomarker discovery, patient stratification, and therapeutic response prediction using covariates such as CNV, mutation status, and RNA-Seq expression.
  • Multimodal Fusion of Histology and Genomics: Prior multimodal learning methods combine heterogeneous data sources, while earlier feature-fusion approaches include Hadamard- and Kronecker-product-based interactions.
  • Graph Convolutional Networks: Before this work, graph convolutional networks had been applied to histology classification but not to deep-learning-based cancer survival prediction.
  • Pathomic Fusion: Pathomic Fusion targets paired histology and genomic data by learning a multimodal representation from histology image, cell-graph, and genomic features.The stated objective is to outperform unimodal representations in supervised learning.
  • Graph Convolutional Networks: The cell-graph pathway represents tissue structure through segmented cells and their spatial relationships, using graph convolutional networks to learn morphometric features.Cells are detected and localized, connected with KNN, and represented using handcrafted and deep features.
  • Pathomic Fusion: The multimodal network is trained with Cox partial likelihood for survival prediction and cross entropy for grade classification.

B. Learning Morphometric Cell and Graph Features using Graph Convolutional Networks

The method represents tumor tissue as cell graphs to capture spatial organization and cellular interactions, then learns graph-level features for survival prediction. Graph construction combines segmented nuclei, nearest-neighbor edges, handcrafted and deep cell features, and graph convolution.

  • Graph motivation: Cell graphs capture histopathology organization from local cell-to-cell interactions to larger invasion and metastasis regions.Their hierarchical topology provides multiple views of tumor microenvironment structure.
  • Graph construction: The pipeline segments and localizes cells, connects adjacent cells with K-Nearest Neighbors, computes cell features, and applies graph convolution for survival prediction.Graphs use nuclei as nodes, spatial connections as edges, and feature matrix X for node attributes.
  • Graph construction: The adjacency graph is restricted to K nearest neighbors to model interactions among adjacent cells.The investigation used K = 5.
  • Cell features: Each cell receives contour and texture descriptors derived from segmentation contours and 64 × 64 image crops.Contour descriptors include shape measurements, while GLCM features characterize texture.
  • Graph convolution: GraphSAGE updates node representations by aggregating neighborhood features and combining them with the current node representation.The method adopts AGGREGATE and COMBINE definitions for neighborhood aggregation.

ReLU

The cell-graph model uses hierarchical attention pooling to retain informative node embeddings across graph scales. This supports representation of both local cellular interactions and coarser tissue organization.

  • Hierarchical pooling: SAGPOOL adaptively learns each node embedding’s contribution during local pooling operations.The attention score Z is assigned to nodes in the graph.
  • Hierarchical pooling: Hierarchical pooling encodes cell-graph structure from fine-grained interactions to coarser regions of invasion and metastasis.This follows from the graph’s hierarchical topology.

SAGEConv

The paper combines graph-based histology representations with genomic modeling for cancer outcome prediction. Its genomic feedforward network uses self-normalizing components to address high-dimensional, low-sample-size training conditions.

  • SAGEConv: The graph branch aggregates information across multiple scales and pools node features into a graph-level representation.The resulting graph feature vector is used in the multimodal architecture.
  • Genomic features: RNA-Seq, copy number variation, and mutation status provide gene-level molecular features used to study cancer survival outcomes.The passage identifies these as common high-throughput molecular characterizations.
  • Genomic modeling: Feedforward networks are prone to overfitting when learning hundreds to thousands of features from relatively few samples.This motivates stronger regularization for genomic modeling.
  • Outcome prediction: Pathomic Fusion produces more concentrated glioma hazard clusters than Histology CNN and better stratifies selected intermediate-to-high-risk groups.The figure comparison reports stronger mid-to-high-risk stratification than Histology CNN and low-to-mid-risk stratification than molecular subtyping.
  • Genomic modeling: Self-normalizing layers, ELU activations, and Alpha Dropout are used to regularize the genomic feedforward network.The final fully connected layer learns a 32-dimensional genomic representation for Pathomic Fusion.
  • Outcome prediction: In clear cell renal cell carcinoma, Pathomic Fusion better separates shorter- and longer-surviving patients than Histology CNN, with a bimodal hazard distribution.The model also disentangles survival curves for G1–G3 tumors.

D. Multimodal Tensor Fusion via Kronecker Product and Gating-Based Attention

Pathomic Fusion addresses the structural mismatch among histology images, cell graphs, and genomic vectors by learning gated multimodal representations. It then uses Kronecker-product fusion to represent unimodal, pairwise, and three-way feature interactions.

  • Motivation: Histology, cell graphs, and genomics have different structural representations, creating a data heterogeneity gap for multimodal cancer pathology.Images are 2D pixel grids, cell graphs are node-edge structures, and genomic data is a 1D covariate vector.
  • Motivation: Genomic biomarkers can change the semantic importance of morphological features when defining glioma decision boundaries.The passage gives IDH1 mutation and 1p19q codeletion as examples of conditioning biomarkers.
  • Kronecker fusion: The Kronecker product explicitly models interactions across histology image, cell graph, and genomic feature representations.The resulting tensor captures unimodal, bimodal, and trimodal interactions.
  • Kronecker fusion: Appending 1 to each unimodal representation preserves unimodal and bimodal interactions within the trimodal tensor.The fused representation forms a differential multimodal tensor in 3D Cartesian space.
  • Gating attention: Gating-based attention reduces noisy or collinear unimodal features before the Kronecker product.The gating mechanism controls each modality’s feature expressiveness and can reduce the feature-space size.
  • Gating attention: Attention weights are learned from all three modalities and applied element-wise to each modality’s feature representation.The gated representation is computed by multiplying features by their attention scores.

E. Multimodal Interpretability

Pathomic Fusion combines histology, cell-graph, and genomic inputs for survival prediction and supports multimodal attribution analysis. Its evaluation uses glioma and CCRCC data with 15-fold cross-validation and c-Index-based assessment.

  • Interpretability: Pathomic Fusion interprets multimodal survival predictions using modified Grad-CAM and Integrated Gradients to visualize image, graph, and genomic feature importance.Grad-CAM localizes influential image features, while Integrated Gradients is adapted to graph inputs and multimodal attribution.
  • Evaluation: The evaluation uses paired TCGA whole-slide images and genomic data from glioma and CCRCC, with ground-truth survival and histologic labels.Glioma and CCRCC experiments use 15-fold cross-validation and compare multimodal configurations with unimodal and clinical baselines.
  • Interpretability: The interpretability figures show subtype- and patient-specific localization of histologic regions, cell-graph structures, and genomic features used for risk prediction.Glioma examples include microvascular proliferation, glial cells, tumor cellularity, and ODG “fried egg cells”; CCRCC examples contrast lower- and higher-grade morphology.
  • Model configurations: The glioma study compares CNN, GCN, and genomic SNN unimodal networks with bimodal and trimodal Pathomic Fusion configurations.The trimodal configuration combines CNN, GCN, and SNN representations; grade classification excludes mRNA-Seq because of missing paired data and task-specific labeling.
  • Evaluation: Survival models are evaluated with the Concordance Index, while predicted hazards are converted into digital grades and compared with molecular subtyping or histologic grading.Glioma hazards are divided into 33-66-100 percentiles and CCRCC hazards into 25-50-75-100 percentiles.

A. Pathomic Fusion Outperforms Unimodal Networks and the WHO Paradigm

Pathomic Fusion improves survival prediction and grade classification over unimodal networks, Cox models, prior fusion, and the WHO paradigm. The gains also support finer risk discrimination across glioma and CCRCC cohorts.

  • Survival prediction: 0.826 c-Index is achieved by glioma Pathomic Fusion, improving 6.31% over the WHO paradigm and 5.76% over concatenation-based fusion.The trimodal CNN⊗GCN⊗SNN model has the largest c-Index, while bimodal CNN⊗SNN performs similarly on some metrics.
  • Survival prediction: Pathomic Fusion outperforms Cox models, unimodal networks, and previous deep learning-based feature-fusion approaches for image-omic survival prediction.Multimodal networks consistently improve upon their unimodal baselines across the evaluated survival tasks.
  • Grade classification: 2.75% AUC, 4.23% average precision, and 4.27% micro-F1 improvements are reported over Histology CNN for glioma grade classification.Grade IV performance also increases by 5.11% over Histology CNN.
  • Ablation analysis: Same-modality duplication causes overfitting, indicating that the reported improvements are not attributable to network ensembling.The comparison uses multimodal fusion with repeated inputs from the same modality as an ensemble test.
  • Patient stratification: Pathomic Fusion provides finer glioma risk discrimination, especially for intermediate versus high-risk patients, while low-to-intermediate separation is comparatively harder.The observed difficulty is consistent with confusion between WHO Grades II and III in molecular-subtype survival estimates.
  • Patient stratification: In CCRCC, Pathomic Fusion separates lower- and higher-survival patients and assigns digital grades that follow Fuhrman Grading System stratification.It also disentangles G1-G3 survival curves, unlike Histology CNN, whose predicted risk is nearly uniform across shorter- and longer-survival patients.

C. Multimodal Interpretability of Pathomic Fusion

Pathomic Fusion provides multimodal explanations by localizing prognostic morphology and genomic features while showing how gene importance changes when conditioned on morphology. The framework supports interpretable biomarker discovery across glioma and CCRCC.

  • Multimodal interpretability: Pathomic Fusion attributes survival predictions to pixel regions, cells in graphs, and genomic features across modalities.These explanations use class-activation maps and gradient-based attribution techniques.
  • Genomic explanations: In glioma, attributions corroborate IDH wildtype status and highlight PTEN, MYC, CDKN2A, EGFR, and FGFR2 as risk-associated features.Conditioning on morphology increases attribution for genes including ANO9 and RB1 across subtypes.
  • Genomic explanations: In CCRCC, decreased CYP3A7 and increased PITX2, DDX43, and XIST expression correlate with risk, while morphology-conditioned attribution increases for HAGHL, MMP1, and ARRP21.The authors suggest this may refine gene signatures and uncover prognostic biomarkers in cancers without multiple molecular subtypes.
  • Morphological explanations: Pathomic Fusion localizes vasculature and cell atypia across histology images and cell graphs in glioma and CCRCC.Examples include microvascular proliferation and glial cells in astrocytic tumors, fried egg cells in oligodendrogliomas, and nucleolar morphology in CCRCC.
  • Framework scope: The framework integrates CNN-, GCN-, and genomic features through Kronecker-product fusion and gating-based attention for fine-grained patient stratification and prognostic-feature interpretation.The authors describe the paradigm as scalable to combinations of imaging and multi-omic data.

APPENDIX A. DEEP LEARNING-BASED SURVIVAL ANALYSIS

The appendix introduces survival analysis through censored event times, survival and hazard functions, and the Cox proportional hazards model. Deep networks use hidden features as Cox covariates and are evaluated with the concordance index.

  • Survival-analysis foundations: Survival analysis models time to an event, with censored observations using the last known encounter as a lower bound on survival time.Patient death is uncensored, whereas survival or last follow-up is censored in cancer outcome prediction.
  • Survival-analysis foundations: The survival function gives the probability of surviving beyond a time point, while the hazard function represents instantaneous event probability.Integrating hazard over time yields the survival function.
  • Cox modeling: The Cox model parameterizes hazard as λ(t|x) = λ0(t)e^βx, leaving the baseline hazard unspecified while modeling covariate effects through β.Its partial log-likelihood enables estimation of β without directly specifying λ0(t).
  • Deep survival modeling: Deep survival networks use hidden-layer features as Cox covariates and back-propagate the derivative of the partial log-likelihood.Performance is evaluated with the Concordance Index, which measures ranking agreement for survival outcomes.
  • Data and representation: The merged TCGA-GBMLGG and TCGA-KIRC analyses use 320 and 357 genomic features, respectively, including mutations and copy-number variation.Pathomic Fusion is trained with different data subsets because genomic features are missing for some patients.

B. Data Missingness and Alignment in TCGA-GBMLGG

TCGA-GBMLGG multimodal modeling requires explicit handling of missing molecular, histological, and grade information. The pipeline combines CNN, GCN, and genomic networks, then aligns their outputs for evaluation across patient-level predictions.

  • Data missingness: Among 769 TCGA-GBMLGG patients, 72 lack molecular subtype and 33 lack histological subtype or grade labels.The dataset uses 1–3 diagnostic 20x histology ROIs and 320 genomic features per patient.
  • Multimodal architecture: The study processes histology images with a VGG19 CNN, cell spatial graphs with a GCN, and molecular profiles with a self-normalizing feedforward network.These architectures correspond to the three image, graph, and genomic modalities.
  • Multimodal architecture: Pathomic Fusion combines gating-based modality attention with Kronecker-product fusion, using task-specific modalities to gate the others.Genomics gates image and graph modalities for survival prediction, whereas histology gates genomic and graph modalities for grade classification.
  • Cell-graph processing: Cell graph construction derives node features from nuclei segmentation, contour and texture descriptors, self-supervised deep features, and graph construction procedures.The passage identifies OpenCV-based contour and texture tooling and PyFlann for graph construction.
  • Training and evaluation: For CNN survival prediction, patient-level hazards are computed by averaging predictions from nine overlapping 512 × 512 crops across each ROI.Unimodal GCN and genomic networks are trained first, after which multimodal linear layers are trained with those modules frozen.

APPENDIX C. ABLATION STUDIES AND COMPARATIVE ANALYSIS

The comparative analyses examine ablations, ensembling, grade classification, molecular-subtype stratification, and fine-grained risk grouping. Across these analyses, Pathomic Fusion is reported to improve multimodal performance and stratification beyond unimodal or same-modality combinations.

  • Ensembling effects: Pathomic Fusion improvements exceed those from fusing the same modality with the same architecture, indicating they are not explained by simple ensembling.Same-modality ensemble models overfit and produced marginal improvements compared with Pathomic Fusion.
  • Grade classification: Pathomic Fusion has greater AUCs than Histology CNN in all reported glioma grade-classification cases.Its particularly strong Grade IV performance may reflect genomic IDH mutation and 1p19q co-deletion information.
  • Molecular-subtype stratification: For TCGA-GBMLGG, Pathomic Fusion’s hazard-based stratification most closely resembles WHO grading across IDHwt astrocytic tumors, IDHmut astrocytic tumors, oligodendrogliomas, and molecular subtypes.The analysis uses 33-66-100 hazard percentiles for three risk categories.
  • Fine-grained stratification: In CCRCC, Pathomic Fusion performs best at disentangling four survival-risk categories using hazard predictions.The comparison includes Histology CNN, Histology GCN, Genomic SNN, and Pathomic Fusion, with Fuhrman Grades I–IV evaluated using percentile thresholds.
  • Interpretability comparison: Global explanations rank genomic features by mean absolute Integrated Gradients attribution and encode feature value and attribution direction across patient cohorts.The figure compares Genomic SNN and Pathomic Fusion in TCGA-GBMLGG and CCRCC.
Loading 1912.08937v3…