PaperlayerView exact revision on arXiv

Source-linked AI summary

Multimodal Data Integration for Oncology in the Era of Deep Neural Networks: A Review

Asim Waqas, Aakash Tripathi, Ravi P. Ramachandran, Paul Stewart, Ghulam Rasool

arXiv:2303.06471v3cs.LG

TL;DR

Cancer data span modalities and scales, but the review identifies the lack of a comprehensive framework encompassing this diversity. It surveys multimodal oncology learning with GNNs and Transformers, and concludes that robust, deployment-ready frameworks remain a key need.

  • Problem

    Despite promising outcomes from multimodal oncology research, a comprehensive framework encompassing the full spectrum of cancer dataset modalities is lacking.

    Source: §6 · p. 20

  • Method

    The review surveys GNN-based fusion approaches, cross-learning methods, and multimodal Transformers alongside challenges in oncology data integration.

    Source: §2.4 · p. 13, §2.5 · p. 14, §3.2 · p. 16

  • Results

    The review finds multimodal oncology research promising but reports that a comprehensive framework spanning the full range of cancer data modalities is not yet available.

    Source: §6 · p. 20

  • Takeaways & Limitations

    Advancing the field requires robust, deployment-ready multimodal learning frameworks that scale across cancer modalities and incorporate uncertainty quantification, interpretability, and generalizability.

    Source: §6 · p. 21

    Explainability and trustworthiness for multimodal GNNs and Transformers remain open challenges.

    Source: §4.6 · p. 19

Abstract

from arXiv · show

Cancer has relational information residing at varying scales, modalities, and resolutions of the acquired data, such as radiology, pathology, genomics, proteomics, and clinical records. Integrating diverse data types can improve the accuracy and reliability of cancer diagnosis and treatment. There can be disease-related information that is too subtle for humans or existing technological tools to discern visually. Traditional methods typically focus on partial or unimodal information about biological systems at individual scales and fail to encapsulate the complete spectrum of the heterogeneous nature of data. Deep neural networks have facilitated the development of sophisticated multimodal data fusion approaches that can extract and integrate relevant information from multiple sources. Recent deep learning frameworks such as Graph Neural Networks (GNNs) and Transformers have shown remarkable success in multimodal learning. This review article provides an in-depth analysis of the state-of-the-art in GNNs and Transformers for multimodal data fusion in oncology settings, highlighting notable research studies and their findings. We also discuss the foundations of multimodal learning, inherent challenges, and opportunities for integrative learning in oncology. By examining the current state and potential future developments of multimodal data integration in oncology, we aim to demonstrate the promising role that multimodal neural networks can play in cancer prevention, early detection, and treatment through informed oncology practices in personalized settings.

1 Fundamentals of Multimodal Learning (MML)

Oncology data spans clinical, molecular, and imaging modalities that provide complementary views of disease across scales. The section outlines a five-stage multimodal-learning pipeline, fusion strategies, and examples of oncology applications.

1.1 Data Modalities in Oncology

  • Clinical records, molecular measurements, radiology, and digitized histopathology provide complementary information about patients and cancer across population-to-cellular scales.Molecular data describe genetic changes and biological processes; radiology captures tumor location and extent, while whole-slide images reveal microstructural changes.Source: §1.1 · p. 3, §1.1.1 · p. 5, §1.1.2 · p. 5

1.2 Taxonomy of MML

  • The five-stage taxonomy comprises preprocessing, feature extraction, data fusion, a primary learner, and a final classifier.Preprocessing includes cleaning, normalization, class balancing, and augmentation; feature extraction may be engineered or learned automatically.Source: §1.2 · p. 5, §1.2.1 · p. 5, §1.2.2 · pp. 5–6

1.3 Data Fusion Strategies

  • Data fusion combines raw features, extracted features, or prediction vectors, with the chosen technique depending on data characteristics and the task.Early fusion joins features before training, intermediate fusion combines modality-model outputs, and late fusion combines modality-specific decisions.Source: §1.2.3 · p. 6, §1.3.1 · p. 6, §1.3.2 · p. 6

1.2.4 Primary Learner

  • Primary learners train on preprocessed data or extracted features, while final classifiers produce labels or class scores from learner outputs or fused data.DNNs can learn high-level representations; CNNs are commonly used for images and video, and RNNs or Transformers for text and sequences. Stacking and boosting are classifier options.Source: §1.2.4 · p. 6, §1.2.5 · p. 6

1.4 MML for Oncology Datasets

  • Oncology applications span late-fusion cancer classification and single-cell integration using alignment, latent-space matching, and autoencoder-based methods.Examples include radiology and text for prostate and lung cancer, genomic and pathology data, mammography, skin lesions, and brain tumors. Single-cell methods include SOMatic, SCIM, GLUE, and autoencoder-based approaches.Source: §1.4 · pp. 7–8

2 Graph Neural Networks (GNNs) in Multimodal Learning

Graphs encode relationships among interacting entities, and GNNs process graph-structured data. In multimodal oncology, modalities can be nodes and their relationships represented as edges.

2 Graph Neural Networks (GNNs) in Multimodal Learning

  • GNNs are designed for graph-structured data, making them suitable for multimodal oncology representations.Each modality or sub-modality can be a node, with relationships and patterns between modalities modeled as edges.Source: §2 · p. 8
  • Graphs represent relational connectivity among interacting entities and are used across fields including brain networks, driving maps, and DNN structures.The passage gives examples of graphs used to study brain networks, driving maps, and DNNs.Source: §2 · p. 8
  • Graph edges can encode relationships among oncology modalities, providing a representation for analyzing their interactions.The paper describes modalities or sub-modalities as nodes and their structures or patterns as edges.Source: §2 · p. 8

2.1 The Graph Data

A graph consists of nodes and edges, with graph types distinguished by direction, node homogeneity, temporal change, and edge attributes. Graph tasks may operate on nodes, edges, or whole graphs.

2.1 The Graph Data

  • A graph G=(V, E) comprises a node set with feature vectors and an edge set; a node’s neighborhood contains nodes connected to it.The passage defines V as the node set, E as the edge set, and N(v) as the neighbors connected to v.Source: §2.1 · p. 8

2.1.1 Graph Types

  • Graphs may be directed or undirected, homogeneous or heterogeneous, static or dynamic, and attributed or unattributed.These distinctions describe edge direction, node types, change over time, and whether edge weights encode relational strength.Source: §2.1.1 · p. 8

2.1.2 Tasks for Graph Data

  • Graph analyses include node-level tasks, edge-level classification or prediction, and graph-level prediction or generation.Examples include node classification and clustering, predicting edge presence or absence, and graph classification.Source: §2.1.2 · p. 8

2.2 ML for Graph Data

Graph learning methods range from shallow embedding and probabilistic approaches to DNN-based GNNs. GNN families differ in how they aggregate neighborhood information and what graph-learning task they support.

2.2.1 Traditional (Shallow) Methods

  • Traditional graph methods include embeddings that preserve structure in low-dimensional vectors and probabilistic models that represent distributions over graph variables.Embedding approaches use linear, kernel, or tensor methods; probabilistic approaches include Bayesian and Markov networks, variational inference, and variable elimination.Source: §2.2.1 · p. 8

2.2.2 DNN-based Methods - GNNs

  • DNN-based graph learning uses GNNs to aggregate neighborhood information into node representations through recursive message passing.GNNs handle relational, non-Euclidean data and can learn for any permutation of input nodes; multiple aggregation layers can encode higher-order structure and smooth features [77].Source: §2.2.2 · pp. 9–10, Figure 7 · p. 10
  • Recurrent GNNs combine an RNN’s temporal modeling with a GNN’s local-structure modeling and repeat information fusion until an equilibrium or desired state.They can operate on graphs with variable sizes and topologies.Source: §2.2.2 · pp. 10–11
  • Convolutional GNNs update node embeddings by aggregating neighboring nodes’ representations across stacked layers.Spatial variants aggregate neighboring vertices; spectral variants operate using the graph Laplacian eigendecomposition. The passage describes convolution ⊆ attention ⊆ message passing.Source: §2.2.2 · p. 11
  • Graph auto-encoders map graphs into low-dimensional latent spaces and decode those representations back to graphs.The passage lists dimensionality reduction, anomaly detection, and graph generation as GAE applications.Source: §2.2.2 · p. 11
  • Graph adversarial networks generate graphs with properties similar to their inputs, while other GNN categories include scalable, dynamic, hypergraph, and heterogeneous models.A GraphAN generator maps random noise to a graph, and its discriminator distinguishes generated from input graphs.Source: §2.2.2 · p. 11

2.3 GNNs and ML using Unimodal Oncology Datasets

GNNs represent oncology data as graphs to capture relationships in pathology, radiology, and molecular datasets. The reviewed studies apply these representations to classification, segmentation, prediction, and other tasks.

2.3.1 Pathology Datasets

  • Histology graphs encode cells, tissue regions, or patches as nodes and their relations as edges; Histographs outperformed CNN-based classification.Histographs used nuclei features and Euclidean distances between nuclei as edge features.Source: §2.3.1 · pp. 11–12

2.3.2 Radiology Datasets

  • GraphSAGE-pool was reported as the best model for brain-tumor segmentation on 3D MRI represented as a supervoxel graph.Other reviewed radiology applications include lesion segmentation, mammogram classification, lymph-node delineation, and prostate-contour prediction.Source: §2.3.2 · pp. 12–13

2.3.3 Molecular Datasets

  • Molecular graphs represent amino-acid residues as nodes and their relationships as edges, supporting tasks including protein-interface prediction, drug interactions, and single-cell imputation and clustering.The review also describes graph methods for EHR knowledge augmentation and single-cell knowledge transfer.Source: §2.3.3 · p. 13

2.4 MML - Data Fusion at the Pre-Learning Stage

Pre-learning fusion combines features extracted from separate modalities before training the primary learner. The reviewed GNN approaches use this strategy to integrate radiological and clinical information for patient-level prediction.

2.4 MML - Data Fusion at the Pre-Learning Stage

  • Pre-learning fusion merges features from individual modalities before training the multimodal primary learner.For GNNs, representations may be hand-engineered or learned by deep-learning models.Source: §2.4 · p. 13
  • Cui et al. fused radiological and clinical features with category-wise attention as node representations to predict lymph-node metastasis in esophageal squamous cell carcinoma.UNet and CNN-based encoders extracted the modality features, which were processed with conventional and correlation-based GATs before classification.Source: §2.4 · p. 13
  • EV-GCN fuses fMRI-derived radiology features and patient clinical features through pairwise associations that generate population-graph edge weights for diagnosis.The partially labeled graph is processed through GCN layers.Source: §2.4 · p. 13

2.5 MML - Data Fusion using Cross-Modality Learning

Cross-modality learning links models trained on separate modalities, including sequential use of learned representations. Reviewed applications span survival prediction, single-cell integration, and other cancer-related tasks.

2.5 MML - Data Fusion using Cross-Modality Learning

  • Hierarchical Transformer-GNN learning from radiological and clinical data outperformed ResNet-Graph for survival prediction in early-stage NSCLC.CT tumor features learned by ViT were used as patient-population-graph node features for subsequent GraphSAGE processing.Source: §2.5 · p. 13
  • scMoGNN was the first GNN framework for multimodal single-cell integration using cross-learning fusion and won first place in the NeurIPS 2021 modality-prediction task.The method used paired data to generate cell-feature graphs and showed superior performance on various tasks.Source: §2.5 · p. 14
  • Other cross-learning GNNs combine cell- and tissue-level features for breast-cancer classification, impute or cluster scRNA-seq data, and predict cancer genes from multi-omics [115].Source: §2.5 · p. 14

2.6 MML - Data Fusion in Post-Learning Regime

Post-learning fusion processes individual modalities before combining them for a downstream prediction task. Reviewed GNN applications span imaging, molecular interactions, and multi-omics, with one benchmark reporting the strongest classification performance for a GAT-based GNN.

2.6 MML - Data Fusion in Post-Learning Regime

  • Post-learning fusion processes modalities individually before combining them for downstream prediction; hand-crafted features perform better at low input dimensionality, while deep features perform better otherwise.Source: §2.6 · p. 14
  • Imaging applications include PathomicFusion for survival prediction, a late-fusion cervical-cancer screening method, and dot-product fusion of CNN and GCN features for breast-cancer classification.Source: §2.6 · pp. 14–15
  • GAT-based GNN had the best classification performance in Leng et al.'s benchmark of 16 deep-learning networks across three multi-omics datasets and two tasks.Source: §2.6 · p. 15
  • Omics applications include drug-combination prediction with MOOMIN, multi-omics classification with MOGONET, and late-fusion models for breast-cancer and NSCLC diagnosis [123].Other reviewed examples use GCNs for side-effect prediction and GNN-CNN fusion for drug-target affinity.Source: §2.6 · pp. 14–15

3 Transformers in MML

Transformers use self-attention to model global relationships and can process varied data modalities; multimodal models combine modalities through several fusion strategies. The section reviews these architectures and their oncology applications.

3 Transformers in MML

  • Transformers use self-attention to capture long-range dependencies and process sequences in parallel, and multimodal variants use cross-attention to model inter-modality dependencies.The review describes Transformers as applied beyond text to images, audio, and time series; multimodal examples include foundation models and systems such as CLIP and Flamingo [134].Source: §3 · p. 15

3.1 Model Architecture

  • The original Transformer uses encoder and decoder blocks: the encoder produces hidden representations, and the decoder generates outputs using those representations and previous tokens [125].Self-attention weights input elements using query-key dot products and a scaling operation, then applies the weights to value vectors.Source: §3.1 · pp. 15–16, Figure 10 · p. 16

3.2 Multimodal Transformers

  • Self-attention treats tokens or embeddings as nodes in a fully connected graph, learning global patterns; tokenization and embedding choices can vary in granularity.The section identifies early fusion, cross-attention, hierarchical attention, and late fusion as approaches to inter-modality interactions.Source: §3.2 · p. 16, Figure 11 · p. 18

3.2.1 Early Fusion

  • Early fusion concatenates modalities into one input for the Transformer, a simple and efficient strategy that assumes modalities are equally important.The assumption may not hold in practice.Source: §3.2.1 · pp. 16–17

3.2.2 Cross-Attention Fusion

  • Cross-attention lets Transformer layers attend selectively across modalities, modeling their interactions and joint representations.The attention can occur at different stages of data processing.Source: §3.2.2 · p. 17

3.2.3 Hierarchical Fusion

  • Hierarchical fusion combines features across scales, as illustrated by approaches for salient-object detection, fine-grained classification, and video summarization.The cited examples use low-level spatial with high-level semantic features, fine-grained fusion, and global dependencies or multi-hop frame relationships, respectively.Source: §3.2.3 · p. 17

3.2.4 Late Fusion

  • Late fusion processes each modality with its own Transformer, then concatenates branch outputs for a final classifier.The described model for depression estimation processes audio and visual temporal information independently before fusing weights.Source: §3.2.4 · p. 17

3.3 Transformers for Processing Oncology Datasets

  • Transformers have been applied to oncology screening, diagnosis, prognosis, treatment selection, and clinical-variable prediction, including multimodal and pathology tasks.Examples include breast-cancer prediction from imaging and genomics, TransMIL for histopathology slides, TransConv for brain-tumor MRI segmentation, and co-attention survival prediction using whole-slide images and genomic sequences [29, 143, 158, 159].Source: §3.3 · pp. 17–18

4 MML - Challenges and Opportunities

Multimodal oncology learning faces challenges involving data quantity and quality, alignment, generalization, missingness, and model behavior. The review identifies these as open problems and opportunities for improving integrative learning.

4.1 Large Amounts of High-quality Data

  • Oncology datasets are limited relative to large deep-learning training corpora, while expert annotation is laborious and variability can reduce reproducibility and consistency.The review contrasts about 1.1 million cancer-keyword GEO samples with 3 billion images in JFT-300M.Source: §4.1 · p. 19

4.2 Data Registration and Alignment

  • Incorrect registration across imaging, molecular, and clinical modalities can make accurate information fusion difficult [166].Alignment may require a common reference frame and shared landmarks or fiducial markers.Source: §4.2 · p. 19

4.3 Pan-Cancer Generalization and Transference; 4.4 Missing Data Samples and Modalities

  • Models face difficulty generalizing across cancer sites or from one modality to another, while missing modalities also affect learning and constrain already small oncology datasets [166].Fine-tuning, multimodal co-learning, and model induction are described as transfer techniques; handling missing modalities remains an open challenge.Source: §4.3 · p. 19, §4.4 · p. 19

4.5 Imbalanced Data

  • Class imbalance, common in oncology, can make deep-learning models struggle to classify underrepresented classes accurately.The review lists augmentation, ensembles, continual learning, and transfer learning as countermeasures.Source: §4.5 · p. 19

4.6 Explainability and Trustworthiness

  • Explainability and trustworthiness for multimodal GNNs and Transformers remain open challenges despite existing work on explaining individual model decisions [169] [175].The review names GNNExplainer, PGM-Explainer, and SubgraphX among GNN methods.Source: §4.6 · p. 19

4.7 Over-smoothing in GNNs

  • GNN over-smoothing makes node representations nearly similar, losing information and harming performance and generalization; scalable deep architectures remain an open challenge.Dropout, weight decay, skip connections, and higher-order graph structures have been proposed as regularization techniques.Source: §4.7 · p. 20

4.8 Modality Collapse; 4.9 Dynamic and Temporal Data

  • Multimodal models may collapse onto one modality and neglect others, while learning from multimodal dynamic data remains challenging.The review also notes that spatio-temporal methods can capture changing data relationships over time.Source: §4.8 · p. 20, §4.9 · p. 20

4.10 Data Privacy and Federated Learning; 4.11 Other Challenges

  • Privacy motivates local processing and federated learning across sites, while multimodal learning also demands substantial computation, robustness, failure detection, and better-explored uncertainty quantification.The review states that addressing these challenges could allow multimodal models to surpass single-modality performance.Source: §4.11 · p. 20, §4.10 · p. 20

5 Multimodal Oncology Data Sources

The review compiles a non-exhaustive, periodically updated collection of oncology data sources to provide researchers with a unified starting point.

5 Multimodal Oncology Data Sources

  • The authors compiled a non-exhaustive, periodically updated collection of oncology dataset sources from NIH and other portals for ML researchers.The collection is presented as a unified source of data.Source: §5 · p. 20

6 Conclusion

The conclusion identifies the absence of a comprehensive framework covering cancer-data modalities as a challenge. It presents multimodal integration as a route toward richer cancer insights and calls for scalable, deployment-ready frameworks with stronger uncertainty quantification, interpretability, and generalizability.

6 Conclusion

  • The lack of a comprehensive framework spanning the full spectrum of cancer dataset modalities remains a challenge despite promising integration research.The authors state that combining methodologies and data across scales could unlock deeper cancer insights.Source: §6 · p. 20
  • The review identifies robust, deployment-ready multimodal frameworks that scale across cancer-data modalities as a priority.It also calls for uncertainty quantification, interpretability, and generalizability in such frameworks.Source: §6 · p. 21
  • Multimodal integration could yield more accurate prognostic and predictive models than single modalities alone.The conclusion frames improved understanding and patient outcomes through more informed, personalized treatment strategies as potential benefits.Source: §6 · pp. 20–21
Loading 2303.06471v3…