PaperlayerView exact revision on arXiv

Source-linked AI summary

Multi-modal Graph Learning for Disease Prediction

Shuai Zheng, Zhenfeng Zhu, Zhizhe Liu, Zhenyu Guo, Yang Liu, Yuchen Yang, Yao Zhao

arXiv:2203.05880v1cs.LGcs.AIcs.CV

TL;DR

Existing disease-prediction graph methods often rely on manually defined graphs and insufficiently model complex multimodal correlations. MMGL jointly learns modality-aware representations and an adaptive graph in an end-to-end framework, achieving favorable performance on two disease prediction tasks and supporting inductive learning.

  • Problem

    Existing graph-based disease-prediction methods manually define graphs and may overlook complex inter-modal correlations, limiting the information available for reliable diagnosis.

  • Method

    MMGL combines modality-aware representation learning of shared and modality-specific information with adaptive graph learning, jointly optimized end to end for disease prediction.

  • Results

    MMGL achieves comparable or significant improvement over state-of-the-art approaches on two disease datasets, including approximately 0.9%–4.0% improvement over LGL on ABIDE.

  • Takeaways & Limitations

    The learned modality dependencies can provide modal-explainable decision support, while the framework remains applicable to inductive testing on unseen data.

Abstract

from arXiv · show

Benefiting from the powerful expressive capability of graphs, graph-based approaches have been popularly applied to handle multi-modal medical data and achieved impressive performance in various biomedical applications. For disease prediction tasks, most existing graph-based methods tend to define the graph manually based on specified modality (e.g., demographic information), and then integrated other modalities to obtain the patient representation by Graph Representation Learning (GRL). However, constructing an appropriate graph in advance is not a simple matter for these methods. Meanwhile, the complex correlation between modalities is ignored. These factors inevitably yield the inadequacy of providing sufficient information about the patient's condition for a reliable diagnosis. To this end, we propose an end-to-end Multi-modal Graph Learning framework (MMGL) for disease prediction with multi-modality. To effectively exploit the rich information across multi-modality associated with the disease, modality-aware representation learning is proposed to aggregate the features of each modality by leveraging the correlation and complementarity between the modalities. Furthermore, instead of defining the graph manually, the latent graph structure is captured through an effective way of adaptive graph learning. It could be jointly optimized with the prediction model, thus revealing the intrinsic connections among samples. Our model is also applicable to the scenario of inductive learning for those unseen data. An extensive group of experiments on two disease prediction tasks demonstrates that the proposed MMGL achieves more favorable performance. The code of MMGL is available at \url{https://github.com/SsGood/MMGL}.

I. INTRODUCTION

Existing multimodal graph methods face difficulties modeling inter-modal complementarity, learning suitable patient graphs, and generalizing to unseen samples. MMGL addresses these issues through modality-aware representation learning and adaptive, end-to-end graph learning, with reported improvements on two disease datasets.

  • Multimodal medical data can provide complementary information about patients, making its effective utilization important for reliable clinical disease diagnosis.
  • Existing methods underuse modality-specific complementary information because shared representation learning primarily captures information common across modalities.
  • Hand-designed graph construction requires tuned similarity measures and separates representation learning, graph construction, and prediction into independent stages.
  • Spectral graph methods and multi-graph approaches are difficult to apply inductively because unseen samples require graph relationships that are cumbersome to determine.
  • MMGL combines modality-aware representation learning with adaptive graph learning in an end-to-end model applicable to inductive disease prediction.
  • MMGL reports comparable or significant improvement over state-of-the-art approaches on two disease datasets, while modality dependencies provide modal-explainable decision support.

II. RELATED WORK

Prior work fuses multimodal features or constructs graphs using predefined relationships, while this paper formulates disease prediction as multimodal graph learning with learned patient connections. Its framework represents patients as graph nodes and supports graph-based prediction from multimodal features.

  • Earlier disease-prediction methods fused multimodal features through concatenation, multi-kernel learning, or deep neural networks.
  • Graph-based approaches construct patient relationships from handcrafted kernels, modality-specific graphs, or selected non-imaging features before applying graph convolution.
  • Figure 1 contrasts direct concatenation and intra-modal attention fusion with modality-aware learning, which performs more cross-modal interactions through multi-modal attention.
  • The problem formulation treats patients as graph nodes, multimodal features as node attributes, and pairwise patient connections as edges with adjacency weights.
  • The stated task is to learn inter-modal and inter-patient correlations while providing an appropriate graph for GNN-based disease prediction.

2) Overview of the Framework:

MMGL combines modality-aware representation learning, adaptive graph learning, and GNN-based prediction in an end-to-end framework. It captures both shared and patient-sensitive modality information before learning graph structure and predicting disease outcomes.

  • MMGL consists of modality-aware representation learning, adaptive graph learning, and GNN-based prediction.The learned representations and adjacency matrix jointly support prediction, including inductive prediction for unseen patients.
  • MARL combines modality-shared information with modality-specified information to represent both commonality and patient-sensitive complementarity.The final patient representation concatenates the shared and specified embeddings.
  • The resulting shared and complementary representations are jointly used by GNN prediction, while an auxiliary branch promotes modality-specific learning.The authors state that the multi-modal attention can also be extended to a multi-head version.
  • The framework transforms heterogeneous modality features into a shared homogeneous space before applying attention-based representation learning.Each modality is projected to the same dimension, enabling cross-modal query, key, and value operations.
  • Inter-modal attention models dependence between modalities and supports shared cross-modal aggregation.The attention matrix measures how much one modality attends to another, while self-preservation retains original modal information with α = 1.
  • The variance of each patient’s inter-modal attention is used to encode modality-specific, patient-sensitive information.This complements the shared representation, whose aggregation may smooth out inter-modal diversity.

C. Adaptive Graph Structure Learning

MMGL learns the patient graph from modality-aware representations rather than relying on manually designed similarity measures. The learned graph is regularized for useful structure and can incorporate unseen patients during testing.

  • Graph learning is needed because obtaining an appropriate graph for biomedical GNN tasks is nontrivial.Existing similarity choices require manual tuning to construct meaningful graphs for downstream prediction.
  • MMGL learns a similarity metric jointly with downstream GNNs instead of manually specifying the graph.The approach treats graph learning as learnable node-similarity metric learning, supporting inductive use.
  • The learned adjacency uses a weighted cosine similarity parameterized by a learnable matrix W_A and is symmetric.Symmetry is motivated by the limited presence of uni-directional effects between patients outside epidemics.
  • Threshold θ = 0.5 sparsifies the graph by removing low-weight links and rescaling the remaining nonzero edges.The procedure converts the fully connected similarity matrix into a non-negative sparse graph.
  • Graph regularization constrains sparsity, connectivity, and smoothness so connected patients have similar representations without collapsing the graph to zero.A logarithmic barrier controls connectivity, while a Frobenius term limits excessive sparsity.

D. Model Optimization and Implementation

The learned sparse graph and modality-aware representations are supplied to a GNN for patient prediction, with an auxiliary classifier used to stabilize training and encourage modality-specific learning.

  • The GNN predicts patient outcomes from the modality-aware representations H and learned sparse graph A.An auxiliary classifier provides additional predictions to stabilize learning and promote modality-specified representation learning.

1) Model optimization:

MMGL jointly optimizes its graph-learning and prediction components through a combined loss and modular iterative training strategy. The optimization uses task-aware losses to obtain patient representations, a population graph, and prediction accuracy.

  • Model optimization: The GNN prediction module uses a vanilla GCN by default, while an auxiliary classifier f(·) is implemented as a single-layer MLP.The auxiliary classifier directly guides the learning of Hsp through the prediction output.
  • Model optimization: MMGL uses a joint loss to optimize all three modules simultaneously, rather than optimizing graph structure only through task-aware prediction loss.The loss balances three terms with hyper-parameters λ and η; classification losses are cross-entropy.
  • Model optimization: Each training epoch first jointly trains MARL and AGL, then jointly trains AGL and the prediction module.This modular iterative strategy is intended to obtain meaningful patient representations and a population graph with high prediction accuracy.

2) Implementation:

The implementation supports inductive learning through mini-batch neighbor aggregation and adds unseen patients to the learned population graph during testing. The model is implemented with Adam in PyTorch and uses hyperopt for tuning.

  • Inductive learning: Inductive training uses mini-batches and sampled neighbors instead of aggregating over the entire adjacency matrix.This approach is applied to the GNN module to avoid the limitation of whole-graph aggregation.
  • Optimization and datasets: The implementation section identifies baseline demographic information tables for the TADPOLE and ABIDE datasets.Table I covers TADPOLE, while Table II covers ABIDE.
  • Inductive learning: During inductive testing, each unseen patient is added to the existing population graph through MARL and AGL before neighbor aggregation.The neighboring nodes of the unseen patient are then sampled for processing.
  • Optimization and datasets: MMGL is implemented in PyTorch with Adam optimization; λ and η are set to 1, while other hyper-parameters are tuned using hyperopt.The fixed values for λ and η are selected by experience.

IV. EXPERIMENTAL RESULTS AND ANALYSIS

MMGL is evaluated on TADPOLE and ABIDE for multimodal disease-prediction tasks using established preprocessing, task definitions, and comparisons with state-of-the-art baselines. The evaluation includes Alzheimer’s and autism-related classification settings.

  • Experimental setup: Experiments evaluate MMGL on the TADPOLE and ABIDE biomedical datasets against state-of-the-art disease-prediction methods.The section first describes the experimental protocol and then presents comparison results.
  • Datasets: TADPOLE provides multimodal MRI, PET, cognitive, CSF, risk-factor, and demographic features for 603 selected subjects.The subjects include 211 NC, 320 MCI, and 72 AD participants; MCI is further divided into 275 sMCI and 45 pMCI.
  • Preprocessing: Preprocessing retains one sample per subject, removes subjects exceeding a 5% missing-feature rate, and mean-imputes remaining missing values.Longitudinal features are removed and feature selection is performed for each modality.
  • Datasets: ABIDE contributes 871 subjects for autism prediction, divided into 468 NC and 403 ASD participants across four modalities.The dataset contains resting-state fMRI and phenotypic data collected from 24 sites.
  • Tasks and metrics: The evaluated tasks are NC versus sMCI versus AD and sMCI versus pMCI on TADPOLE, plus NC versus ASD on ABIDE.The sMCI-versus-pMCI task is included for Alzheimer’s disease early diagnosis.

1) Baselines:

MMGL is compared with single-graph, multi-graph, and graph-learning baselines across two benchmark datasets. Reported results favor MMGL in transductive learning, multimodal comparison, and learned-representation quality.

  • Baselines: The baselines include PopGCN, InceptionGCN, Multi-GCN, LSTMGCN, EV-GCN, and LGL, covering handcrafted and learned graph constructions.PopGCN and InceptionGCN are single-graph methods, Multi-GCN is multi-graph based, and LGL and EV-GCN are closely related state-of-the-art methods.
  • Baselines: The comparison reports quantitative results over the TADPOLE and ABIDE benchmark datasets.Table III is labeled as quantitative comparisons over two benchmark datasets, with results reported in percentages.
  • Quantitative Results: MMGL significantly outperforms Multi-GCN across three tasks on both TADPOLE and ABIDE datasets.The authors attribute this result to MARL’s ability to capture inter-correlation among modalities.
  • Quantitative Results: MMGL improves over LGL by approximately 0.9%–4.0% on the ABIDE dataset.The authors state that this indicates AGL is more effective than LGL’s graph-learning approach.
  • Transductive Learning: With training data increased from 10% to 80% of patients, MMGL performs best among the reported transductive-learning baselines.Compared with PopGCN, MMGL performs better with a small amount of training data.
  • Representation Analysis: Similarity matrices show class-specific blocks for both LGL and MMGL, with MMGL producing more distinctive learned representations.The block structure reflects differences between classes in both datasets.

3) Qualitative Results:

MMGL learns patient representations with clearer class structure and a population graph containing fewer incorrect links than manually constructed alternatives. Its modality-aware representation also separates classes while preserving disease-stage ordering in TADPOLE.

  • Representation quality: MMGL’s learned representations form clearer class-specific blocks than LGL, indicating more distinctive patient representations.The reported visualization shows smaller within-class scatter and larger inter-class scatter for MMGL representations.
  • Learned graph quality: MMGL’s adaptive graph contains only a few incorrect patient links, whereas PopGCN’s manually designed graph has a relatively high incorrect-link proportion.Incorrect links are defined as links connecting patients from different classes.
  • Learned graph quality: The learned graph supports adaptive end-to-end graph construction rather than relying solely on demographic information or predefined similarity metrics.The comparison attributes MMGL’s graph quality to adaptive graph learning and contrasts it with manually designed graph structures.
  • Representation quality: MMGL’s modality-aware representation clusters patients by class and preserves the ordering of different disease stages in TADPOLE.The clustering behavior is reported for both TADPOLE and ABIDE datasets.

E. Ablation study

The ablation study evaluates modality-aware representation learning, adaptive graph learning, and modality contribution analysis. Results favor combining MARL with AGL, while contribution scores provide modality-specific interpretability, especially for ASD subjects in ABIDE.

  • Module ablations: Combining MARL and AGL achieves substantially better performance than the other tested module combinations.The ablation compares MARL and AGL with MLP, direct concatenation, PopGCN-based construction, and kNN graphs.
  • Module ablations: Hand-constructed PopGCN graphs perform worst, especially on ABIDE, while inappropriate graph metrics can reduce MARL’s performance.Both MARL+PopGCN and MARL+kNN worsen MARL, whereas AGL remains favorable without MARL.
  • Modality interpretability: For randomly selected ASD subjects, the fourth modality, fMRI connectivity networks, receives substantially higher contribution scores than the other modalities.The authors relate this modality-specific pattern to clinical judgment about ASD diagnosis.
  • Modality interpretability: Average modality contribution distributions are broadly consistent across the individual case study and the NC and ASD class-level analyses.The authors interpret these modality-specific representations as providing explanations for predictions.
  • Framework scope: MMGL jointly learns modality-shared and modality-specified representations with an adaptive graph optimized end-to-end for disease prediction.The framework is also described as supporting inductive testing on unseen data.
Loading 2203.05880v1…