PaperlayerView exact revision on arXiv

Source-linked AI summary

Multi-modal Graph Neural Network for Early Diagnosis of Alzheimer's Disease from sMRI and PET Scans

Yanteng Zhanga, Xiaohai He, Yi Hao Chan, Qizhi Teng, Jagath C. Rajapakse

arXiv:2307.16366v1eess.IVcs.CV

TL;DR

The paper addresses limited multimodal integration and the challenge of constructing GNN inputs from sMRI and PET. It builds subject brain-network features and combines imaging with phenotypic information in a multimodal population-GNN framework. The authors report improved performance over single-modality methods on AD and MCI-conversion classification.

  • Problem

    CNN-based multimodal methods have limited integration of imaging and phenotypic information, and constructing GNNs directly from sMRI and PET ROI features is unclear.

    Source: §1 · p. 2

  • Method

    The method constructs subject brain-network features from sMRI and PET, combines them with phenotypic information in modality-specific GNN branches, and fuses branch decisions.

    Source: §1 · p. 2, §3 · p. 3, §3.1.1 · p. 3

  • Results

    IFDCGCN achieved 91.07 accuracy for AD vs. NC and 75.50 accuracy for sMCI vs. pMCI, with multimodal methods superior to single-modality approaches.

    Source: §4.2 · p. 9

  • Takeaways & Limitations

    The study provides a technical approach for applying population-based GNNs to sMRI and PET while incorporating phenotypic information.

    Source: §1 · p. 2

Abstract

from arXiv · show

In recent years, deep learning models have been applied to neuroimaging data for early diagnosis of Alzheimer's disease (AD). Structural magnetic resonance imaging (sMRI) and positron emission tomography (PET) images provide structural and functional information about the brain, respectively. Combining these features leads to improved performance than using a single modality alone in building predictive models for AD diagnosis. However, current multi-modal approaches in deep learning, based on sMRI and PET, are mostly limited to convolutional neural networks, which do not facilitate integration of both image and phenotypic information of subjects. We propose to use graph neural networks (GNN) that are designed to deal with problems in non-Euclidean domains. In this study, we demonstrate how brain networks can be created from sMRI or PET images and be used in a population graph framework that can combine phenotypic information with imaging features of these brain networks. Then, we present a multi-modal GNN framework where each modality has its own branch of GNN and a technique is proposed to combine the multi-modal data at both the level of node vectors and adjacency matrices. Finally, we perform late fusion to combine the preliminary decisions made in each branch and produce a final prediction. As multi-modality data becomes available, multi-source and multi-modal is the trend of AD diagnosis. We conducted explorative experiments based on multi-modal imaging data combined with non-imaging phenotypic information for AD diagnosis and analyzed the impact of phenotypic information on diagnostic performance. Results from experiments demonstrated that our proposed multi-modal approach improves performance for AD diagnosis, and this study also provides technical reference and support the need for multivariate multi-modal diagnosis methods.

1. Introduction

The introduction motivates early AD diagnosis and the complementary structural and functional information in sMRI and PET. It proposes population-based GNNs to combine imaging and phenotypic information, addressing challenges in applying graph analysis to these modalities.

1. Introduction

  • sMRI and PET provide complementary structural and functional information, while single-modality studies cannot capture both aspects of the brain.The introduction connects early diagnosis with the possibility of early intervention to delay disease progression.Source: §1 · pp. 1–2
  • CNN-based approaches have limited subject-correlation modeling and multimodal extensibility, while GNNs can combine modalities through node features and graph adjacency.In a population graph, subjects can be nodes and edges can encode similarity between them.Source: §1 · p. 2
  • Prior GNN analyses commonly used fMRI or DTI, but constructing graphs directly from sMRI and PET ROI features is unclear.The introduction notes that sMRI and PET are 3D, unlike 4D fMRI and DTI data.Source: §1 · p. 2
  • The study constructs subject brain-network features from sMRI and PET, combines them with phenotypic information in a population graph, and fuses modalities at node and adjacency levels.The authors report improved AD detection and sMCI-versus-pMCI prediction, and identify MMSE as influential in AD detection.Source: §1 · p. 2

2. Dataset and Materials

The study uses paired baseline sMRI and FDG-PET data from ADNI-1 and ADNI-2, alongside clinical and demographic information. The dataset includes AD, normal-control, and MCI subjects, with MCI divided by subsequent conversion status.

2. Dataset and Materials

  • Paired MPRAGE sMRI and FDG-PET scans came from ADNI-1 and ADNI-2 baseline visits, using images from the same subject and closest acquisition date.The PET data comprised six 5-minute frames acquired 30–60 minutes post injection.Source: §2 · p. 2
  • The dataset contains 792 subjects: 215 AD, 246 NC, and 331 MCI, including 120 pMCI and 211 sMCI.MCI converters developed AD within three years; nonconverters did not convert during that period.Source: §2 · p. 3
  • The dataset includes MMSE scores and APOE4 gene types, with Table 1 reporting demographic statistics for each subject category.MMSE scores mainly range from 10 to 30; APOE4 types are tagged 0, 1, or 2.Source: §2 · p. 3, Table 1 · p. 3

3. Methods

The proposed method uses a separate GCN branch for each imaging modality within a population-graph framework. It combines modality-specific subject features with graph relationships and fuses branch decisions for the final prediction.

3. Methods

  • Each modality has its own GCN branch, and each graph node represents one subject’s single-modality features.The preprocessing pipeline aligns raw scans to MNI152 space before extracting ROI features with the AAL atlas.Source: §3 · p. 3, Figure 1 · p. 3
  • The method builds an individual brain network from each subject’s imaging features and combines these with phenotypic information to define population-graph adjacency.The resulting edges represent similarity between subjects’ features.Source: §3 · p. 3
  • Each branch produces softmax decision scores, which are combined to make the final prediction.Source: §3 · p. 3

3.1. Individual Feature extraction

The feature-extraction method derives subject-specific features from brain networks rather than using ROI vectors directly. For PET, it builds networks by comparison with normal subjects; for sMRI, it applies the corresponding approach to selected tissue features.

3.1. Individual Feature extraction

  • Brain-network-derived features are used instead of raw ROI inputs because small differences between sMRI and PET ROI features can yield suboptimal inputs.Source: §3.1 · p. 3

3.1.1. Individual features based on PET

  • PET subject networks are constructed indirectly by comparing each subject’s ROI SUVR vector with a group of normal subjects.The PET ROI features are metabolic SUVR values; the comparison uses normal-subject means and standard deviations.Source: §3.1.1 · pp. 3–4
  • PET ROI effect-size differences are transformed into correlations and then weights, which are applied to the normal-control connectivity matrix to form the subject’s brain network.The matrix is flattened using its upper-triangle values as individual features.Source: §3.1.1 · p. 4

3.1.2. Individual features based on sMRI

  • For sMRI, separate individual brain networks can be constructed from GM, WM, or combined GM+WM ROI features.The authors explore which feature type provides more effective inputs for the multimodal GNN.Source: §3.1.2 · p. 4

3.2. Graph construction

The graph represents subjects as nodes and their imaging-feature similarities as weighted edges. The construction incorporates phenotypic information and combines modality-specific adjacency matrices and node features.

3.2. Graph construction

  • Each subject is a node represented by upper-half brain-network features, with subject similarities encoded in a normalized adjacency matrix.The graph is undirected, and its edge weights determine which nodes contribute to convolution.Source: §3.2 · p. 4

3.2.1. Edge connections and weights

  • Gender, gene, and MMSE information modify edge weights; matching or similar phenotypes double the weight, while other cases receive zero.This phenotype-weighted construction is described for a single modality; the paper addresses multimodal construction separately.Source: §3.2.1 · p. 5

3.2.2. Integration mechanism for adjacency matrices

  • The sMRI- and PET-based adjacency matrices are combined by Hadamard product to form an integrated matrix.The paper motivates this integration by the complementarity of structural and functional information.Source: §3.2.2 · p. 5

3.2.3. Fusion mechanism for node vectors

  • For node-vector fusion, the two modalities’ feature vectors are concatenated and their correlation distance is used to calculate a shared adjacency matrix.The fused feature vectors are formed for pairs of subjects.Source: §3.2.3 · p. 5

3.2.4. Integrated fusion mechanism

  • The integrated fusion adjacency matrix is the Hadamard product of the image-based and fused-feature-based adjacency matrices.This combines the matrices produced by the two preceding fusion mechanisms.Source: §3.2.4 · p. 5

3.3. Chebyshev GCN

The method applies spectral graph convolution to signals using a graph filter. It approximates the filter with a truncated Chebyshev expansion, with polynomial order affecting the filtering effect.

3.3. Chebyshev GCN

  • Spectral graph convolution applies a filter to the graph signal using the graph’s spectral representation.The passage expresses the operation through the eigenvector matrix and graph Laplacian eigenvalues.Source: §3.3 · p. 5
  • A truncated Chebyshev-polynomial expansion approximates the graph filter, with coefficients and polynomial terms defining the approximation.The passage identifies U as the eigenvector matrix and describes the Laplacian’s construction from the adjacency and degree matrices.Source: §3.3 · p. 5
  • The filtering effect changes with polynomial order K, and the reported best performance occurs when K is 3 or 4.The architecture illustration describes a two-layer graph network followed by late fusion.Source: §3.3 · p. 6, Figure 4 · p. 6

3.4. Multi-modal network architecture

The multi-modal network uses a separate graph-convolution branch for each modality and combines branch output probabilities for a final prediction. It is trained on labeled graph nodes and predicts labels for unlabeled test nodes.

3.4. Multi-modal network architecture

  • Each modality has a two-layer Chebyshev GCN branch with ReLU-activated hidden layers, 32 units, dropout, and a softmax output.The softmax output provides class probabilities for the branch.Source: §3.4 · p. 6
  • The final prediction is formed by decision fusion of the softmax output probabilities from the two branches.The branch outputs correspond to the two modalities.Source: §3.4 · p. 6
  • The trained population-based GNN predicts labels for unlabeled test-set nodes after training on a labeled subset of graph nodes.The architecture caption describes each branch score as a diagnostic result for the corresponding subject.Source: §3.4 · p. 6

4. Experiments and Results

The experiments evaluated brain-network features from sMRI and PET, multimodal GNN fusion, and phenotypic information across AD-vs-NC and sMCI-vs-pMCI diagnosis. IFDCGCN achieved the strongest reported multimodal results, while performance varied with modality, fusion design, and task.

4.1. Experimental design

  • The experiments used 70%/15%/15% train, validation, and test splits across four non-overlapping test sub-datasets, with five-fold cross-validation on two sub-datasets.Tasks were AD vs. NC and sMCI vs. pMCI; performance was assessed with accuracy, sensitivity, specificity, and AUC.Source: §4.1 · pp. 6–7

4.2. Experimental results and discussion

  • Brain-network features outperformed direct ROI features; GM features gave the best AD-vs-NC average accuracy at 87.71%, while brain-matter features led sMCI-vs-pMCI at 72.40%.For PET, brain-network features also performed better than ROI features.Source: §4.2 · pp. 7–8, Table 2 · p. 7
  • IFDCGCN achieved 91.07 accuracy, 90.22 sensitivity, 91.87 specificity, and 91.04 AUC for AD vs. NC; for sMCI vs. pMCI, it achieved 75.50 accuracy.The latter task's sensitivity, specificity, and AUC were 49.90, 88.70, and 69.30, respectively, and were improved over single-modality methods.Source: Table 3 · p. 8, Table 4 · p. 8, §4.2 · p. 9
  • Multimodal methods outperformed single-modality approaches, and IFDCGCN had the best reported classification performance and stability; late fusion improved prediction accuracy.The shared-adjacency construction also affected GNN performance, and the proposed multimodal mechanisms outperformed simple late fusion in DGCN.Source: §4.2 · p. 9, Figure 8 · p. 9
  • Phenotypic adjacency information from gender, gene, and MMSE benefited diagnosis; MMSE was especially beneficial for AD vs. NC, while weighting all three gave the best sMCI-vs-pMCI results.On a fixed two-sub-test set, MMSE-combining IFDCGCN achieved 98.00% and 96.29% average accuracy, with standard deviations of 0.78 for each.Source: §4.2 · p. 10, Table 5 · p. 10, Figure 9 · p. 10
  • The MMSE-integrated adjacency showed strong within-group correlation for AD vs. NC, with 96.68% average accuracy; multiple phenotypic information showed relatively stronger within-group correlation for sMCI vs. pMCI.The authors note that MCI conversion prediction needs further exploration and point to the need for more perceptive individual features.Source: §4.2 · pp. 10–11, Figure 11 · p. 11
  • Against ADNI state-of-the-art methods, the proposed approach had satisfactory performance, an advantage or comparable diagnostic specificity, and outperformed some multimodal CNN methods.Source: §4.2 · p. 11
  • The study constructs individual features from sMRI and PET ROI-based brain networks, then combines multimodal features, adjacency matrices, and late decision fusion; PET showed greater AD-vs-NC matrix differences than GM.Selected ROIs included memory-related regions, as reported in the cited clinical studies.Source: §4.2 · pp. 8, 11, Figure 6 · p. 8

5. Conclusion

The population-based multimodal GNN combines brain-network imaging features, phenotypic information, shared adjacency matrices, and decision fusion. On ADNI, it improved prediction results, especially for AD detection, and showed better or equivalent performance to several state-of-the-art methods, including for sMCI versus pMCI.

5. Conclusion

  • The approach constructs brain networks to obtain individual imaging features, then combines imaging and phenotypic data to represent associations between individuals and the population.It also uses shared adjacency matrices and a decision-making mechanism.Source: §5 · pp. 11–12
  • The method showed better or equivalent diagnostic performance to several state-of-the-art methods, including on sMCI versus pMCI; ADNI tables cover AD versus NC and sMCI versus pMCI.The reported improvements were especially notable in AD detection.Source: Table 6 · p. 12, Table 7 · p. 12, §5 · p. 12
  • The study was exploratory on ADNI, and the authors state that additional datasets may be needed to confirm the findings.Source: §5 · p. 12
Loading 2307.16366v1…