Source-linked AI summary
Multimodal Attention-based Deep Learning for Alzheimer's Disease Diagnosis
Michal Golovanevsky, Carsten Eickhoff, Ritambhara Singh
TL;DR
Accurate AD and MCI diagnosis is difficult because disease pathology can precede symptoms and clinical information alone is limited. MADDi combines imaging, genetic, and structured clinical data using self- and cross-modal attention, achieving 96.88% accuracy for CN, MCI, and AD classification. The study supports multimodal attention-based decision support while identifying limitations in diagnostic alignment and MRI coverage.
Problem
Clinical symptoms alone diagnose AD pathology and progression accurately in less than 50% of patients, while existing multimodal methods generally do not model cross-modal interactions.
Method
MADDi uses modality-specific neural networks, self-attention within modalities, and bidirectional cross-modal attention across imaging, genetic, and structured clinical data.
Results
96.88% average test accuracy was achieved for CN, MCI, and AD classification across five random model initializations.
Takeaways & Limitations
Combining self- and cross-modal attention and retaining structured clinical information supported highly accurate multimodal AD diagnostic decision support.
Takeaways & Limitations
MRI processing used central slices rather than full brain volumes, which may omit brain regions relevant in other applications.
Abstract
from arXiv · showhide
Alzheimer's Disease (AD) is the most common neurodegenerative disorder with one of the most complex pathogeneses, making effective and clinically actionable decision support difficult. The objective of this study was to develop a novel multimodal deep learning framework to aid medical professionals in AD diagnosis. We present a Multimodal Alzheimer's Disease Diagnosis framework (MADDi) to accurately detect the presence of AD and mild cognitive impairment (MCI) from imaging, genetic, and clinical data. MADDi is novel in that we use cross-modal attention, which captures interactions between modalities - a method not previously explored in this domain. We perform multi-class classification, a challenging task considering the strong similarities between MCI and AD. We compare with previous state-of-the-art models, evaluate the importance of attention, and examine the contribution of each modality to the model's performance. MADDi classifies MCI, AD, and controls with 96.88% accuracy on a held-out test set. When examining the contribution of different attention schemes, we found that the combination of cross-modal attention with self-attention performed the best, and no attention layers in the model performed the worst, with a 7.9% difference in F1-Scores. Our experiments underlined the importance of structured clinical data to help machine learning models contextualize and interpret the remaining modalities. Extensive ablation studies showed that any multimodal mixture of input features without access to structured clinical information suffered marked performance losses. This study demonstrates the merit of combining multiple input modalities via cross-modal attention to deliver highly accurate AD diagnostic decision support.
1 Introduction
AD diagnosis requires integrating imaging, genetic, and clinical information, yet existing multimodal approaches largely concatenate features without modeling cross-modal interactions. MADDi addresses this gap with self- and cross-modal attention for three-class classification.
- Motivation: AD is difficult to detect early because pathology precedes symptoms, MCI may progress to AD over years or decades, and clinical symptoms alone yield less than 50% accurate diagnosis.Early MCI detection is therefore relevant to preventive care because AD progression cannot currently be cured, only delayed.
- Research gap: Existing multimodal studies combine imaging, genetic, and clinical information but generally concatenate modality features, limiting shared-representation learning.MADDi is proposed to model interactions between MRI, SNP, and structured clinical data directly.
- Findings: MADDi achieved 96.88% average test accuracy across five initializations for the multimodal three-class task, while clinical information complemented the other modalities most effectively.Images were strongest in unimodal evaluation at 92.28% accuracy, but the combined model benefited most from the clinical modality.
- Data: Clinical inputs included 29 quantitative, categorical, or binary features from neurological exams, cognitive assessments, and demographics, while imaging used three central brain slices and genetic data underwent SNP filtering.The imaging input comprised standardized MRI slices, and clinical features that directly indicated AD were removed.
- Data: The multimodal dataset merges imaging, SNP, and clinical records by matching clinical timestamps to the closest MRI date and retaining the most recent evaluation per patient.The resulting overlap dataset included 239 patients with all three modalities available.
- Approach: MADDi uses modality-specific fully connected or convolutional backbones, self-attention within modalities, bidirectional cross-modal attention between modality pairs, and a final dense classifier.The framework predicts CN, MCI, or AD from pre-processed clinical, genetic, and imaging inputs.
3 Results
MADDi combined imaging, clinical, and genetic data for multimodal three-class diagnosis, achieving strong held-out performance. Results also showed that attention—especially cross-modal attention—and structured clinical information were important to performance.
- Unimodal performance: 92.28% average accuracy was achieved by the imaging unimodal model, outperforming the clinical model’s 80.5% average accuracy.The unimodal evaluations used five random initializations; imaging performed best overall among the modalities shown.
- Multimodal performance: 96.88% average accuracy and 91.41% average F1-Score were achieved on the held-out multimodal three-class classification test set across five random initializations.The comparison covered MADDi and five previous studies addressing a similar multimodal three-class task.
- Attention ablation: The combination of self-attention and cross-modal attention produced the narrowest F1-score distribution and highest median across 100 random seeds.The F1-scores were calculated on a held-out test set; cross-modal attention alone had the second-highest median with a wider distribution.
- Attention ablation: 7.9% average F1-Score separated the final attention model from the no-attention model, with the difference significant at p-value < 0.0001.The comparison used a two-sample Z-test and provided evidence that attention benefited multimodal data integration.
- Class-level performance: MADDi identified Alzheimer’s Disease correctly across initializations, while some MCI patients were misclassified as controls.The authors hypothesized that the absence of different MCI stages may have contributed to confusion involving patients with mild symptoms.
- Modality importance: Combining imaging, clinical, and genetic modalities performed best across all evaluation metrics, while excluding clinical data reduced performance.Modality comparisons were conducted on the same overlap patient set using held-out test data; memory, executive function, and language scores were identified as dominant clinical features.
4 DISCUSSION
The framework integrates three modalities for AD diagnosis and can be extended to additional data sources and clinical tasks. The discussion also identifies labeling and imaging-preprocessing constraints that bound the current results.
- Clinical implications: MADDi integrates imaging, genetic, and clinical data and can be expanded as more data sources become available.The proposed integration scheme is intended for multimodal clinical applications.
- Future extensions: The framework could incorporate PET scans, clinical notes, biomarkers, and more than three modalities because it does not rely on modality-specific processing.The authors also plan to distinguish stable and progressive MCI in future work.
- Limitations: Genetic-model baselines used 805 of 808 patients because genetic diagnoses were unavailable and MRI and clinical diagnoses had to agree.This restriction was used to reduce ground-truth labeling error when diagnoses differed across time or modalities.
- Limitations: Using only the MRI middle slice may omit brain regions, although performance differed by less than 1% from full-volume processing on this task.The simplification reduced computational cost, while full-volume integration could improve performance in other applications.
5 Conclusions
The paper concludes that MADDi uses attention-based deep learning to integrate multimodal inputs for AD detection. Combining self-attention and cross-modal attention produced state-of-the-art performance on the three-class task.
- Framework: MADDi uses attention-based deep learning to detect Alzheimer’s disease from multimodal inputs.The framework integrates multimodal inputs, performs multi-task classification, and uses cross-modal attention to capture interactions.
- Performance: The model’s performance was consistently high regardless of chance initialization and exceeded existing multimodal machine-learning methods.The paper presents this result as state-of-the-art performance for the task.
- Attention mechanisms: 96.88% accuracy resulted from combining self-attention with cross-modality attention on the three-class classification task.Self-attention reinforces important extracted features, while cross-modality attention reinforces relationships between modalities.
7 Funding Statement
The study reports no specific grant from any funding agency in the public, commercial, or not-for-profit sectors.
- The research received no specific grant from any funding agency in the public, commercial, or not-for-profit sectors.
S1 Genetic Data Pre-processing
The genetic preprocessing pipeline starts with whole-genome sequencing data, filters variants using quality and frequency criteria, and then restricts analysis to Alzheimer’s-related genes. Clinical feature importance was assessed separately with a Random Forest classifier.
- Genetic data: Whole-genome sequencing data from 805 ADNI participants were generated at roughly 30–40× coverage and provided as variant call files.The sequencing was performed using an Illumina non-CLIA laboratory, with files generated using ADNI’s BWA and GATK-based workflow.
- Variant filtering: Variants were filtered by Hardy-Weinberg equilibrium, genotype quality, minor allele frequency, and missingness before downstream analysis.The stated thresholds were HWE p < 0.05, GQ < 20, MAF < 0.01, and missing rate > 0.05.
- Gene selection: After filtering, the pipeline selected genes known to be related to Alzheimer’s disease using the AlzGene Database and UCSC genome browser.The AlzGene Database list contained 680 genes.
- Clinical features: A Random Forest classifier was used to estimate clinical-feature importance and check whether variables could provide an unfair advantage.The procedure was part of examining the contribution of clinical features to model performance.
- Clinical features: Clinical features were ranked from most important to least important in the clinical feature-importance graph.
S3 Imaging Data Pre-processing
The imaging pipeline uses ADNI-corrected MRI data and evaluates how the number and location of brain slices affect unimodal model performance. Slices farther from the brain center add noise rather than meaningful information, while adding more images does not significantly improve performance.
- Pre-processing: ADNI preprocessing applies Gradwarp, B1 non-uniformity correction, and N3 intensity correction to the MRI images.Gradwarp corrects gradient nonlinearity, B1 correction addresses intensity non-uniformity, and N3 reduces residual intensity variation.
- Image selection: The study compares unimodal imaging performance using different numbers of brain slices, from the middle three slices to 50 additional images per angle.Both F1-score and accuracy are reported for these image-count settings.
- Image selection: The unimodal imaging model does not significantly benefit from adding more images.Validation F1-score and accuracy trends remain largely unchanged as the number of images increases.
- Image selection: Slices farther from the center lack meaningful information and add noise, contributing to declining performance.The figure illustrates the middle three slices, outer slices, and outermost slices as distance from the center increases.
S4 Hyperparameter Tuning Methods
Hyperparameters are selected through held-out testing and three-fold cross-validation, with separate tuning for each model type. The evaluation also uses macro-averaged F1-score alongside accuracy and related classification metrics.
- Tuning procedure: The data are split into 90% training and 10% testing, with the test set withheld during hyperparameter tuning.Architecture, batch size, epochs, and learning rate are selected using three-fold cross-validation.
- Tuning procedure: Each model type receives its own best hyperparameters using the same tuning grid for fair attention comparisons.The multimodal framework incorporates the best unimodal neural-network values into its architecture.
- Evaluation metrics: Macro-averaged F1-score is computed as the arithmetic mean of the per-class F1-scores.For each class, true positives, true negatives, false positives, and false negatives are calculated separately.
S6 Performance of Unimodal Models
Among the unimodal models, imaging performs best overall, while genetics performs worst. The section reports the modality-specific evaluation results and summarizes the corresponding result tables and attention-baseline distribution.
- Modality comparison: The imaging model gives the best unimodal performance overall, whereas the genetic modality gives the lowest performance.The comparison includes imaging, clinical, and genetic models across four evaluation metrics.
- Evaluation results: Table S3 reports accuracy, precision, recall, and F1-score for the best neural-network model for each modality.The table provides the numeric results underlying the unimodal-model comparison.
- Attention baselines: Table S4 presents F1-score distributions for attention-based and attention-free baselines across 100 random seeds.The combination of self-attention and cross-modal attention has the best and narrowest distribution.
S8 Investigating Individual Class Performance
The robustness and modality analyses use repeated random initializations and held-out evaluation to assess model behavior. Cross-validation results resemble test-set results, while the modality-overlap analysis supplies numeric performance and contribution results.
- Class performance: Confusion matrices summarize the best multimodal model across five random initializations, with one matrix for each random seed.These matrices supplement the corresponding model-robustness results.
- Robustness: Cross-validation results averaged over five random initializations are similar to the held-out test-set results.This similarity supports treating the test set as a fair sample of the data.
- Modality importance: The modality-importance analysis reports each modality’s contribution and performance on an overlap patient set.Metrics are averaged over five random initializations on a held-out test set, with numeric results provided in Table S6.