PaperlayerView exact revision on arXiv

Source-linked AI summary

A Multimodal Explainable Deep Learning Framework for Alzheimer's Disease Diagnosis using 3D Magnetic Resonance Imaging and Clinical Data

Yusuf Brima, Marcellin Atemkeng, Lakshmana Rao Namamula, Antoine Vacavant

arXiv:2609.12410v1cs.CVstat.ML

TL;DR

Evidence remains unclear on how multimodal AD models’ explanations behave across modalities, fusion strategies, and cohorts. This study evaluates 3D MRI and clinical data separately and jointly across diagnostic tasks and cohorts. Performance and explanations vary by task, modality, fusion, and cohort, with no fusion strategy consistently outperforming single modalities.

  • Problem

    It remains unclear whether multimodal AD models provide stable explanations across independent cohorts or identify transferable MRI information beyond cognitive assessments.

    Source: §1 · p. 3

  • Method

    The study evaluates separate and joint models of 3D structural MRI and routinely collected clinical and demographic data across diagnostic tasks in ADNI and OASIS-3.

    Source: §1.2 · p. 4

  • Results

    No fusion strategy consistently improved on unimodal models; modality performance varied by task and cohort, while cognitive measures dominated tabular explanations and image attributions varied.

    Source: §5 · pp. 19–20

  • Takeaways & Limitations

    External validation and modality-specific explanation are important for assessing whether multimodal AD models learn predictive and transferable information.

    Source: §5 · p. 20

    OASIS-3 differs from ADNI in diagnostic composition, recruitment, acquisition, and clinical-variable availability, so observed changes cannot be attributed to one source of distribution shift.

    Source: §4.3 · p. 19

Abstract

from arXiv · show

Dementia is a major and growing global health burden, with Alzheimer's disease (AD) accounting for most cases. Timely and accurate diagnosis is central to managing this burden and increasingly depends on integrating complementary clinical and imaging information. Multimodal deep learning can combine these modalities for AD diagnosis, but how its explanations behave across modalities, fusion strategies, and cohorts remains unclear. We developed an explainable multimodal framework pairing a 3D CNN encoder for T1-weighted MRI with a feedforward network for harmonized clinical and demographic data, comparing varied model setups on three-way and pairwise diagnostic tasks using 6,479 internal records from the ADNI and 1,703 independent records from the OASIS-3. On ADNI, the tabular-only model achieved the highest three-class AUC-ROC of 0.879 and best discriminated cognitively normal (CN) versus mild cognitive impairment (MCI; 0.903), while cross-attention performed best for MCI versus AD (0.861); CN versus AD was highly discriminative overall. On OASIS-3, the vision-only model performed best (three-class AUC-ROC 0.910); CN versus MCI remained difficult, and no fusion strategy consistently outperformed single modalities across tasks and cohorts. SHAP and Integrated Gradients identified the MMSE as the dominant tabular feature in both cohorts, with global feature rankings agreeing strongly in ADNI ($\rho=0.94$) and OASIS-3 ($\rho=0.96$); CAM-based explanations, however, changed with model configuration and cohort. These findings show that multimodal performance and explanations are task, modality, fusion, and cohort-dependent: a dominant cognitive signal persisted across cohorts, but feature contributions and CAM explanations did not, underscoring the need to evaluate explainability under cohort shift rather than as a stable, intrinsic property.

1 Introduction

The introduction frames multimodal AD diagnosis as a way to combine MRI with routinely collected clinical information, while emphasizing unresolved questions about modality contributions and explanation stability across cohorts. The study evaluates these questions using separate and jointly modeled modalities across diagnostic tasks and cohorts.

1 Introduction

  • Dementia is a growing global burden, and AD accounts for an estimated 60–70% of cases; MRI and routine clinical data may offer scalable diagnostic information.Differentiating CN, MCI, and AD remains challenging because MCI symptoms overlap and subject-level MRI discrimination is modest.Source: §1 · pp. 2–3
  • Prior work motivates representation-level fusion and XAI because multimodal models can be opaque, while attribution stability and clinical transferability remain open questions.Source: §1 · p. 3
  • Multimodal AD studies have not established whether dual-modality explanations remain stable across independent cohorts or whether MRI adds transferable information beyond cognitive assessments.The introduction also notes that prior work may depend on features impractical in resource-constrained settings.Source: §1 · p. 3

1.1 Research questions

  • The research asks how imaging and clinical contributions vary across AD stages, whether ADNI-trained models retain performance in an independent cohort, and whether explanations remain consistent externally.Source: §1.1 · p. 4

1.2 Contributions

  • The study evaluates 3D structural MRI and routinely collected clinical and demographic data separately and jointly across diagnostic tasks, ADNI, and external OASIS-3 validation.It assesses modality contribution, fusion effects, and cross-cohort consistency of feature-level explanations.Source: §1.2 · p. 4

2 Materials and methods

The methods pair 3D structural MRI with harmonized clinical and demographic data, using ADNI for model development and OASIS-3 for external validation. The study compares modality-specific and fused models across diagnostic tasks, with consistent evaluation and attribution procedures.

2.1 Dataset; 2.2 External Validation Cohort: OASIS3

  • ADNI provides 6,479 scans split by subject into training, validation, and test sets; OASIS-3 contributes 1,703 scans reserved exclusively for external inference.ADNI has 3,026 MCI, 2,240 CN, and 1,213 AD scans; OASIS-3 has 1,323 CN, 295 MCI, and 85 AD scans.Source: §2.1 · pp. 4–5, §2.2 · p. 6

2.2 External Validation Cohort: OASIS3

  • OASIS-3 inputs use a harmonized feature set because several ADNI-specific instruments are unavailable, with imputation, encoding, and scaling fitted on ADNI training data.The same preprocessing pipeline is applied to MRI volumes in both cohorts.Source: §2.2 · p. 6

2.4 Model Architecture

  • Late representation-level fusion encodes MRI and tabular data separately before combining their embeddings, preserving modality correspondence for branch-specific attribution.The framework uses a 3D CNN for volumetric MRI and an MLP for tabular data, with imaging CAMs and tabular SHAP attributions.Source: §2.4 · p. 7, Figure 2 · p. 8

2.4.1 Vision Encoder; 2.4.2 Tabular Encoder

  • The imaging branch maps 128 × 128 × 128 MRI volumes to 128-dimensional embeddings, while the tabular branch maps seven features to 64-dimensional embeddings.The imaging encoder uses a single-channel 3D DenseNet-121; the tabular encoder uses a 128-unit hidden layer before its final embedding projection.Source: §2.1 · p. 4, §2.4.1 · p. 7, §2.4.2 · p. 8

2.4.3 Fusion strategies

  • Concatenation passes both embeddings to a shared classifier, whereas cross-attention conditions tabular information on an imaging query before classification.The cross-attention representation uses a residual connection and layer normalization; the classifier head is shared in design across fusion strategies.Source: §2.4.3 · pp. 8–9

2.4.4 Unimodal Learning

  • Matched unimodal classifiers isolate modality contributions by retaining the multimodal branches’ representational capacity while replacing fusion with modality-specific classification heads.The vision-only model uses the same 3D DenseNet-121 backbone as the imaging branch.Source: §2.4.4 · p. 9

2.5 Learning Objective

  • Training covers the full CN/MCI/AD task and three pairwise diagnostic tasks, using class-weighted categorical or binary cross-entropy as appropriate.Class weights are recomputed for each setup using its retained diagnostic groups; the formulation is intended to compare discriminability across diagnostic transitions.Source: §2.5 · p. 10

2.6 Performance Evaluation; 2.7 Experimental Setup

  • Performance is evaluated with accuracy, micro-averaged F1, and macro-averaged AUC–ROC, with stratified-bootstrap 95% confidence intervals; model training uses fixed optimizer settings and early stopping.The evaluation protocol is applied across all four tasks and unimodal and multimodal variants; attribution methods include Grad-CAM++, ScoreCAM, and SHAP.Source: §2.6 · pp. 10–11, §2.7 · p. 11

3 Results

Diagnostic performance varied by task and cohort, and explanation behavior was examined across modalities and cohorts. MMSE remained the dominant global tabular signal, while local feature contributions and image-level attribution patterns changed with fusion and cohort.

3 Results

  • The study evaluated predictions before examining explanations, using SHAP and Integrated Gradients for tabular inputs and Grad-CAM++ and Score-CAM for MRI.This design distinguishes agreement between explanation methods from changes associated with fusion and external cohort variability.Source: §3 · p. 11

3.1 Diagnostic performance

  • CN-versus-AD was highly distinguishable (AUC–ROC 0.939–1.000); on ADNI, tabular-only led CN-versus-MCI (0.903), while cross-attention led MCI-versus-AD (0.861).Clinical variables carried the CN-versus-MCI transition, while cross-attention carried MCI-versus-AD.Source: §3.1 · pp. 11–12
  • ADNI favored tabular-only (three-class AUC–ROC 0.879), whereas OASIS-3 favored vision-only (three-class AUC–ROC 0.910); CN-versus-MCI remained difficult in OASIS-3 (AUC–ROC 0.563–0.671).Neither modality nor fusion strategy retained a uniform performance ranking across cohorts.Source: §3.1 · pp. 11–13, Table 2 · p. 12, Table 3 · p. 12

3.2 Tabular explainability across cohorts

  • MMSE was the most prominent SHAP feature across ADNI classes and remained dominant in OASIS-3; SHAP and Integrated Gradients rankings agreed strongly in both cohorts (ρ = 0.94 and ρ = 0.96).The remaining feature rankings varied across cohorts and diagnostic classes.Source: §3.2 · p. 13

3.3 Visual explainability across cohorts

  • For representative subjects, Grad-CAM++ predictions differed across cohorts and models; the examples indicate cohort-sensitive image explanations but do not quantify cohort-level spatial stability.In ADNI, the vision-only model assigned the MCI and AD subjects to CN, while concatenation and cross-attention assigned the CN subject to MCI; OASIS-3 showed a different pattern.Source: §3.3 · pp. 13–14

3.4 Multimodal Explainability

  • Fusion changed tabular-branch SHAP contributions for the same clinical inputs, and across cohorts local feature contributions and image-level attribution patterns changed with model and cohort.The dominant global cognitive signal was reproducible across cohorts.Source: §3.4 · pp. 14–15

4 Discussion

The relative value of imaging, clinical data, and fusion changed across diagnostic tasks and cohorts, while explanation behavior also varied with the predictive regime. Tabular explanations retained a dominant cognitive signal, but the study’s visual attribution analyses and cross-cohort comparisons have important limits.

4.1 Model performance and generalization

  • Modality performance reversed across cohorts: clinical variables led key ADNI tasks, while imaging became the strongest three-class predictor only on OASIS-3.No modality or fusion strategy was intrinsically superior, and external validation should examine explanation evidence as well as discrimination.Source: §4.1 · pp. 16–17

4.2 Explainability across modalities and cohorts

  • MMSE remained the dominant global tabular feature, but secondary-feature ordering changed, so the full decision rule did not transfer unchanged.SHAP and Integrated Gradients showed strong within-cohort agreement (ρ = 0.94 in ADNI and ρ = 0.96 in OASIS-3).Source: §4.2 · p. 17
  • Image predictions and attribution maps changed together across fusion configurations and cohorts, indicating that explanation behavior tracked the predictive regime.The authors frame explainability as an empirical property to evaluate under distribution shift.Source: §4.2 · pp. 17–18

4.3 Limitations and future directions

  • Visual analyses used representative cases without a cohort-level metric of spatial attribution similarity, while cohort differences prevent attributing observed changes to one shift source.OASIS-3 differs from ADNI in diagnostic composition, recruitment, acquisition characteristics, and clinical-variable availability.Source: §4.3 · p. 19
  • Diagnosis-proximal clinical variables such as MMSE may reflect proximity to the target label rather than complementary disease biology, motivating systematic ablations and quantitative image-attribution analyses.The authors also call for cross-cohort and subject-level stability assessment and additional independent cohorts.Source: §4.3 · p. 19

5 Conclusion

Across ADNI and OASIS-3, the relative value of clinical, imaging, and multimodal models depended on diagnostic task and cohort. Explanations likewise showed dominant cognitive measures in tabular data but cohort- and model-dependent image attributions.

5 Conclusion

  • No fusion strategy consistently improved on unimodal models; imaging achieved strong three-class discrimination on OASIS-3 as clinical and multimodal performance varied by task.Clinical variables were particularly discriminative for CN versus MCI on ADNI, while cross-attention performed strongest for ADNI MCI versus AD.Source: §5 · p. 19
  • Tabular behavior was dominated by cognitive measures, whereas image attributions varied with model and cohort, supporting external validation and modality-specific explanation.The authors conclude that modality value depends on diagnostic transition and data distribution.Source: §5 · p. 20

Declarations

The authors report funding support, identify the data sources and access conditions, declare no competing interests, and state that anonymized secondary data were used.

Funding

  • The research was supported in part by the National Research Foundation of South Africa.The cited reference number is CSRP23040990793.Source: Funding · p. 20

Availability of data and materials

Competing interests

Ethics approval and consent to participate

Appendix A Data Characteristics

The ADNI cohort was mostly aged 65–85 years, with broadly comparable age distributions across diagnostic groups, and was approximately balanced by sex.

Age Distribution

  • Most ADNI subjects were aged 65–85 years, with broadly comparable age distributions across CN, MCI, and AD.Source: Figure A1 · p. 21

Sex Distribution

  • ADNI was approximately balanced between female and male subjects, although MCI had a slightly higher male proportion.Source: Figure A2 · p. 21

Appendix B Tabular Variables

Table B1 lists the harmonized clinical, demographic, and neuropsychological tabular features used for ADNI training and OASIS-3 external validation.

Appendix B Tabular Variables

  • Table B1 covers harmonized clinical, demographic, and neuropsychological features used for ADNI training and OASIS-3 external validation.Source: Table B1 · p. 22

Appendix C Medical Image Pre-processing Pipeline

The pipeline standardizes T1-weighted MRI volumes through correction, skull stripping, registration, intensity normalization, and resampling. It uses the same procedure across cohorts without cohort-specific contrast calibration.

Appendix C Medical Image Pre-processing Pipeline

  • Volumes were converted to 32-bit precision, N4-corrected, skull-stripped, and affinely registered to MNI152 space.Source: Appendix C · p. 22
  • Per-volume Z-score normalization standardized intensities to approximately zero mean and unit variance.Source: Appendix C · pp. 22–23
  • The same preprocessing was used for both cohorts without cohort-specific intensity or contrast calibration, retaining residual scanner-dependent contrast differences; volumes were resampled to 128×128×128 voxels.Source: Appendix C · p. 23

Appendix D Clinical Longitudinal Data Integration and Multimodal Alignment

The integration pipeline links preprocessed MRI volumes with longitudinal clinical records, aligns each scan to a nearby clinical encounter, and removes incomplete or duplicate samples.

Appendix D Clinical Longitudinal Data Integration and Multimodal Alignment

  • MRI file metadata supplied participant identifiers that were matched to clinical registries to link imaging and longitudinal EHR data.Source: Appendix D · p. 23
  • Demographic predictors and clinical classifications were standardized, while duplicate images, missing clinical entries, and samples with missing target labels were removed.Source: Appendix D · p. 23
  • Each MRI scan was matched to its nearest clinical assessment by absolute timestamp distance, and duplicate validation ensured one complete clinical observation per volumetric tensor.Source: Appendix D · p. 23

Appendix E ADNI ScoreCAM Explainability

The appendix presents ADNI Score-CAM maps for vision-only and multimodal models on the same subjects. In the vision-only examples, MCI and AD were misclassified as CN, while the CN subject was correctly classified.

Appendix E ADNI ScoreCAM Explainability

  • The vision-only Score-CAM examples correctly classify the CN subject but misclassify the MCI and AD subjects as CN.Source: Figure E3 · p. 24
  • A multimodal Score-CAM figure presents attribution maps for the same ADNI subjects using a cross-attention-fusion model.Source: Figure E4 · p. 24
  • A second multimodal Score-CAM figure presents cross-attention-fusion attribution maps for those same ADNI subjects.Source: Figure E5 · p. 25

F.1 Tabular Explainability

The appendix presents class-specific tabular attributions and representative vision-only and multimodal explanations, including prediction outcomes for the same OASIS-3 subjects across model configurations.

F.1 Tabular Explainability

  • OASIS-3 SHAP summaries show local feature attributions and impact profiles for CN, MCI, and AD target classes.Source: Figure F6 · p. 25
  • Per-class OASIS-3 global feature importance is shown using both SHAP and Integrated Gradients.Source: Figure F7 · p. 26

F.2 Visual Explainability

  • Grad-CAM++ maps for the vision-only model show representative OASIS-3 subjects across six axial slices each.Source: Figure F8 · p. 26

F.3 Multimodal Explainability

  • On the same OASIS-3 subjects, concatenation and cross-attention misclassify CN and MCI as AD and correctly classify AD, unlike the vision-only examples.The vision-only model correctly classifies CN, misclassifies MCI as CN, and misclassifies AD as MCI.Source: Figure F8 · p. 26, Figure F9 · p. 27, Figure F11 · p. 29
  • Predicted-class SHAP attributions for the tabular branches show positive contributions in blue and negative contributions in red.The examples cover the concatenation-fusion and cross-attention models.Source: Figure F10 · p. 28, Figure F14 · p. 31
  • Multimodal Score-CAM attribution maps are provided for the representative OASIS-3 subjects.The captions describe maps for concatenation- and cross-attention-fusion models.Source: Figure F12 · p. 29, Figure F13 · p. 30
Loading 2609.12410v1…