PaperlayerView exact revision on arXiv

Source-linked AI summary

Cross-Cohort Generalizability of Deep and Conventional Machine Learning for MRI-based Diagnosis and Prediction of Alzheimer's Disease

Esther E. Bron, Stefan Klein, Janne M. Papma, Lize C. Jiskoot, Vikram Venkatraghavan, Jara Linders, Pauline Aalten, Peter Paul De Deyn, Geert Jan Biessels, Jurgen A. H. R. Claassen, Huub A. M. Middelkoop, Marion Smits, Wiro J. Niessen, John C. van Swieten, Wiesje M. van der Flier, Inez H. G. B. Ramakers, Aad van der Lugt

arXiv:2012.08769v3eess.IVcs.CVphysics.med-ph

TL;DR

The generalizability of MRI-based machine-learning diagnosis and prediction for AD remains uncertain, despite strong reported performance and limited external validation. This study compares SVM and CNN classifiers with different MRI preprocessing approaches across ADNI and an independent tertiary memory-clinic cohort; performance was similar for AD-control classification, while MCI prediction and external validation showed task- and cohort-dependent differences.

  • Problem

    Although machine-learning methods show strong AD-control classification performance, evidence on external-cohort generalizability is limited and reported performance drops vary, making generalizability important to understand before routine clinical use.

    Source: §1 · p. 2

  • Method

    The study compares SVM and CNN classifiers using minimally processed structural MRI or modulated gray-matter maps, optimized with ADNI cross-validation and then evaluated for MCI conversion and in an independent cohort.

    Source: §1 · p. 2

  • Results

    SVM and CNN performed similarly for AD-control diagnosis and benefited from modulated gray-matter maps; MCI prediction favored SVM in ADNI but not PND, while external AD-control AUCs were 0.896 for SVM and 0.876 for CNN.

    Source: §4 · pp. 6–7

  • Takeaways & Limitations

    The authors conclude that ADNI classification generalized well to the multi-center PND cohort with only a minor performance drop, and that external validation may contribute to next steps toward clinical implementation.

    Source: §4 · p. 8

    AD diagnoses relied on clinical criteria rather than post-mortem histopathology, and limited follow-up diagnostic information left PND’s MCI sample relatively small with large prediction-performance confidence intervals.

    Source: §4 · p. 8

Abstract

from arXiv · show

This work validates the generalizability of MRI-based classification of Alzheimer's disease (AD) patients and controls (CN) to an external data set and to the task of prediction of conversion to AD in individuals with mild cognitive impairment (MCI). We used a conventional support vector machine (SVM) and a deep convolutional neural network (CNN) approach based on structural MRI scans that underwent either minimal pre-processing or more extensive pre-processing into modulated gray matter (GM) maps. Classifiers were optimized and evaluated using cross-validation in the ADNI (334 AD, 520 CN). Trained classifiers were subsequently applied to predict conversion to AD in ADNI MCI patients (231 converters, 628 non-converters) and in the independent Health-RI Parelsnoer data set. From this multi-center study representing a tertiary memory clinic population, we included 199 AD patients, 139 participants with subjective cognitive decline, 48 MCI patients converting to dementia, and 91 MCI patients who did not convert to dementia. AD-CN classification based on modulated GM maps resulted in a similar AUC for SVM (0.940) and CNN (0.933). Application to conversion prediction in MCI yielded significantly higher performance for SVM (0.756) than for CNN (0.742). In external validation, performance was slightly decreased. For AD-CN, it again gave similar AUCs for SVM (0.896) and CNN (0.876). For prediction in MCI, performances decreased for both SVM (0.665) and CNN (0.702). Both with SVM and CNN, classification based on modulated GM maps significantly outperformed classification based on minimally processed images. Deep and conventional classifiers performed equally well for AD classification and their performance decreased only slightly when applied to the external cohort. We expect that this work on external validation contributes towards translation of machine learning to clinical practice.

1. Introduction

Earlier diagnosis of Alzheimer’s disease (AD) could support treatment development and supportive care, but machine-learning performance may not generalize across cohorts. This study compares SVM and CNN classifiers, preprocessing approaches, and validation across ADNI and an external tertiary memory-clinic cohort.

1. Introduction

  • AD processes may begin 20 years or more before symptoms, while dementia diagnosis takes an average of 2.8 years in late-onset and 4.4 years in young-onset cases.Earlier and more accurate diagnosis could support development of disease-modifying treatments and access to supportive therapies that may delay institutionalization.Source: §1 · pp. 1–2
  • Prior studies reported AUCs of 85–98% for AD-control classification and 62–82% for predicting MCI conversion, with T1-weighted MRI and SVM commonly used.CNNs had not been shown to significantly outperform conventional classifiers, and the value of extensive MRI preprocessing for CNNs remained unclear.Source: §1 · p. 2
  • Only a few studies assessed external generalizability, with performance drops ranging from minor to severe; transfer appeared related to how well training data represented testing data.The authors considered a better understanding of generalizability crucial before routine clinical application.Source: §1 · p. 2
  • The study compares CNN and SVM classification using minimally processed MRI or modulated gray-matter maps, with optimization in ADNI and subsequent validation in ADNI MCI and an external cohort.It evaluates both AD diagnosis and prediction of AD in people with MCI.Source: §1 · p. 2

2. Methods

The study uses ADNI and an independent, multicenter tertiary memory-clinic cohort to compare MRI-based classification methods and preprocessing approaches. It evaluates SVM and CNN models across diagnostic and conversion-prediction tasks using cross-validation and independent test sets.

2.1. Study population

  • ADNI included 336 AD patients, 520 controls, 231 MCI converters, and 628 non-converters; the PND cohort included 199 AD patients, 138 participants with SCD, and 139 MCI patients with follow-up.Of the PND MCI group, 48 converted to dementia and 91 remained stable.Source: §2.1 · pp. 2–3

2.2. Imaging data

  • PND baseline T1-weighted MRI scans were acquired at 1.5T or 3T using protocols from eight centers.Most scans used an 8-channel head coil (423; 76%).Source: §2.2 · p. 3, Table 3 · p. 3

2.3. Image pre-processing

  • Minimal preprocessing corrected intensity non-uniformity, aligned scans to MNI space, and normalized intensities; the alternative pipeline produced modulated gray-matter maps adjusted for intracranial volume.The modulated maps used segmentation and deformation-field Jacobians to account for compression and expansion.Source: §2.3 · p. 3

2.4. Classification approaches

  • The study compared a linear-kernel SVM and a fully convolutional CNN for classification from structural MRI.The CNN used stride-two convolutional layers instead of pooling layers.Source: §2.4 · p. 4, §2.4.2 · p. 4

2.4.1. Support vector machine (SVM)

  • The linear SVM optimized its c-parameter with five-fold cross-validation on training data and normalized voxel features using training-set statistics.Statistical significance maps were used to identify features contributing to SVM decisions, with voxel clusters thresholded at α ≤ 0.05 and no multiple-comparison correction.Source: §2.4.1 · p. 4

2.4.2. Convolutional neural network (CNN)

  • The CNN used seven convolutional blocks and same-class mixup augmentation, blending two images in an 80:20 ratio to increase training samples and address class imbalance.Training used early stopping based on validation AUC; guided-backpropagation saliency maps illustrated predictive brain regions.Source: §2.4.2 · p. 4

2.5. Analysis and statistics

  • Performance was measured by AUC and accuracy; AD-CN models were evaluated over 20 stratified 90:10 train-test splits, with training-only optimization and corrected resampled-t confidence intervals.Retrained AD-CN classifiers were tested on ADNI MCI and both PND tasks; confidence intervals used 500 bootstrap samples, and classifier differences used McNemar tests with Bonferroni correction.Source: §2.5 · pp. 4–6

3. Results

In ADNI, modulated GM maps supported similar AD-CN AUCs for SVM and CNN, while SVM performed better for MCI conversion prediction. External validation showed similar AD-SCD classification patterns but lower MCI conversion performance; the classifiers also highlighted different brain regions.

3. Results

  • AUCs for AD-CN classification using modulated GM maps were 0.940 for SVM and 0.933 for CNN, indicating similar performance.Both classifiers performed better with modulated GM maps than with minimally processed T1w images.Source: §3 · p. 6
  • For ADNI MCI conversion using modulated GM maps, SVM achieved AUC=0.756 versus CNN's AUC=0.742, a significant difference (McNemar’s test, p < 0.01).SVM accuracy was 0.695; CNN accuracy was 0.658.Source: §3 · p. 6
  • In external AD-SCD diagnosis, AUC was 0.896 for SVM and 0.876 for CNN; performance patterns matched ADNI, but overall performance was slightly lower.Modulated GM maps yielded higher classification performance than minimally processed T1w images; PND and ADNI AUC confidence intervals overlapped.Source: §3 · p. 6
  • For PND MCI conversion using modulated GM maps, AUC was 0.702 for CNN and 0.665 for SVM; classifier and preprocessing differences were not significant.Performance was lower than in ADNI, with relatively large confidence intervals that overlapped ADNI intervals.Source: §3 · p. 6
  • SVM p-maps emphasized hippocampal clusters with modulated GM maps, whereas CNN saliency maps emphasized subcortical, periventricular white-matter, and cerebellar regions.For minimally processed T1w images, CNN saliency maps showed limited temporal-lobe contribution and emphasized brain-edge, periventricular white-matter, and cerebellar voxels.Source: §3 · p. 6

4. Discussion

Across ADNI and external PND validation, SVM and CNN showed similar AD-CN performance and both benefited from modulated GM maps; SVM’s MCI-conversion advantage appeared only in ADNI. The discussion also notes cohort differences, interpretability limits, and trade-offs for possible extensions.

  • SVM and CNN showed similar AD-CN performance across cohorts, both benefited from modulated GM maps, and SVM exceeded CNN for MCI conversion only in ADNI.Source: §4 · pp. 6, 8
  • CNNs needed less dedicated preprocessing than SVMs, although preprocessing still improved performance; alternative implementations were expected to preserve the conclusions.Source: §4 · p. 7
  • SVM and CNN visualizations implicated different regions, but the contrast may reflect classifier differences, visualization methods, or both.Source: §4 · p. 7
  • 0.896 for SVM and 0.876 for CNN were the PND AD-SCD AUCs, 0.04–0.07 below ADNI; PND MCI AUCs were 0.04–0.10 lower than ADNI.Source: §4 · p. 7
  • Cohorts differed in MRI protocols, country, control and MCI populations, and diagnostic criteria; studies with larger inclusion and diagnostic differences reported worse generalizability.Source: §4 · pp. 7–8
  • Clinical rather than post-mortem diagnoses may have misclassified some patients, while PND MCI estimates had wide confidence intervals and 1–5-year prediction horizons.Source: §4 · p. 8
  • Possible extensions include alternative CNN architectures, data augmentation, and multimodal inputs; added modalities may increase between-study differences and reduce generalizability.Source: §4 · pp. 6–8

Supplementary files

Supplementary Figure S1 concerns participant inclusion in the PND data set.

  • Supplementary Figure S1 depicts inclusion of participants in the PND data set.Source: Figure S1 · p. 11

Related papers

Loading 2012.08769v3…