Convolutional Neural Networks for Classification of Alzheimer's Disease: Overview and Reproducible Evaluation

Highlights

• The review identified data leakage that can bias results in some studies of CNN-based AD classification.The review focused on classification of AD using CNNs and T1 MRI.Source: Highlights · p. 2
• The authors proposed a framework for reproducible evaluation of AD classification methods.Source: Highlights · p. 2
• The proposed framework was demonstrated on three public datasets, with generalizability assessed within and between datasets.Source: Highlights · p. 2

1. Introduction

• CNNs can automatically learn latent features, unlike conventional machine learning approaches that use predefined features, motivating their investigation for medical-image classification.The introduction hypothesizes that deep learning may depend less on preprocessing and feature selection.Source: §1 · p. 5
• CNN study results are difficult to compare because participant sets, preprocessing, cross-validation procedures, and evaluation metrics differ, while inadequate validation can make generalization uncertain.The introduction notes risks including absent independent test sets and test sets drawn from the same study as training and validation data.Source: §1 · p. 5
• More than 30 studies had proposed CNNs for AD classification from anatomical MRI, but their validation procedures and potential data leakage required systematic review.The review focuses on validation practices and possible leakage.Source: §1 · p. 5
• The work reviews CNN studies using anatomical MRI, extends an open-source AD classification framework to deep learning, and rigorously evaluates CNN architectures against linear SVMs and across datasets.The evaluation examines key components, within-dataset generalization on ADNI, and cross-dataset generalization from ADNI to AIBL or OASIS.Source: §1 · pp. 5–6
• The framework and experiment code are publicly available through Clinica and the paper-specific AD-DL repository, with trained models hosted on Zenodo.The paper identifies code version v.0.0.1 as the version used to obtain its results.Source: §1 · p. 6

2.1. Main classification tasks

• AD vs CN was most common (25 of 32 studies), followed by MCI vs CN (9) and sMCI vs pMCI (7); studies also examined AD vs MCI and multiclass tasks.Source: §2.1 · p. 7

2.2. Main causes of data leakage

• The review identifies four leakage causes: wrong subject-level splits, preprocessing before the split, overlapping transfer-learning domains, and no independent test set.In a longitudinal dataset, an image-level split increased accuracy by 8 percentage points versus a subject-level split.Source: §2.2 · pp. 7–8

2.3.1. 2D slice-level CNN

• 2D slice CNNs can reuse natural-image architectures and increase training samples, but ignore 3D context and often confuse slice-level with subject-level accuracy.Only one reviewed study was free of detected leakage and biased metrics; only Qiu et al. (2018) explained slice-to-subject fusion.Source: §2.3.1 · p. 10

2.3.2. 3D patch-level CNN

• 3D patch methods use smaller inputs with lower memory demands, but independent CNNs per patch forgo the potential sample-size gain and make the framework complex.The review found fewer leakage problems for these methods, with a transfer-learning concern in Manhua Liu et al. (2018) that did not affect its AD vs CN result.Source: §2.3.2 · pp. 10–11

2.3.3. ROI-based CNN

• ROI-based CNNs focus on hippocampal inputs to reduce framework complexity, but examine only one or a few regions despite AD alterations spanning multiple brain areas.Smaller inputs (~3000 voxels in the reviewed studies) may reduce overfitting risk.Source: §2.3.3 · pp. 11–12

2.3.4. 3D subject-level CNN

• 3D subject-level CNNs use the whole MRI and integrate spatial information, but one sample per subject leaves relatively few samples and raises overfitting risk.The review detected leakage in several original-architecture studies and suspected it in others, limiting fair comparisons; some suspected-leakage studies reported 5 to 10 percentage-point accuracy gains from autoencoder pre-training.Source: §2.3.4 · p. 12

2.3.5. Conclusion

• About 50% of the 32 surveyed studies could report biased results: 10 were labeled Unclear and 6 Clear for data leakage.The count excludes Bäckström et al. (2018), who introduced leakage deliberately to study its impact.Source: §2.3.5 · p. 12
• Different datasets, subject selection, and preprocessing hinder comparison, while often-unreported architecture choices, unavailable code, and missing implementation details impede reproducibility.Source: §2.3.5 · p. 12

2.4. Other deep learning approaches for AD classification

• CNNs have served as feature extractors paired with other classifiers; Shmulev et al. (2018) found this more efficient than CNN classification.Reported classifiers include random forests, SVMs, logistic regression, extreme ML, and XGBoost.Source: §2.4 · p. 13
• CNNs have also been applied to clinical-score regression outputs and non-Euclidean data, including patient graphs and cortical surfaces (Parisot et al., 2018).Source: §2.4 · p. 13
• Non-CNN alternatives include stacked fully connected and probabilistic neural networks, plus deep Boltzmann machines, autoencoders, and deep polynomial networks; they often require extensive preprocessing and feature selection or embedding.The cited studies use an SVM for classification with representations learned by the Boltzmann machine, autoencoder, or polynomial network.Source: §2.4 · p. 13
• Longitudinal recurrent networks model temporal correlations, while multimodal methods combine imaging, demographic, genetic, clinical, or cerebrospinal-fluid data; both can be limited by small cohorts with complete data.Using multiple time points or modalities is expected to improve classification performance.Source: §2.4 · p. 13

3. Materials

• The study used T1-weighted MRI from three public datasets: ADNI, AIBL, and OASIS.MRI protocols are described in Samper-González et al. (2018).Source: §3 · p. 14
• The ADNI experiments used 1,455 participants with a T1w MR image available at at least one visit and considered five diagnosis groups: CN, AD, MCI, pMCI, and sMCI.pMCI subjects progressed to AD during the 36 months following the current visit; sMCI subjects did not. MCI sessions were from subjects diagnosed with MCI, EMCI, or LMCI at baseline who had no multiple reversions and conversions and did not convert back to CN.Source: §3 · p. 14
• AIBL included 598 participants with an available T1-weighted MRI and age value; its diagnosis-group criteria matched ADNI, and preprocessing removed seven sessions.The removals did not change the number of subjects.Source: §3 · p. 15
• OASIS included 193 participants aged at least 62 years; preprocessing excluded 22 AD and 17 CN subjects.The minimum age reflects the youngest participants diagnosed with AD.Source: §3 · p. 15
• ADNI and AIBL assigned CN, MCI, and AD labels after clinical tests, whereas OASIS used CDR-only assignment and had CN and AD labels, with MCI labelled AD.Because diagnostic criteria differ, OASIS labels are not strictly equivalent to those in ADNI and AIBL (Ellis et al., 2010, 2009; Petersen et al., 2010; Marcus et al., 2007).Source: §3 · p. 15

4.1. Converting datasets to a standardized data structure

• The framework converted ADNI, AIBL, and OASIS data to BIDS and organized experiment outputs in a standardized BIDS-inspired structure.Preprocessing used Nipype, deep-learning models used PyTorch, and the linear SVM used scikit-learn.Source: §4.1 · p. 16, §4.7 · p. 22

4.2. Preprocessing of T1w MRI

• Minimal preprocessing used bias correction and affine registration, while extensive preprocessing added nonlinear registration and skull-stripping; intensity rescaling was optional in the minimal pipeline.For extensive preprocessing, a mask-based skull-stripping approach was chosen for robustness despite lower accuracy than direct image-based methods.Source: §4.2 · p. 16

4.3. Classification models

• The study compared four CNN approaches: 3D subject-level, 3D ROI-based, 3D patch-level, and 2D slice-level.The subject-level CNN used five convolutional blocks and three fully connected layers; the ROI- and patch-level CNNs used four convolutional blocks and three fully connected layers.Source: §4.3 · p. 17, §4.3.1 · p. 17, §4.3.2 · p. 18

4.3.3. 2D slice-level CNN

• The 2D approach fine-tuned an ImageNet-pretrained ResNet on sagittal slices, excluding the first and last 20 slices to yield 129 RGB slices per subject.The other ResNet layers were frozen to limit overfitting.Source: §4.3.3 · p. 19

4.3.4. Majority voting system

• For 3D patch, 3D ROI, and 2D slice models, subject-level decisions used soft voting weighted by validation accuracy; multi-CNN patch classifiers below 0.7 balanced accuracy received zero weight.The test-set voting weights were computed on the validation set, and the resulting probabilities were not calibrated.Source: §4.3.4 · p. 20

4.4. Transfer learning

• Transfer learning used autoencoder pre-training for 3D CNNs and ImageNet-pretrained ResNet for 2D CNNs; AD-versus-CN models were transferred to sMCI-versus-pMCI prediction.Autoencoders used all available training-set diagnostic groups, with the same training/validation splits as classification to avoid data leakage.Source: §4.4 · p. 21, §4.4.1 · p. 21, §4.4.2 · p. 21

4.6.1. Validation procedure

• The test set was reserved until model selection finished, with 5-fold cross-validation on training/validation data and nested 10-fold CV to choose the linear SVM regularization parameter.The ADNI test set contained 100 age- and sex-matched subjects per diagnostic class; AIBL and OASIS served as external test sets, with different expected generalization because OASIS had different inclusion criteria and acquisition parameters.Source: §4.6.1 · pp. 21–22

4.6.2. Metrics

• Performance metrics included balanced accuracy, AUC, accuracy, sensitivity, and specificity, while the manuscript reported balanced accuracy for concision.Classification used early stopping based on validation loss and selected the model with the highest validation balanced accuracy.Source: §4.6.2 · p. 22, §4.7 · p. 23

5.1. Results on training/validation set

• All tested pipelines achieved the expected balanced accuracy on synthetic datasets: 0.5 for inseparable data and 1.00 for fully separable data.Source: §5.1 · p. 23

5.1.1. 3D subject-level

• For 3D subject-level AD-vs-CN classification, intensity rescaling raised BA from 0.50 to 0.80, while minimal and extensive preprocessing yielded 0.85 and 0.86.AE pretraining yielded 0.82 versus 0.80 without it; longitudinal training yielded 0.85 versus 0.82 with baseline data only. For sMCI versus pMCI, baseline and longitudinal data both yielded 0.73 BA.Source: §5.1.1 · p. 23

5.1.2. 3D ROI-based

• For 3D ROI-based models, AD-vs-CN BA was 0.88 with baseline and 0.86 with longitudinal data; sMCI-vs-pMCI BA was 0.77 and 0.78, respectively.The reported sMCI-vs-pMCI performance was substantially higher than for the 3D subject-level approach.Source: §5.1.2 · p. 24

5.1.3. 3D patch-level

• For 3D patch-level AD-vs-CN classification, multi-CNN BA was 0.81 with baseline and 0.83 with longitudinal data, outperforming the single CNN.The single CNN yielded 0.74 and 0.76 BA, respectively. Multi-CNN sMCI-vs-pMCI BA was 0.75 with baseline and 0.77 with longitudinal data, slightly below ROI-based performance.Source: §5.1.3 · p. 24

5.1.4. 2D slice-level

• The 2D slice-level approach generally underperformed the 3D approaches, and longitudinal training did not improve its BA, which was 0.79 with baseline data and 0.74 with longitudinal data.A slice-level data-split strategy produced BA of 1.00, as expected under data leakage.Source: §5.1.4 · p. 24

5.1.5. Linear SVM

• The linear SVM achieved AD-vs-CN BA of 0.88 with baseline and 0.87 with longitudinal data, outperforming most deep-learning models and matching the best ones.For sMCI versus pMCI, SVM BA was 0.68 with either training set from scratch and 0.70 with either set after transfer learning; these results were below deep-learning models.Source: §5.1.5 · p. 24

5.2. Results on the test sets

• AD-vs-CN models generally performed better on ADNI and AIBL than on OASIS, where test-set performance declined.The 3D subject-level models lost over 0.15 balanced-accuracy points on OASIS; the 3D ROI-based models lost 0.13 to 0.19 points, and 2D slice models lost 0.13 to 0.14 points. The table summarizes test results across ADNI, AIBL, and OASIS.Source: §5.2.1 · p. 28, §5.2.2 · p. 28, §5.2.3 · p. 28
• For sMCI-vs-pMCI, test performance was generally stronger on ADNI than AIBL, although the SVM generalized excellently to ADNI.Subject-level models lost over 0.20 points on AIBL, while patch-level performance on ADNI was 0.05 to 0.07 points below validation and was very poor on AIBL.Source: §5.2.1 · p. 28, §5.2.2 · p. 28, §5.2.3 · p. 28

• Potential data leakage affected half of 32 surveyed studies (evident in six, possible in ten); simulated leakage raised BA to 1.00 on validation versus 0.75 on ADNI test and 0.80 on AIBL test.The authors also flag inadequate descriptions of validation and possible test-set use for model selection as concerns in prior work.Source: §6 · p. 32
• The open-source framework extends existing tools to DL with BIDS-based data management, MRI preprocessing, representative CNNs, and rigorous validation procedures.It is designed to let researchers embed new pipelines or architectures and evaluate their added value.Source: §6 · p. 32
• On ADNI testing, CNN diagnostic BA ranged from 0.76 to 0.89 for AD vs CN and from 0.69 to 0.74 for sMCI vs pMCI, within the reported state-of-the-art ranges.The authors suggest that performance below top-performing studies may reflect their fully independent test set, which was never used to select architectures or parameters.Source: §6 · p. 33
• On test-set results, 3D subject-level, ROI-based, and patch-level CNNs achieved approximately the same performance; collectively, 3D approaches outperformed 2D slice-level CNNs, with no strong superiority conclusion among 3D methods.Source: §6 · p. 33
• With ADNI-sized data, a linear SVM with voxel-based features was at least as good as the best CNNs on both tasks; tested CNNs provided no performance increase.The authors do not rule out better performance from more sophisticated methods or larger datasets.Source: §6 · p. 33
• Models generalized well to ADNI and to AIBL for AD vs CN, but poorly to AIBL sMCI vs pMCI and OASIS.The authors suggest older AIBL participants, the small sMCI vs pMCI sample (33 participants), and less rigorous OASIS diagnosis criteria may contribute; they conclude that cross-population generalization remains unresolved.Source: §6 · p. 34
• Intensity rescaling was essential for CNN training, whereas nonlinear versus linear registration produced no improvement; AE pretraining and longitudinal data yielded only slight BA gains.Prior studies reporting AE-pretraining gains were suspected of data leakage, so the authors conclude that a significant BA increase from pretraining is unproven.Source: §6 · p. 34
• Under the same thresholding conditions, multi-CNN and single-CNN 3D-patch approaches had comparable results, favoring simpler frameworks such as 3D ROI or 3D subject.Without thresholding, AD vs CN BA was 0.76 and 0.72 for multi-CNN on baseline and longitudinal data, versus 0.74 and 0.76 for single-CNN.Source: §6 · p. 35

Supplementary Material

• The supplement provides literature-review methods, topic introductions, dataset descriptions, model architecture and training hyperparameters, and plots of training and validation loss and accuracy.Architecture hyperparameters appear in eTables 1–3, training hyperparameters in eTables 4–5, and monitoring plots in eFigures 1–4.Source: Supplementary Material · p. 45

eMethod 1. Literature search methodology

• The search covered PubMed and Scopus through January 15, 2019, using terms for Alzheimer’s disease, classification, deep learning, and neuroimaging, with Scopus excluding EEG-related records.The neuroimaging query covered all fields because some papers did not specify imaging modalities in their title, abstract, or keywords.Source: eMethod 1. Literature search methodology · p. 47
• The review ended with 71 conference and journal articles, including 32 focused on the paper’s topic, after screening 395 records through abstract, type, and content filters.The type filter retained all journal articles and conference papers published since 2017; the final content screen excluded 22 papers.Source: eMethod 1. Literature search methodology · pp. 47–48

eMethod 2. Convolutional neural networks

• CNN convolutional layers learn image features, pooling reduces feature-map dimensionality, and activation and fully connected layers support nonlinear mapping from features to class predictions.The supplement also explains losses and optimization, including cross-entropy and stochastic gradient descent.Source: eMethod 2. Convolutional neural networks · p. 49
• Classical CNN designs span AlexNet, VGGNet, GoogleNet, ResNet, and DenseNet, with innovations including small filters, inception layers, shortcut connections, and dense connectivity.These architectures were originally designed for 2D natural images, though some have been adapted for MRI.Source: eMethod 2. Convolutional neural networks · pp. 49–50
• Because small neuroimaging datasets make deep-learning models prone to overfitting, the supplement describes augmentation, dropout, weight decay, batch normalization, and early stopping as countermeasures.Augmentation may transform training images or synthesize samples; early stopping can select parameters at the lowest validation error.Source: eMethod 2. Convolutional neural networks · pp. 50–51
• Transfer learning initializes an AD classifier from an autoencoder encoder or an ImageNet-trained model, aiming to provide a better starting point when labeled data are limited or random initialization generalizes poorly.For autoencoder pretraining, the encoder compresses the input and its weights initialize the target network; the decoder reconstructs the input.Source: eMethod 2. Convolutional neural networks · p. 51

eMethod 3. MRI preprocessing

• MRI preprocessing is important for classification; bias-field inhomogeneities alter tissue intensities across locations and can impair image analysis, including registration.N3 and N4 are popular correction methods (Sled et al., 1998; Tustison et al., 2010).Source: eMethod 3. MRI preprocessing · p. 52
• MRI intensity ranges and tissue distributions vary across scans, motivating global rescaling and intensity standardization such as histogram matching.Global rescaling can map image intensities between 0 and 1 using minimum and maximum values; histogram matching is one standardization technique.Source: eMethod 3. MRI preprocessing · p. 52
• Skull stripping removes extracranial tissues that can obstruct image analysis, but noise and artifacts can cause brain over- or under-segmentation.Examples include BET in FSL and BSE in BrainSuite.Source: eMethod 3. MRI preprocessing · p. 52
• Image registration aligns scans so corresponding voxels contain comparable information, using global rigid or affine alignment or local non-rigid alignment.FLIRT and FNIRT in FSL, as well as SPM and ANTs, provide linear and/or non-linear registration tools.Source: eMethod 3. MRI preprocessing · p. 52

eMethod 4. Main datasets used for AD classification

• ADNI, AIBL, and OASIS are the main public datasets for AD classification, but their diagnostic criteria make their labels non-equivalent.The paper also notes that OASIS labels do not map strictly to those of ADNI and AIBL.Source: eMethod 4. Main datasets used for AD… · p. 53
• ADNI has four dependent, longitudinal cohorts, and its physician-assigned labels include AD, MCI, NC, and SMC, with MCI later divided into EMCI and LMCI.EMCI and LMCI denote the beginning and end of the prodromal phase; the categories are often fused as MCI because they remain similar.Source: eMethod 4. Main datasets used for AD… · p. 53
• AIBL is a longitudinal study with clinical, MRI, PET, blood, and CSF data, and its clinical-test diagnoses include AD, MCI, and NC.Diagnosis follows a series of clinical tests (Ellis et al., 2010, 2009).Source: eMethod 4. Main datasets used for AD… · p. 53
• OASIS has three cohorts; OASIS-1 is cross-sectional, while OASIS-2 and OASIS-3 are longitudinal, and its labels are based on the CDR scale.OASIS-1 AD includes nonzero CDR scores and spans cases distributed between ADNI's MCI and AD categories; some OASIS controls are younger than 55.Source: eMethod 4. Main datasets used for AD… · pp. 53–54

• Five 2x2x2 max-pooling layers use stride 2 and adaptive padding, which adds zeros until each dimension is a stride multiple.Source: eTable 1. Architecture hyperparameters… · p. 55
• Five 3x3x3 Conv+BN+ReLU layers use 8, 16, 32, 64, then 128 filters, each with stride 1 and padding 1.Source: eTable 1. Architecture hyperparameters… · p. 55
• After dropout of 0.5, the fully connected layers have 1300, 50, and 2 neurons; preprocessing changes only FC1's input size.Source: eTable 1. Architecture hyperparameters… · p. 55

eTable 2. Architecture hyperparameters for 3D ROI-based and patch-level CNN

• Zero columns are added along a dimension until its size is a multiple of the pooling stride.The required padding depends on the input dimensions.Source: eTable 2. Architecture hyperparameters… · p. 56

A. Characteristics of the downsampling layers

• For shortcuts linking feature maps of different sizes, a downsampling layer is applied to the larger map.Source: eTable 3. Architecture hyperparameters… · p. 57

B. Architecture of the 2D slice-level CNN (adaptation of the ResNet-18)

• The 2D slice-level CNN adapts ResNet-18; shortcut arrows add linked feature maps and apply ReLU before passing the result to the next layer.Source: eTable 3. Architecture hyperparameters… · p. 57

eTable 4. Training hyperparameters for classification experiments

• Classification hyperparameters are listed by experiment number, with experiment summaries and corresponding settings organized in Tables A and B.The summary identifies intensity rescaling, data split, and training approach as experiment descriptors.Source: eTable 4. Training hyperparameters for… · pp. 58–59
• Classification experiments use Adam with betas=(0.9, 0.999), epsilon=1e-8, and cross-entropy loss.These are the common hyperparameters for all classification experiments.Source: eTable 4. Training hyperparameters for… · p. 58
• Transfer-learning experiment numbers appear in brackets; the corresponding settings are in eTable 5 for AE pretraining and eTable 4 for cross-task transfer learning.Source: eTable 4. Training hyperparameters for… · p. 58

eTable 5. Training hyperparameters for autoencoder pretraining experiments

• Autoencoder-pretraining summaries and their hyperparameters are organized in Tables A and B using the same experiment numbers.Source: eTable 5. Training hyperparameters for… · p. 60
• Autoencoder-pretraining experiments use Adam with betas=(0.9, 0.999), epsilon=1e-8, and mean squared entropy loss.The stated common settings also specify AD + MCI + CN training data, a subject-level split, and stopping at the maximal number of epochs.Source: eTable 5. Training hyperparameters for… · p. 60
• The fragments list single-CNN entries alongside preprocessing labels, including MinMax, Extensive, Minimal, and Longitudinal; run numbers 2, 4, and 6 also appear, but their pairings are unclear.Source: eTable 5. Training hyperparameters for… · p. 60

eFigure 1. Training process monitoring for 3D subject-level CNN

• For the 3D subject-level CNN, training and validation accuracy/loss were evaluated after the forward pass of 20 batches.Source: eFigure 1. Training process monitoring… · p. 61
• For the 3D subject-level CNN, accuracy and loss curves were smoothed with a threshold of 0.6.Source: eFigure 1. Training process monitoring… · p. 61
• For each subject-level plot, a subfigure letter identifies the plot and its experiment information is found in the table below or eTable 4 by experiment number.Source: eFigure 1. Training process monitoring… · p. 61

eFigure 2. Training process monitoring for 3D ROI-based CNN

• For the 3D ROI-based CNN, training and validation accuracy/loss were evaluated after each epoch.Source: eFigure 2. Training process monitoring… · p. 62
• For the 3D ROI-based CNN, accuracy and loss curves were smoothed with a threshold of 0.6.Source: eFigure 2. Training process monitoring… · p. 62
• For each ROI-based plot, a subfigure letter identifies the plot and its experiment information is found in the table below or eTable 4 by experiment number.Source: eFigure 2. Training process monitoring… · p. 62

eFigure 3. Training process monitoring for 3D patch-level CNN

• Training and validation accuracy and loss were evaluated after each epoch, and the curves were smoothed with threshold 0.6.Plot letters map to experiment information by experiment number; multi-CNN plots also identify the CNN number.Source: eFigure 3. Training process monitoring… · p. 63

eFigure 4. Training process monitoring for 2D slice-level CNN

• Training and validation accuracy and loss were evaluated after each epoch, and the curves were smoothed with threshold 0.6.Plot letters identify experiments by experiment number.Source: eFigure 4. Training process monitoring… · p. 64

eTable 6. Experiments performed with the single-CNN using thresholding

• CNN intensity rescaling used MinMax values to map intensities to [0, 1].AE denotes autoencoder.Source: eTable 6. Experiments performed with… · p. 65
• Patches with balanced accuracy < 0.7 were excluded from soft voting because their label probabilities could harm majority voting.Source: eTable 6. Experiments performed with… · p. 65

eTable 7. Experiments performed with the multi-CNN without thresholding

• For sMCI-versus-pMCI tasks, weights and biases learned on AD-versus-CN were transferred to a model fine-tuned on the target task.CNN intensities were rescaled by MinMax to [0, 1]; AE denotes autoencoder.Source: eTable 7. Experiments performed with… · p. 65
• All classifiers composing the multi-CNN were included in soft voting, without thresholding.Source: eTable 7. Experiments performed with… · p. 65