PaperlayerView exact revision on arXiv

Source-linked AI summary

Convolutional Neural Networks for Classification of Alzheimer's Disease: Overview and Reproducible Evaluation

Junhao Wen, Elina Thibeau-Sutre, Mauricio Diaz-Melo, Jorge Samper-Gonzalez, Alexandre Routier, Simona Bottani, Didier Dormont, Stanley Durrleman, Ninon Burgos, Olivier Colliot

arXiv:1904.07773v6cs.LGeess.IVstat.ML

TL;DR

Existing CNN studies for AD classification are difficult to compare and reproduce, and some may report biased performance because of inadequate validation or data leakage. The paper reviews this literature, extends an open-source CNN evaluation framework, and rigorously compares architectures. Its evaluation finds substantial leakage concerns and shows that tested CNNs do not outperform a linear SVM, with generalization constrained across differing populations.

  • Problem

    CNN-based AD classification studies are difficult to compare and reproduce because datasets, preprocessing, validation, implementation details, and code availability vary.

  • Method

    The authors reviewed CNN studies, extended an open-source framework for reproducible CNN evaluation, and used strict validation with untouched test sets.

  • Results

    About 50% of surveyed studies could report biased results from clear or unclear data leakage, while tested CNNs were at least no better than a linear SVM.

  • Takeaways & Limitations

    The framework provides a more objective baseline for CNN AD classification and supports evaluation of generalization within and across datasets.

  • Takeaways & Limitations

    The study cannot exclude that other architecture or hyperparameter choices would produce better results, and cross-validation was performed only once.

Abstract

from arXiv · show

Over 30 papers have proposed to use convolutional neural network (CNN) for AD classification from anatomical MRI. However, the classification performance is difficult to compare across studies due to variations in components such as participant selection, image preprocessing or validation procedure. Moreover, these studies are hardly reproducible because their frameworks are not publicly accessible and because implementation details are lacking. Lastly, some of these papers may report a biased performance due to inadequate or unclear validation or model selection procedures. In the present work, we aim to address these limitations through three main contributions. First, we performed a systematic literature review and found that more than half of the surveyed papers may have suffered from data leakage. Our second contribution is the extension of our open-source framework for classification of AD using CNN and T1-weighted MRI. Finally, we used this framework to rigorously compare different CNN architectures. The data was split into training/validation/test sets at the very beginning and only the training/validation sets were used for model selection. To avoid any overfitting, the test sets were left untouched until the end of the peer-review process. Overall, the different 3D approaches (3D-subject, 3D-ROI, 3D-patch) achieved similar performances while that of the 2D slice approach was lower. Of note, the different CNN approaches did not perform better than a SVM with voxel-based features. The different approaches generalized well to similar populations but not to datasets with different inclusion criteria or demographical characteristics.

Highlights

The work reviews CNN-based AD classification, identifies data leakage and proposes a reproducible evaluation framework. It demonstrates the framework on public datasets and assesses generalizability within and across datasets.

  • The review identified data leakage that can produce biased results in some AD classification studies.
  • The authors proposed a framework for reproducible evaluation of AD classification methods.
  • The framework was demonstrated on three public datasets.
  • The study assessed generalizability both within a dataset and between datasets.

Graphical abstract

The paper concerns CNNs, reproducibility, Alzheimer's disease classification, and MRI-based analysis.

  • CNNs are a central methodological focus of the paper.
  • Reproducibility is a central theme of the evaluation framework.
  • The application domain is Alzheimer's disease classification using MRI.

1. Introduction

CNN-based AD classification studies are difficult to compare and reproduce because they vary in data, preprocessing, validation, and implementation details. This work reviews those studies, extends an open-source evaluation framework to deep learning, and applies it to compare CNN architectures and generalization.

  • CNN studies for AD classification are difficult to compare because participant sets, preprocessing, and validation procedures differ.
  • The authors extended an open-source reproducible evaluation framework to deep-learning approaches.
  • The framework supports modular image preprocessing, classification architectures, and evaluation procedures dedicated to deep learning.
  • The study rigorously compared CNN architectures, examined key components affecting accuracy, and compared CNNs with a linear SVM.
  • The framework and experiments were publicly released through Clinica, paper-specific code, and trained models.

2. State of the art

The literature on CNN-based AD classification using anatomical MRI is difficult to compare and may contain biased evaluations, especially from data leakage. Studies use varied input formats and report heterogeneous results, while reproducibility and methodological transparency remain limited.

  • Literature scope: 32 studies used end-to-end CNNs on anatomical MRI among 71 studies identified by the literature search.The search covered PubMed and Scopus publications from January 1990 through 15 January 2019.
  • Evaluation bias: Data leakage can arise from subject-level split errors, late preprocessing or augmentation, overlapping transfer-learning domains, or omission of an independent test set.Subject-level splitting is especially important when images contain multiple slices, patches, or longitudinal scans.
  • Reported performance: Reported examples include ACC=0.87 for a 3D patch-level study, ACC=0.93 for a 2D slice-level study, ACC=0.80 and ACC=0.88 for 3D subject-level studies, and ACC=0.90 for two 3D patch-level studies.These values come from heterogeneous studies and should not be treated as directly comparable.
  • Input representations: 2D slice-level methods analyze slices independently, can confuse slice-level with subject-level accuracy, and may overlook the three-dimensional structure of MRI.Slice selection and subject-level fusion also vary across studies.
  • Evaluation bias: About 50% of surveyed CNN studies could report biased results because 10 were unclear and 6 had clear data leakage.The deliberately induced leakage study was excluded from this count.
  • Comparability and reproducibility: The reviewed studies are difficult to compare because datasets, participant selection, preprocessing, architectures, hyperparameters, and reporting practices differ.Code and key implementation details were often unavailable, further limiting reproducibility.

3. Materials

The study used T1-weighted MRI from ADNI, AIBL, and OASIS, with dataset-specific diagnostic groups and participant-selection criteria. These differences limit strict equivalence across datasets.

  • T1-weighted MRI was obtained from the ADNI, AIBL, and OASIS public datasets.
  • ADNI included five diagnosis groups: CN, AD, MCI, pMCI, and sMCI.The supplied passages list the group definitions separately.
  • AIBL comprised 598 participants with an available T1-weighted MR image and age value at least once.Seven sessions were removed during preprocessing without changing the number of subjects.
  • OASIS comprised 193 participants aged at least 62 years, with 22 AD and 17 CN subjects excluded after preprocessing.
  • OASIS used CN and AD labels only, with MCI subjects labelled as AD, whereas ADNI and AIBL used CN, MCI, and AD labels.The diagnostic criteria differed across studies, so the labels are not strictly equivalent.

4. Methods

The framework standardized data management, preprocessing, model selection, and evaluation across four CNN input strategies. The study explored architectural and training choices for subject-level, ROI-based, patch-level, and slice-level models.

  • The framework included public-dataset conversion, MRI preprocessing, classification models, transfer learning, classification tasks, evaluation, and implementation components.
  • Preprocessing: The study compared Minimal and Extensive MRI preprocessing procedures because the benefit of more extensive preprocessing for CNNs remained unclear.
  • Four CNN approaches were evaluated: 3D subject-level, 3D ROI-based, 3D patch-level, and 2D slice-level.
  • CNN architectures: The 3D subject-level CNN used five convolutional blocks and three fully connected layers after iterative architecture exploration.
  • CNN architectures: The 2D slice-level model used ResNet-18 with added fully connected layers, dropout, and selected-layer fine-tuning.
  • CNN architectures: The 3D ROI-based and patch-level CNNs used four convolutional blocks and three fully connected layers.
  • CNN architectures: Patch-level processing generated 36 non-overlapping 50×50×50 mm^3 patches, trained either with one CNN or 36 patch-specific CNNs.

5. Experiments and results

Experiments compared CNN configurations using balanced accuracy and evaluated their generalization across ADNI, AIBL, and OASIS test sets. Performance depended on preprocessing and architecture, while cross-dataset generalization varied substantially.

  • Component analyses: 0.80 balanced accuracy with intensity rescaling versus 0.50 without it, which also produced a generalization gap.Intensity rescaling was therefore used in subsequent experiments.
  • Component analyses: 0.82 balanced accuracy with autoencoder pre-training versus 0.80 without it, a small difference that motivated using pre-training thereafter.The comparison concerns transfer learning through AE pre-training.
  • Component analyses: 0.85 balanced accuracy with longitudinal data versus 0.82 with baseline data only.The authors continued exploring this choice because architectures differed in parameter count and effective sample augmentation.
  • Component analyses: 0.86 balanced accuracy with extensive preprocessing versus 0.85 with minimal preprocessing, leading to selection of minimal preprocessing.The two preprocessing procedures produced almost equivalent performance.
  • CNN comparisons: For AD vs CN, multi-CNN balanced accuracy was 0.81 with baseline data and 0.83 with longitudinal data, outperforming the single-CNN approach.For sMCI vs pMCI, multi-CNN balanced accuracy was 0.75 with baseline data and 0.77 with longitudinal data, slightly below the 3D ROI-based approach.
  • CNN comparisons: 2D slice-level performance was lower than several 3D approaches, while a slice-level data split produced 1.00 balanced accuracy under data leakage.For 2D slice-level training, balanced accuracy was 0.79 with baseline data and 0.74 with longitudinal data.
  • Test-set generalization: Models generalized well to ADNI, while performance decreased on datasets with differing characteristics, including OASIS for AD vs CN and AIBL for sMCI vs pMCI.For AD vs CN, OASIS losses exceeded 0.15 points; for sMCI vs pMCI, AIBL losses exceeded 0.20 points.

6. Discussion

The study identifies substantial reproducibility and validation problems in CNN-based AD classification, then applies an open-source framework to compare architectures under rigorous testing. Three-dimensional CNN approaches performed similarly, 2D slices performed worse, CNNs did not outperform a linear SVM, and generalization weakened across populations with differing characteristics.

  • Framework: The authors extended an open-source framework for reproducible CNN evaluation and applied it to ADNI, AIBL, and OASIS.The framework supports image processing, classification architectures, and evaluation procedures dedicated to deep learning.
  • Literature review: Data leakage was potentially present in half of the 32 surveyed studies, potentially biasing reported performance upward.Leakage was evident in six studies and possible because of inadequate validation descriptions in ten others.
  • Performance: On ADNI, CNN diagnostic BA ranged from 0.76 to 0.89 for AD vs CN and from 0.69 to 0.74 for sMCI vs pMCI.The independent test set was not used to choose architectures or parameters, providing a baseline for new methods.
  • Architecture comparison: 3D subject-level, ROI-based, and patch-level CNNs achieved approximately the same performance, whereas the 2D-slice approach was less efficient.The comparison covered four CNN approaches and was based on test-set results.
  • Comparison with conventional ML: With the ADNI sample size, CNNs did not improve performance compared with a linear SVM using standard voxel-based features.The SVM was at least as good as the best CNNs for both AD vs CN and sMCI vs pMCI.
  • Generalization: Models generalized well to ADNI and some AIBL settings but generalized poorly to AIBL sMCI vs pMCI and OASIS.The latter datasets differed in participant characteristics, inclusion criteria, imaging acquisitions, or sample size.
  • Limitations: The study’s limitations include untested architecture and hyperparameter choices, a single cross-validation run, and overfitting associated with limited dataset sizes.The authors note that larger datasets could help address overfitting, while the framework permits testing alternative models without touching test datasets.

Supplementary Material

The supplementary material details the literature-search strategy, screening criteria, and retained studies for the review.

  • Literature search: The search covered AD, classification, deep learning, and neuroimaging concepts in PubMed and Scopus.The search was conducted for publications available through 15 January 2019.
  • Literature search: 406 records were identified before screening and filtering.Records from Scopus and PubMed were merged and duplicates removed before additional exclusions.
  • Screening: Records were screened in three stages based on abstract, publication type, and content.The screening retained studies focused on AD-stage classification using anatomical MRI as input to a neural network.
  • Screening: Recent conference papers were retained alongside journal articles, leaving 93 items after type-based screening.Older conference papers were excluded, while conference papers published since 2017 were kept.
  • Screening: Content screening excluded studies using other modalities, duplicate content, insufficient methodological explanation, or inaccessible material.This step excluded 22 papers across the listed criteria.

eMethod 2. Convolutional neural networks

This section introduces CNN components, classical architectures, overfitting countermeasures, and autoencoder-based transfer learning for AD classification.

  • CNN components: CNNs combine convolutional, pooling, activation, and fully connected layers between input and output layers.These hidden layers support feature extraction, dimensionality reduction, nonlinear representation, and prediction.
  • CNN components: Convolutional layers automatically extract features using learnable 2D or 3D filters applied across receptive fields.For MRI, filters are described as 3D matrices of weights.
  • Classical architectures: Classical CNN architectures such as AlexNet, VGGNet, ResNet, and DenseNet differ in depth, filter design, shortcut connections, or dense connectivity.These architectures were originally designed for 2D natural images, with adaptations for MRI applications.
  • Overfitting: AD neuroimaging datasets are small relative to computer-vision datasets, making deep-learning models prone to overfitting.The section discusses augmentation, dropout, weight decay, and batch normalization as strategies to alleviate overfitting.
  • Transfer learning: Autoencoder transfer learning initializes a target CNN with encoder weights learned through unsupervised reconstruction.The encoder compresses input data into a representation, while the decoder reconstructs the input from that representation.

eMethod 4. Main datasets used for AD classification

The study uses ADNI, AIBL, and OASIS, whose diagnostic labels and cohort characteristics differ across datasets.

  • Datasets: ADNI, AIBL, and OASIS are the three principal publicly available datasets described for AD studies.The datasets provide MRI alongside differing clinical and diagnostic information.
  • ADNI: ADNI defines AD, MCI, and normal-control groups, with later cohorts separating early MCI from late MCI.The early- versus late-MCI division is based on memory-task scores corrected by education level.
  • AIBL: AIBL is a longitudinal cohort containing clinical, imaging, blood, and cerebrospinal-fluid data.Its goals include studying biomarkers, cognition, and lifestyle factors related to AD development.
  • OASIS: OASIS diagnosis labels are based on the clinical dementia rating scale, with OASIS-1 distinguishing nonzero-CDR AD from zero-CDR controls.OASIS-1 controls can include participants younger than 55, unlike ADNI controls.

eTable 1. Architecture hyperparameters for 3D subject-level CNN

The 3D subject-level CNN uses five convolutional blocks, dropout, three fully connected layers, and a two-class softmax output, with input-dependent FC1 sizing.

  • Preprocessing dependence: The architecture differs between Minimal and Extensive preprocessing only through the input size of FC1.The reported output sizes depend on the preprocessing used.
  • Padding: Padding of 1 in convolutional layers prevents output shrinkage that would make image reconstruction too small for the Extensive preprocessing autoencoder.Pooling-layer padding instead depends on the input size and stride.
  • Architecture: The architecture contains five convolutional blocks followed by dropout and three fully connected layers.The convolutional blocks use convolution, batch normalization, ReLU activation, and max pooling.
  • Architecture: The fully connected layers contain 1300, 50, and 2 neurons, respectively.The final two-neuron layer feeds a softmax output for classification.

eTable 2. Architecture hyperparameters for 3D ROI-based and patch-level CNN

The architecture notes specify adaptive pooling padding and abbreviations used for the 3D ROI-based and patch-level CNNs.

  • Pooling layers receive zero-padding until each dimension is a multiple of the stride size.The padding amount depends on the input dimension.
  • BN denotes batch normalization, Conv convolutional layer, FC fully connected, and MaxPool max pooling.

eTable 3. Architecture hyperparameters for 2D slice-level CNN

The 2D slice-level CNN is documented through its ResNet-18 adaptation, shortcut operations, experiment summaries, and shared training settings.

  • Architecture: Shortcuts connecting feature maps of different sizes apply downsampling to the largest feature map.
  • Architecture: The 2D slice-level CNN adapts ResNet-18, with shortcuts adding linked feature maps followed by ReLU.
  • Experiment design: Experiments are summarized in Table A, while corresponding hyperparameters are identified by matching experiment numbers in Table B.
  • Training settings: All experiments use Adam with betas=(0.9, 0.999), epsilon=1e-8, and cross-entropy loss.
  • Training settings: Transfer-learning experiments are cross-referenced to eTables 4 and 5 for cross-task transfer and autoencoder pretraining, respectively.
  • Experiment design: The experiment tables organize training approach, intensity rescaling, data split, architectures, training data, and image preprocessing.

eFigure 1. Training process monitoring for 3D subject-level CNN

Training-process monitoring records accuracy and loss for training and validation, with smoothed curves and experiment-specific plot references.

  • Monitoring procedure: Accuracy and loss curves are smoothed using a threshold of 0.6.
  • Figure organization: Each plot uses a subfigure letter linked to experiment information in the accompanying table or eTable 4.
  • Monitoring procedure: Training and validation accuracy and loss are evaluated after each epoch during the training process.
  • Figure organization: For multi-CNN experiments, the plot information also identifies the CNN number.
  • Preprocessing and transfer learning: MinMax intensity rescaling maps CNN values to the range [0, 1].AE denotes autoencoder, and transfer learning moves weights and biases from AD versus CN to sMCI versus pMCI before fine-tuning.
  • Ensembling: Patches with balanced accuracy below 0.7 are excluded from soft voting, whereas all multi-CNN classifiers are included.
Loading 1904.07773v6…