Source-linked AI summary
Reproducible evaluation of classification methods in Alzheimer's disease: framework and application to MRI and PET data
Jorge Samper-González, Ninon Burgos, Simona Bottani, Sabrina Fontanella, Pascal Lu, Arnaud Marcoux, Alexandre Routier, Jérémy Guillon, Michael Bacci, Junhao Wen, Anne Bertrand, Hugo Bertin, Marie-Odile Habert, Stanley Durrleman, Theodoros Evgeniou, Olivier Colliot
TL;DR
AD classification studies are difficult to reproduce and compare because participant selection, preprocessing, and validation procedures are often unavailable or inconsistent. The paper introduces a reproducible, modular framework built around three public datasets and evaluates MRI and PET classification components at scale. FDG PET outperformed T1 MRI across classification tasks, while several preprocessing and feature choices showed no performance difference and ADNI-trained classifiers generalized well to AIBL and OASIS.
Problem
AD classification methods are difficult to reproduce and compare because participant selection, input data, preprocessing, and cross-validation procedures are often unavailable.
Method
The paper develops an open framework that converts ADNI, AIBL, and OASIS into BIDS and combines modular preprocessing, feature extraction, classification, and evaluation pipelines.
Results
FDG PET outperformed T1 MRI across classification tasks; linear SVM and L2-logistic regression performed similarly and both outperformed random forests.
Takeaways & Limitations
The framework provides reproducible baselines for isolating and benchmarking components of AD classification methods across public datasets.
Takeaways & Limitations
The evaluation did not assess all potentially relevant components, including anatomical feature types, feature selection, and intracranial-volume normalization.
Abstract
from arXiv · showhide
A large number of papers have introduced novel machine learning and feature extraction methods for automatic classification of AD. However, they are difficult to reproduce because key components of the validation are often not readily available. These components include selected participants and input data, image preprocessing and cross-validation procedures. The performance of the different approaches is also difficult to compare objectively. In particular, it is often difficult to assess which part of the method provides a real improvement, if any. We propose a framework for reproducible and objective classification experiments in AD using three publicly available datasets (ADNI, AIBL and OASIS). The framework comprises: i) automatic conversion of the three datasets into BIDS format, ii) a modular set of preprocessing pipelines, feature extraction and classification methods, together with an evaluation framework, that provide a baseline for benchmarking the different components. We demonstrate the use of the framework for a large-scale evaluation on 1960 participants using T1 MRI and FDG PET data. In this evaluation, we assess the influence of different modalities, preprocessing, feature types, classifiers, training set sizes and datasets. Performances were in line with the state-of-the-art. FDG PET outperformed T1 MRI for all classification tasks. No difference in performance was found for the use of different atlases, image smoothing, partial volume correction of FDG PET images, or feature type. Linear SVM and L2-logistic regression resulted in similar performance and both outperformed random forests. The classification performance increased along with the number of subjects used for training. Classifiers trained on ADNI generalized well to AIBL and OASIS. All the code of the framework and the experiments is publicly available at: https://gitlab.icm-institute.org/aramislab/AD-ML.
1. Introduction
Existing AD classification studies are difficult to reproduce and compare because participant selection, data preparation, preprocessing, and validation details vary or remain unavailable. The paper proposes an open, modular framework using three public datasets to support reproducible benchmarking of these components.
- Validation and comparison require many longitudinal patients, yet published studies often use differing datasets, modalities, and methods.
- Earlier comparisons enabled benchmarking on shared data and preprocessing but provided static evaluations that were difficult to update with new subjects or methods.
- The framework evaluates AD classification with MRI and PET data from ADNI, AIBL, and OASIS.
- Automatic dataset conversion to BIDS supports management and continuous updating as new subjects become available.
- Modular preprocessing, feature-extraction, classification, and evaluation components provide baselines for benchmarking individual methods.
- The experiments assess modality, feature type, preprocessing, diagnostic criteria, classification algorithm, and cross-dataset generalization.
2. Materials
The study combines ADNI, AIBL, and OASIS data with defined diagnostic subsets and participant summaries. ADNI includes T1w MRI, FDG PET, and amyloid-status subsets, while AIBL and OASIS provide additional cohorts.
- ADNI was established as a public-private study combining serial MRI, PET, biological, clinical, and neuropsychological measures to study MCI and early AD.
- AIBL enrolled 1,100 participants and collected more than 4.5 years of longitudinal data, including AD, MCI, and healthy-control groups.
- OASIS contributed 416 cross-sectional subjects aged 18–96, including 100 participants over 60 clinically diagnosed with very mild to moderate AD.
- ADNI: The ADNI subsets distinguish participants by available T1w MRI, FDG PET, and known amyloid status.
- ADNI: ADNI diagnostic groups include CN, AD, MCI, progressive MCI, and stable MCI, with progression defined over at least 36 months for pMCI and sMCI.
- AIBL and OASIS: AIBL included 608 participants with baseline T1-weighted MRI, whereas the analyzed OASIS cohort included 193 participants aged at least 61 years.
2.3. Imaging data
The imaging data comprise T1-weighted MRI and FDG PET acquisitions from the three datasets, with dataset-specific acquisition protocols and available preprocessing stages.
- T1w MRI: ADNI T1w images were available either as acquired or after corrections for geometry distortion and intensity non-uniformity.
- FDG PET: ADNI FDG PET used six five-minute frames in ADNI 1 or four five-minute frames in ADNI GO/2, acquired 30–60 minutes post-injection.
- FDG PET: ADNI FDG PET downloads included stages such as frame averaging, spatial alignment, standard voxel interpolation, and 8 mm smoothing.
- AIBL: AIBL T1w MRI used the ADNI 3D T1w sequence with 1 × 1 mm in-plane resolution and 1.2 mm slice thickness.
- OASIS: OASIS subjects had three or four T1w images acquired in one session, with 1 × 1 mm in-plane resolution and 1.25 mm slice thickness.
3. Methods
The framework unifies data management, image preprocessing, feature extraction, classification, and evaluation through modular tools integrated into Clinica.
- Unified tools cover data management, image preprocessing, feature extraction, classification, and evaluation.
- Dataset conversion tools support updates as new subjects become available.
- Modular pipelines use Nipype for processing and scikit-learn for classification and evaluation.
3.1. Converting datasets to a standardized data structure
The framework organizes heterogeneous ADNI, AIBL, and OASIS data into BIDS format to make participant selection and imaging data management more structured and reproducible.
- BIDS conversion addresses unstructured imaging and clinical data, multiple acquisitions per visit, and complementary information spread across numerous CSV files.
- ADNI conversion: The ADNI converter derives subject-session lists from ADNIMERGE and compares them with available imaging data without requiring user intervention.
- ADNI conversion: ADNI image selection prioritizes quality-controlled scans, corrected images when available, 1.5 T images for ADNI 1, and co-registered averaged FDG PET scans.
- AIBL conversion: The AIBL converter uses modality-specific scan lists and converts the single available T1w image per participant session.
- OASIS conversion: The OASIS converter identifies subjects from downloaded folders, converts selected imaging files to NIfTI, creates the BIDS hierarchy, and converts clinical data.
3.2. Preprocessing pipelines
Two modular Nipype-based pipelines preprocess T1w MRI and PET images using established tools, with PET processing linked to corresponding anatomical MRI data.
- Separate pipelines preprocess anatomical T1w MRI and PET images, and their modular design allows components to be connected or replaced.
- T1w MRI: T1w MRI preprocessing uses SPM12 Unified Segmentation for tissue segmentation, bias correction, and spatial normalization.
- T1w MRI: The MRI pipeline then creates a group template with DARTEL from subjects’ tissue probability maps and derives subject deformation fields.
- PET: PET preprocessing registers each PET image to its corresponding native-space T1w image and optionally applies regional voxel-based partial volume correction.
- PET: The resulting PET SUVR images are masked and represented in common MNI space for voxel-wise correspondence across subjects.
3.3. Feature extraction
The study extracts voxel-level and region-level imaging features after preprocessing, using regional averages from multiple atlases with different anatomical detail.
- Two feature types are extracted: voxel features containing all brain voxels and regional features summarizing signals within regions of interest.
- Voxel features: For T1w MRI, voxel signals represent gray matter density, whereas FDG PET voxel signals represent SUVR in MNI space.
- Region features: Regional features are average gray matter density or SUVR values computed within atlas-defined regions of interest.
- Atlases: The selected atlases include cortical and subcortical regions covering brain areas affected by AD.
- Atlases: Atlas anatomical detail varies substantially, with LPBA40, Hammers, and Neuromorphometrics containing 56, 69, and 140 regions, respectively.
3.4. Classification models
The framework compares linear SVM, L2-regularized logistic regression, and random forest classifiers, applying each where computational requirements permit.
- Three classifiers are evaluated: linear SVM, L2-regularized logistic regression, and random forest.
- Linear SVM: Linear SVM is applied to voxel and regional features because its dual-form computational complexity depends on the number of subjects.
- Computational constraints: Logistic regression with L2 regularization and random forest are restricted to regional analyses because their feature-dependent complexity makes voxel-based training infeasible.
- Model outputs: Feature weights are retained for each classification task, enabling representations of feature importance across brain voxels or regions.
- Model optimization: The SVM optimizes the penalty parameter C using a precomputed linear-kernel Gram matrix, while logistic regression also optimizes C.
- Random forest: Random forest optimization selects the number of trees and the number of features considered for each split, and it is used only with regional features.
3.5. Evaluation strategy
The evaluation uses nested cross-validation to separate performance estimation from hyperparameter optimization, with repeated stratified random splits as the primary outer procedure. It reports multiple performance metrics and cautions that empirical cross-validation variance underestimates true variance.
- Nested cross-validation used an outer loop for performance evaluation and an inner loop for model hyperparameter optimization.
- 250 repeated stratified random-split iterations were used for each experiment, with full metric distributions, means, and empirical standard deviations reported.Repeated random splits were preferred when computationally feasible because they provide more stable performance estimates.
- Empirical cross-validation variance was acknowledged to substantially underestimate the true variance, and classifier differences were not statistically tested.The authors note that no unbiased estimate of cross-validation variance exists and that there is no universal solution for statistical testing of classifier performance.
- Inner 10-fold cross-validation selected models by balanced accuracy and averaged the selected models across splits.Model averaging was intended to have a stabilizing effect.
- Outputs included balanced accuracy, AUC, accuracy, sensitivity, specificity, and per-subject predicted classes.The predicted classes allow users to calculate additional metrics.
3.6. Classification experiments
The experiments systematically vary datasets, modalities, feature representations, preprocessing parameters, classifiers, and diagnostic tasks. They cover clinical classification and predictive tasks while accounting for dataset-specific data availability.
- Classification tasks covered clinical diagnosis and prediction of MCI progression, with task availability differing across ADNI, AIBL, and OASIS.MCI-to-AD progression tasks were not performed for AIBL because its sMCI and pMCI groups were small.
- Voxel experiments used linear SVMs and Gaussian smoothing levels from 0 to 12 mm FWHM.
- Region-based experiments tested linear SVM, L2-regularized logistic regression, and random forest classifiers with features from five atlases.The atlases were AAL2, AICHA, Hammers, LPBA40, and Neuromorphometrics.
- T1w MRI and FDG PET were available for ADNI, whereas AIBL and OASIS provided T1w MRI only.Both voxel and regional features were extracted for each available modality; FDG PET features were generally not PVC-corrected.
- The experiment matrix crossed dataset, imaging modality, feature type and task, with task definitions listed in Table 6 and classifier parameters in Table 7.Table 8 summarizes the complete set of classification experiments.
4. Results
The results compare preprocessing, feature, classifier, dataset, and diagnostic choices across the evaluation framework. FDG PET and some classifier and dataset choices performed better, while several preprocessing choices showed little effect.
- Influence of atlas: No atlas consistently produced the highest accuracy across tasks and datasets.Different atlases led on different tasks, while AAL2 was highest for CN vs pMCI and sMCI vs pMCI in the reported ADNI comparisons.
- Influence of smoothing: Balanced accuracy generally changed little across smoothing levels, with variations mainly in CN vs pMCI and sMCI vs pMCI tasks.The comparison used voxel features and Gaussian kernels ranging from no smoothing to 12 mm FWHM.
- Influence of feature type: Voxel and regional features produced no notable differences in mean balanced accuracy in ADNI, while AIBL region-based classification reached 0.86 [±0.042] versus 0.79 [±0.059] for voxel-based classification.The AIBL standard deviations were high, qualifying the apparent difference.
- Influence of classification method: Linear SVM and L2-regularized logistic regression had similar balanced accuracies and consistently exceeded random forest across tested tasks and modalities.
- Influence of partial volume correction: Partial volume correction produced little balanced-accuracy difference, with nearly identical voxel results and a very small regional increase without correction.
- Influence of the dataset: ADNI-trained classifiers generalized well to AIBL and OASIS, and training on ADNI substantially improved OASIS performance.With equal-sized subsets, the improvement disappeared for voxel features but remained for regional features.
- Influence of training set size: Balanced accuracy increased slightly as more subjects were used for training, while combining ADNI and AIBL offered little advantage over ADNI alone at equivalent sample size.The regional-feature difference was largely within the standard deviation.
- Influence of diagnostic criteria: Amyloid-refined diagnostic tasks achieved higher or equal mean balanced accuracy despite using fewer subjects.
5. Discussion
The framework targets more reproducible and objective AD machine-learning evaluation by standardizing datasets, pipelines, features, classifiers, and validation. Its large-scale application found stronger performance for FDG PET than T1 MRI, no systematic effect from several preprocessing and feature choices, and benefits from larger and cross-dataset training.
- Framework and evaluation: The open-source framework normalizes three public datasets into BIDS and provides standardized preprocessing, feature extraction, classification, and cross-validation components.These components are intended to support reproducible experiments and baseline comparisons of individual method components.
- Framework and evaluation: The framework enables more objective comparisons by allowing researchers to replace one pipeline component while retaining baseline preprocessing, features, classifiers, and validation.Its validation includes repeated random splits, full accuracy distributions, standard deviations, and nested cross-validation.
- Modalities and components: FDG PET consistently outperformed T1-weighted MRI across tasks, feature types, and classification methods.The authors suggest this may reflect earlier detection of hypometabolism than atrophy, while presenting the performance difference as a robust finding in their evaluation.
- Modalities and components: No systematic performance effect was found for atlas choice, smoothing, PET partial-volume correction, or regional versus voxel-based features.The study used more subjects and stricter validation than some earlier reports that found effects from atlas or region selection.
- Classifiers: Linear SVM and L2-regularized logistic regression were more suitable than random forests for single-modality imaging classification in this evaluation.The authors relate this pattern to dependence and relatively homogeneous values across brain-imaging features, which favor smooth feature combinations.
- Data and training: Very strong class imbalance reduced voxel-based performance, whereas moderate imbalance was adequately handled and regional features showed no influence from imbalance.The authors suggest balanced training groups when imbalance is very strong and features are highly dimensional.
- Data and training: Classifiers trained on ADNI generalized well to AIBL and OASIS, while OASIS-only performance was lower and improved substantially when training on ADNI.The authors associate these differences with training-set size, image quality, and diagnostic criteria; equal-size subsets removed the voxel-feature improvement.
- Data and training: Classification performance increased with training-set size, and the available sample size had not yet reached the point where performance stopped improving.Combining OASIS with ADNI and AIBL lowered performance relative to combining ADNI and AIBL, consistent with lower OASIS performance.
6. Conclusions
The framework addresses comparability and reproducibility issues in machine learning-based AD classification. Its application to T1w MRI and FDG PET data enables extensive assessment of modalities, preprocessing, features, and algorithms while providing baseline performances for comparison.
- The framework targets current comparability and reproducibility issues in machine learning-based AD classification.
- Applying the framework to T1w MRI and FDG PET data enabled extensive assessment of imaging modality, preprocessing options, features, and algorithms.
- The resulting performances provide a baseline against which other approaches can be compared.