Source-linked AI summary
TADPOLE Challenge: Prediction of Longitudinal Evolution in Alzheimer's Disease
Razvan V. Marinescu, Neil P. Oxtoby, Alexandra L. Young, Esther E. Bron, Arthur W. Toga, Michael W. Weiner, Frederik Barkhof, Nick C. Fox, Stefan Klein, Daniel C. Alexander, the EuroPOND Consortium
TL;DR
The paper addresses the limited unbiased evaluation of algorithms predicting future Alzheimer’s disease progression from multimodal data. It describes TADPOLE, a challenge in which participants train on historical data and submit forecasts for outcomes later measured in ADNI participants. The design supports post-deadline comparison of clinical diagnosis, ADAS-Cog13, and ventricle volume forecasts, with evaluation contingent on sufficient future ADNI3 rollover data.
Problem
Few Alzheimer’s progression models had been tested on genuinely unseen future data, while studies used differing datasets and processing pipelines.
Method
TADPOLE standardizes training, prediction, and test datasets while allowing participants to use historical ADNI or custom data for probabilistic forecasts of three longitudinal outcomes.
Results
TADPOLE defines an unbiased post-deadline comparison of forecasts for clinical diagnosis, ADAS-Cog13, and ventricle volume.
Takeaways & Limitations
Identifying predictive features and algorithms may aid clinical-trial cohort refinement, endpoint assessment, and prognostic information in clinical settings.
Takeaways & Limitations
Evaluation may be delayed until sufficient ADNI3 rollover data are available.
Abstract
from arXiv · showhide
The Alzheimer's Disease Prediction Of Longitudinal Evolution (TADPOLE) Challenge compares the performance of algorithms at predicting future evolution of individuals at risk of Alzheimer's disease. TADPOLE Challenge participants train their models and algorithms on historical data from the Alzheimer's Disease Neuroimaging Initiative (ADNI) study or any other datasets to which they have access. Participants are then required to make monthly forecasts over a period of 5 years from January 2018, of three key outcomes for ADNI-3 rollover participants: clinical diagnosis, Alzheimer's Disease Assessment Scale Cognitive Subdomain (ADAS-Cog13), and total volume of the ventricles. These individual forecasts are later compared with the corresponding future measurements in ADNI-3 (obtained after the TADPOLE submission deadline). The first submission phase of TADPOLE was open for prize-eligible submissions between 15 June and 15 November 2017. The submission system remains open via the website: https://tadpole.grand-challenge.org, although since 15 November 2017 submissions are not eligible for the first round of prizes. This paper describes the design of the TADPOLE Challenge.
1. Introduction
Alzheimer’s disease creates major healthcare and treatment challenges, while existing prediction models have rarely been evaluated on genuinely future data. TADPOLE addresses this gap through an unbiased community comparison of multimodal approaches forecasting disease progression over five years.
- 47 million people worldwide had dementia in 2015, with Alzheimer’s disease the most common cause and dementia costing $818 billion globally.
- Quantitative biomarkers from MRI, PET, and CSF can support earlier diagnosis because some become abnormal decades before symptom onset.
- Existing approaches predict clinical, cognitive, and imaging outcomes using regression or disease-progression models based on multimodal biomarker data.
- Few proposed models have been tested on truly unseen future data, and studies use different datasets and processing pipelines.
- TADPOLE compares data, features, and approaches for forecasting clinical status, cognitive decline, and brain atrophy over a five-year timescale.
2. Competition Design
TADPOLE uses historical individual-level ADNI measurements to forecast future outcomes, with evaluation performed after the relevant measurements have been acquired.
- Participants use each individual’s historical ADNI measurements, including imaging, psychology, demographics, and genetics, to forecast future outcomes.
- Forecasts are submitted before a deadline and evaluated afterward against measurements acquired from the same individuals.
3. Forecasts
Participants make probabilistic, month-by-month forecasts for diagnosis, cognition, and ventricular volume because future acquisition times are unknown.
- Each individual receives monthly forecasts for clinical diagnosis, ADAS-Cog13 score, and ventricle volume divided by intracranial volume.
- Clinical-status forecasts assign relative likelihoods to cognitively normal, mild cognitive impairment, and probable Alzheimer’s disease.
- ADAS-Cog13 and ventricle-volume forecasts provide a best-guess value and a 50% confidence interval for each individual.
4. Data
TADPOLE supplies standardized ADNI-derived data while allowing custom training and prediction sets, using multimodal clinical, imaging, cognitive, and genetic measurements. The challenge design separates training, prediction, and post-deadline test data.
- Participants receive a standard ADNI-derived training dataset but may add ADNI or other-study data and use custom training sets.
- TADPOLE distinguishes training data with outcomes, prediction data without outcomes, and test data used for post-deadline forecast evaluation.
- The challenge design trains models on D1 and/or other datasets, forecasts D2 and D3, and evaluates predictions on subsequently acquired D4 data.
- ADNI data include CSF amyloid-beta and tau, MRI, PET, diffusion tensor imaging, cognitive assessments, and APOE4 status.
- MRI and PET data undergo standardized preprocessing, with regional imaging features extracted using Freesurfer pipelines.
5. TADPOLE Datasets
TADPOLE organizes forecasting around training, prediction, and future test datasets, with longitudinal and cross-sectional prediction options. The estimated D4 test set contains about 330 subjects and is intended to support reliable evaluation.
- TADPOLE defines training data with associated outcomes, prediction data with baseline measurements but no outcomes, and test data containing outcomes for forecast evaluation.
- Standard datasets: D1 contains longitudinal measurements from ADNI1, ADNI GO, and ADNI2 for individuals observed at least twice.
- Standard datasets: D2 provides all available time points for ADNI rollover individuals, whereas D3 provides one recent time point and fewer variables for each individual.
- Standard datasets: D3 is expected to yield worse forecasts than D2 but reflects information typically available when selecting a clinical-trial cohort.
- Test dataset: Forecasts are evaluated against D4 outcomes acquired after the submission deadline, and participants may use additional training datasets.
- Test dataset: D4 is an estimated ADNI3 rollover test set of around 330 subjects with one visit per subject, judged sufficient for reliable submission evaluation.
6. Submissions
TADPOLE offers simple and full submission routes with different forecast-completeness and participation requirements. Submission limits and configurable datasets support comparisons of training data, preprocessing, and modeling choices.
- Entry types: A simple entry requires a minimal forecast and methods description, while a full entry requires complete predictions for all three outcomes and all D2 subjects.
- Submission limits: Each participant may submit at most three predictions, limiting opportunities to tune methods against the test set.
- Comparative evaluation: Prizes are awarded regardless of training-set and prediction-set choices, while additional submissions help separate data, preprocessing, and modeling effects.
- Custom datasets: Participants may construct custom training or prediction sets, including alternative features from the same D2 and D3 individuals.
7. Forecast Evaluation
TADPOLE evaluates clinical-status forecasts with multiclass discrimination and balanced accuracy, while ADAS13 and ventricle-volume forecasts are assessed using accuracy, confidence-weighted error, and confidence-interval coverage.
- Clinical status: Clinical-status forecasts are evaluated with multiclass AUC and balanced classification accuracy.mAUC uses probabilistic class likelihoods, whereas BCA uses hard assignments to CN, MCI, or AD.
- Multiclass AUC: mAUC averages pairwise class AUCs after symmetrizing the AUC for each class pair.The class probabilities are pCN, pMCI, and pAD, normalized to sum to one.
- Balanced classification accuracy: BCA averages class-specific balanced accuracies based on true-positive, false-positive, true-negative, and false-negative counts.The measure accounts for class imbalance but requires assigning each point to the class with the highest likelihood.
- Continuous outcomes: ADAS13 and ventricle-volume forecasts use MAE, WES, and CPA to assess point accuracy, confidence-weighted error, and confidence-estimate accuracy.MAE ignores confidence intervals; WES incorporates confidence estimates; CPA evaluates confidence estimates independently of best-guess accuracy.
- Coverage probability accuracy: CPA compares actual and nominal confidence-interval coverage, with TADPOLE setting nominal coverage to 0.5 and lower scores preferred.Actual coverage is the proportion of measurements falling within the corresponding confidence interval, and CPA ranges from 0 to 1.
8. Prizes
TADPOLE allocated a £30,000 prize fund across six categories, covering individual outcome measures, overall performance, and student teams.
- £30,000 was sponsored by Alzheimer’s Research UK, The Alzheimer’s Society, and The Alzheimer’s Association.
- Six prizes covered clinical status, ADAS13, ventricle volume, overall performance, and university and high-school teams.
- The overall best team was defined as achieving the lowest sum of ranks across clinical status, ADAS13, and ventricle volume.
9. Discussion
The discussion presents TADPOLE as a transparent challenge for identifying predictive algorithms and features, with potential relevance to clinical-trial cohort design and endpoint assessment.
- TADPOLE aims to identify algorithms and features that best predict Alzheimer’s disease evolution using three key outcomes.
- Participants use historical ADNI data to predict clinical diagnosis, ADAS-Cog13, and ventricle volume.
- The challenge may aid clinical-trial cohort refinement, endpoint assessment, and prognostic information in clinical settings.
- Open scripts, a public forum, and a live leaderboard were created to make the challenge transparent and accessible.
- Forecast evaluation was expected after sufficient ADNI3 rollover data became available, with publication planned after the initial results.
Appendix A. Expected number of subjects and available data for D4
Appendix A estimates the expected D4 subject population and available biomarker data using prior ADNI rollover patterns and observed acquisition proportions.
- D4 subject numbers were estimated from the ADNI1-to-ADNI2 dropout rate and the total number of D2 subjects.
- Diagnostic-category proportions were estimated using diagnostic rates observed in D2 and prior rollover information.
- Available biomarker data were estimated from the proportion observed during the first year of ADNI2, from July 2012 to July 2013.
- AV1451 availability was estimated using AV45 because AV1451 was introduced later and more subjects were expected to undergo those scans in ADNI3.
- A Python script for computing the Table 2 estimates is provided in the TADPOLE repository.