Source-linked AI summary
Alzheimer's Dementia Recognition through Spontaneous Speech: The ADReSS Challenge
Saturnino Luz, Fasih Haider, Sofia de la Fuente, Davida Fromm, Brian MacWhinney
TL;DR
Existing AD speech studies lacked balanced, standardised datasets for systematic comparison. The paper describes the ADReSS shared benchmark, its spontaneous-speech classification and MMSE regression tasks, and simple baselines; test accuracy reached 0.625 for acoustic and 0.75 for linguistic classification, while MMSE RMSE was 6.14 and 5.20 respectively.
Problem
Existing speech-and-language studies used different, often unbalanced and acoustically varied datasets, limiting reproducibility, replicability, and comparability.
Method
The paper presents the ADReSS shared benchmark, using acoustically pre-processed, age- and gender-balanced spontaneous-speech data for AD classification and MMSE-score regression, with baseline models.
Results
0.625 was the acoustic classification baseline and 0.75 the linguistic baseline; MMSE regression baselines were RMSE 6.14 for acoustic and 5.20 for linguistic features.
Takeaways & Limitations
ADReSS provides a shared platform for methodological comparison of spontaneous-speech approaches to AD recognition and MMSE prediction.
Abstract
from arXiv · showhide
The ADReSS Challenge at INTERSPEECH 2020 defines a shared task through which different approaches to the automated recognition of Alzheimer's dementia based on spontaneous speech can be compared. ADReSS provides researchers with a benchmark speech dataset which has been acoustically pre-processed and balanced in terms of age and gender, defining two cognitive assessment tasks, namely: the Alzheimer's speech classification task and the neuropsychological score regression task. In the Alzheimer's speech classification task, ADReSS challenge participants create models for classifying speech as dementia or healthy control speech. In the the neuropsychological score regression task, participants create models to predict mini-mental state examination scores. This paper describes the ADReSS Challenge in detail and presents a baseline for both tasks, including feature extraction procedures and results for classification and regression models. ADReSS aims to provide the speech and language Alzheimer's research community with a platform for comprehensive methodological comparisons. This will hopefully contribute to addressing the lack of standardisation that currently affects the field and shed light on avenues for future research and clinical applicability.
1. Introduction
The ADReSS Challenge addresses the lack of balanced, standardised datasets for comparing spontaneous-speech approaches to Alzheimer’s detection. It establishes a shared benchmark covering AD recognition and MMSE prediction, with baselines for both tasks.
- Speech and language features offer clinical information for AD detection, but existing studies lack balanced and standardised datasets for systematic comparison.
- ADReSS defines a shared task to compare automated AD detection approaches using spontaneous speech rather than laboratory-collected samples.
- The challenge provides a statistically balanced, acoustically enhanced benchmark with segmented recordings and detailed timestamped transcriptions.
- ADReSS includes binary AD-versus-non-AD classification and regression of participants’ Mini-Mental State Examination scores.
- The paper presents feature-extraction procedures and initial classification and regression baselines for both challenge tasks.
2. ADReSS Challenge Dataset
The ADReSS dataset uses matched participant groups and speech recordings from a standardised picture-description task. It includes acoustically segmented speech and corresponding transcripts for AD and non-AD participants.
- The dataset is matched for age and gender to minimise bias in the prediction tasks.
- Participants produced spoken descriptions of the Cookie Theft picture from the Boston Diagnostic Aphasia Exam, with transcripts annotated using CHAT.
- Voice activity detection segmented recordings using a 65 dB log-energy threshold and a maximum speech-segment duration of 10 seconds.
- The segmented dataset contains 1,955 speech segments from 78 non-AD subjects and 2,122 segments from 78 AD subjects.
3. Acoustic and Linguistic Features
The paper extracts multiple acoustic feature sets from segmented speech and language measures from CHAT transcripts. Acoustic features are filtered for correlation with segment duration before machine-learning experiments.
- Acoustic features were extracted with openSMILE v2.1 and in-house software without feature-set reduction, because the paper targets simple baselines.
- The acoustic representations include emobase, ComParE, eGeMAPS, MRCG functionals, and Minimal duration, pause, and speech-rate statistics.
- The extracted acoustic sets contained 88 eGeMAPS, 988 emobase, 6,373 ComParE, 6,912 MRCG, and 13 Minimal features per segment.
- Correlation filtering removed non-Minimal acoustic features significantly correlated with speech-segment duration when |R| > 0.2.
- CHAT transcripts were processed with CLAN EVAL to compute 34 language outcome measures, including utterance, lexical, syntactic, and duration measures.
4. AD classification task
The AD classification task evaluates binary discrimination between AD and non-AD speech using any combination of speech and transcripts. Baselines use several classifiers with segment-level predictions aggregated to subject labels.
- The task classifies speech as AD or non-AD, allowing models to use speech data, transcriptions, or both on the common dataset.
- Baseline experiments compare LDA, decision trees, 1NN, random forests, and linear-kernel SVM models.
- Classification uses a two-step procedure: segment-level labels are assigned first, then majority voting produces each subject’s class label.
- 4.1. Results: 0.574 was the best LOSO accuracy for acoustic ComParE features using 1NN, while LDA and decision trees were more stable across feature sets.
- 4.1. Results: 0.625 was the test-set baseline accuracy for acoustic features, compared with 0.75 for linguistic features requiring manual transcription.
5. MMSE prediction task
The MMSE prediction task evaluates regression models that estimate individual participants’ MMSE scores from acoustic or linguistic speech data. The paper establishes separate baselines for both feature types using several regression methods and LOSO/test evaluations.
- Task definition: MMSE prediction models estimate individual participants’ scores from acoustic or linguistic speech data.The baseline experiments report acoustic and linguistic models separately.
- Methods: Five regression methods were evaluated: decision trees, linear regression, Gaussian process regression, least-squares boosting, and support vector machines.The implementations used MATLAB’s statistics and machine learning toolbox.
- LOSO results: DT using MRCG features achieved the best LOSO RMSE of 7.28, with r = −0.759, and the best average acoustic RMSE of 6.86.Minimal and eGeMAPS features also showed promising performance, with RMSE values of 7.46 and 8.02 across models.
- System architecture: The system aggregates segment-level outputs using majority voting for classification and mean regression scores for prediction.Speech recordings are segmented with voice activity detection before feature extraction and prediction.
- Test results: The acoustic baseline on test data achieved RMSE 6.14 (r = 0.22), while the linguistic model achieved RMSE 5.20 (r = 0.57).The DT model with MRCG features was selected as the acoustic baseline.
6. Discussion
The discussion places the baselines in context by comparing them with prior DementiaBank and other speech studies. Higher prior scores often relied on manual transcripts or datasets that were not balanced or directly comparable.
- Comparison with prior work: Prior spontaneous-speech studies reported classification accuracies from 64% to 81.92%, higher than the baselines discussed here.Most of those studies used manual transcripts and unbalanced datasets.
- Comparability: The ADReSS approach is speaker independent, but comparison with an Italian episodic-storytelling dataset is limited by different elicitation and the absence of age and gender balancing.That Italian study reported 80% accuracy using acoustic, demographic, and educational features.
- MMSE comparison: A prior MMSE study reported MAE 3.83 using manually derived lexicosyntactic and semantic features on the unbalanced, longitudinal DementiaBank Pitt dataset.The comparison differs from ADReSS because its linguistic features came from manual transcription rather than automatically extracted acoustic features.
7. Conclusions
The paper presents ADReSS as a shared benchmark with simple baselines for spontaneous-speech AD classification and MMSE prediction. Acoustic and linguistic speech features both support baseline performance, with linguistic features performing better in the reported test results.
- Contribution: ADReSS establishes a shared challenge and simple baselines for AD classification and MMSE prediction.The challenge is intended to support methodological comparisons on a common dataset.
- Classification: Acoustic features reached up to 62.5% accuracy, while linguistic features from manual transcripts reached 76.85% on AD classification.Both results exceeded the 50% chance level.
- Regression: The MMSE regression baselines were RMSE 6.14 for acoustic features and 5.21 for linguistic features on test data.These values define the reported challenge baselines for the regression task.
- Future work: The authors expect further preprocessing, feature reduction, and more complex models to improve challenge accuracy scores.ADReSS is intended to bring researchers together for comprehensive comparisons and to highlight research caveats and future directions.