Source-linked AI summary
Detecting cognitive decline using speech only: The ADReSSo Challenge
Saturnino Luz, Fasih Haider, Sofia de la Fuente, Davida Fromm, Brian MacWhinney
TL;DR
The ADReSSo Challenge addresses speech-only prediction of Alzheimer’s Dementia, cognitive testing scores, and cognitive decline, motivated by the need for scalable dementia assessment. The paper describes standardized datasets and baseline models using acoustic and linguistic information extracted from audio, reporting results across all three tasks.
Problem
The paper addresses the need for cost-effective, scalable speech-based detection of Alzheimer’s Dementia and prediction of disease progression, amid limited standardization for benchmarking.
Method
The paper presents standardized datasets, three prediction tasks, and baseline models using acoustic features and linguistic features from automatic transcripts without manual transcription.
Results
Baseline models achieved 78.87% accuracy for AD classification, 5.28 RMSE for MMSE prediction, and 66.67% maximum F1 for cognitive-decline prediction.
Takeaways & Limitations
ADReSSo advances speech-based cognitive-status prediction toward clinical applications by evaluating raw, non-annotated, nontranscribed speech and longitudinal cognitive change.
Abstract
from arXiv · showhide
Building on the success of the ADReSS Challenge at Interspeech 2020, which attracted the participation of 34 teams from across the world, the ADReSSo Challenge targets three difficult automatic prediction problems of societal and medical relevance, namely: detection of Alzheimer's Dementia, inference of cognitive testing scores, and prediction of cognitive decline. This paper presents these prediction tasks in detail, describes the datasets used, and reports the results of the baseline classification and regression models we developed for each task. A combination of acoustic and linguistic features extracted directly from audio recordings, without human intervention, yielded a baseline accuracy of 78.87% for the AD classification task, an MMSE prediction root mean squared (RMSE) error of 5.28, and 68.75% accuracy for the cognitive decline prediction task.
1. Introduction
The ADReSSo Challenge addresses the need for scalable speech-based detection and monitoring of dementia-related cognitive decline through three standardized prediction tasks. It introduces datasets and baseline models that operate directly from speech, without manual transcription.
- Dementia involves a long-term, usually gradual decline in cognitive functioning, creating a need for early detection and progression monitoring.The paper emphasizes cost-effective and scalable methods for Alzheimer’s Dementia detection and disease progression prediction.
- Speech can be collected passively, naturally, and continuously, while machine learning enables automatic cognitive screening.This contrasts with conventional cognitive tests that are cheap and quick but require active administration and scoring.
- The lack of standardized datasets and tasks has hindered benchmarking and translation of speech-based methods into clinical practice.
- ADReSSo requires models to predict cognitive status and decline directly from spontaneous speech without manual transcription, although automatic transcription is allowed.The challenge uses a new standardized dataset and targets speech-only modeling rather than features extracted from human-provided transcripts.
- The challenge defines AD classification, MMSE score regression, and cognitive decline inference as its three prediction tasks.The tasks use acoustic features, automatically generated transcripts and linguistic features, or baseline speech for longitudinal prediction.
- The paper describes the ADReSSo datasets and presents baseline feature-extraction procedures and models for all three tasks.
2. Related work
Prior speech-based dementia research has concentrated on Alzheimer’s classification, with fewer studies addressing MMSE regression and disease progression. Reported approaches vary in feature types, datasets, and performance measures.
- Most prior speech-technology research has focused on Alzheimer’s Dementia classification, while some studies target mild cognitive impairment detection.
- 85.45% and 89.6% accuracy were achieved in earlier ADReSS classification models using acoustic features and text-based features from manual transcripts.
- Few studies rely exclusively on acoustic features or text extracted through automatic speech recognition, with reported classification accuracies ranging from 62.3% to 78.7% in cited datasets.
- Research on MMSE regression is smaller and more recent, including reported mean absolute errors of 3.42 and 3.1.
- Studies addressing cognitive progression are comparatively rare and include both MMSE prediction across visits and classification of intra-subject cognitive change.
3. The ADReSSo Datasets
ADReSSo provides separate diagnostic and prognostic speech datasets for Alzheimer’s classification, MMSE prediction, and two-year cognitive-decline prediction. The diagnostic data are age- and gender-matched, while the prognostic data are longitudinal and use baseline speech.
- The challenge provides a semantic-fluency speech dataset for predicting cognitive decline over two years and a picture-description dataset for cognitively normal and AD participants.
- No transcripts were provided for either dataset, but optional vocalisation-sequence segmentations with speaker identifiers were available.Recordings also included experimenter instructions and occasional short dialogues, and audio was acoustically enhanced and volume-normalized.
- The diagnostic dataset was matched on age and gender using propensity scores to reduce bias in AD and MMSE prediction.The reported standardized mean differences indicate adequate matching for the specified covariates and interactions.
- 242 audio files comprised the resulting diagnostic dataset, split into 70% training and 30% test sets while preserving age and gender matching.
- Table 1 reports characteristics of patients in the diagnostic tasks dataset.
- Cognitive decline was defined as a baseline-to-year-2 MMSE decrease of at least 5 points for patients assessed in a longitudinal cohort.The prognostic task classifies patients as decline or no-decline using baseline speech from a category-fluency test.
4. Data representation
The baseline representation combines automatically extracted acoustic features with linguistic features derived from automatic speech recognition transcripts. Acoustic processing uses frame-level eGeMAPS features, while linguistic processing uses CHAT, MOR, and CLAN-compatible analyses.
- eGeMAPS acoustic features were extracted from non-overlapping 100 ms audio windows.The feature set was designed as a compact collection of voice-related acoustic measures.
- The acoustic feature-extraction method operates automatically without supplied speech segmentation or diarisation information.
- Google Cloud Speech Recognizer produced automatic transcripts that were converted into CHAT format for linguistic analysis.
- The transcripts were morphologically annotated with MOR and analyzed using EVAL and related CLAN commands to produce linguistic measures.EVAL creates a composite profile containing 34 measures.
5. Diagnosis baseline
The diagnosis baseline evaluates acoustic, linguistic, and fused models for AD-versus-control classification and MMSE score regression. Decision-tree models performed best in cross-validation, while test-set results favored linguistic features and fusion for classification.
- Five classifiers were evaluated for AD-versus-control classification, including DT, KNN, LDA, TB, and SVM.
- 78.92% and 72.89% accuracy were achieved by DT with acoustic and linguistic features, respectively, in cross-validation.
- 78.87% test-set accuracy resulted from late fusion of acoustic and linguistic models.
- Five regression model types were used for MMSE prediction, including linear regression, DT, SVR, RF, and GP.
- 5.28 minimum RMSE was obtained on the test set using linguistic features in an SVR model.
- 5.29 RMSE was obtained after weighted fusion of acoustic and linguistic test results, compared with 6.37 on validation data.
6. Prognosis baseline
The prognosis baseline predicts whether patients develop cognitive decline within two years using acoustic and linguistic speech features. Performance was limited and decision fusion did not improve the best individual-model results.
- 6. Prognosis baseline: The task identifies patients who later exhibited cognitive decline within two years of the baseline speech visit.
- 6. Prognosis baseline: Average F1 was reported because the cognitive-decline classes were imbalanced.
- 6. Prognosis baseline: Decision fusion of the best acoustic and linguistic predictions did not yield any improvement.
7. Discussion
Across the three baseline tasks, automatically extracted linguistic features contributed despite transcript errors, while fusion benefits were task- and evaluation-dependent. Cognitive-decline prediction was the most difficult task and showed substantial variation between validation and test results.
- 78.87% maximum test-set accuracy was obtained for AD classification by fusing linguistic and acoustic models.
- Linguistic features contributed considerably to AD predictions despite relatively high ASR transcript word error rates.
- Late fusion improved MMSE RMSE in cross-validation, but the test-set RMSE remained practically unchanged.
- The prognosis task was the most difficult, with test-set F1 reaching a maximum of 66.67% even after prediction fusion.
- Acoustic features reached F1 = 66.05% versus 59.06% for linguistic features in cross-validation, but this pattern did not persist on the test set.
- The prognosis cross-validation results varied considerably, with linguistic models showing a standard deviation of 11.64.
8. Conclusions
The challenge extends speech-based cognitive assessment to raw, non-annotated, nontranscribed speech and explicitly addresses changes in cognition over time. These additions broaden relevance beyond status monitoring toward clinical contexts concerned with progression.
- The challenge targets cognitive status prediction from raw, non-annotated, nontranscribed speech.
- It adds prediction of changes in cognition over time to speech-based assessment.
- AD classification and MMSE regression provide limited insight into disease progression, a limitation shared with most dementia-focused approaches.
- Automatic cognitive status monitoring remains relevant when combined with wearable and ambient technology.
- The progression task may extend relevance to more traditional clinical contexts.