Source-linked AI summary
Cleaner Speech, Weaker Generalization: Revisiting Pitt-Derived Benchmarks for Alzheimer's Disease Detection
Luqi Sun, Shreeram Suresh Chandra, Lin Zhang, You-Jin Li, Brian MacWhinney, Yu Tsao, Emily Mower Provost, Berrak Sisman
TL;DR
Speech-enhanced and curated Pitt-derived benchmarks are widely used, but their effects on real-world AD detection generalization remain unclear. The paper evaluates speech quality, cross-dataset performance, matched enhancement conditions, and LALM behavior, finding that enhanced data often improves in-domain results while reducing cross-domain robustness. It concludes that preprocessing and curation substantially influence AD detection behavior, while the evidence is limited by the available English Pitt-derived benchmark ecosystem and unresolved noise robustness in Pitt-origin.
Problem
The paper addresses insufficient evidence about whether enhancement and curation of widely used Pitt-derived benchmarks improve robust AD detection or affect generalization and prediction behavior.
Method
The study evaluates Pitt-origin and derived datasets through speech-quality analysis, multiple supervised speech models, matched and mismatched enhancement tests, and recent LALMs.
Results
Speech-enhanced datasets often improve in-domain performance while reducing cross-dataset robustness, and LALMs show systematic prediction bias under enhanced conditions.
Takeaways & Limitations
Speech enhancement and dataset curation substantially influence downstream AD detection behavior, so cleaner speech datasets are not necessarily more robust for real-world detection.
Takeaways & Limitations
The study focuses on English speech and relies mainly on Pitt-origin and its derived datasets because independent publicly available English AD speech datasets are very limited.
Abstract
from arXiv · showhide
Speech-based Alzheimer's disease (AD) detection increasingly relies on speech-enhanced and curated versions of the Pitt Corpus, where speech enhancement, sample selection, and demographic balancing are often treated as beneficial preprocessing steps. However, whether these transformations improve real-world AD detection or instead affect model generalization and prediction behavior remains unclear. In this work, we revisit the role of speech preprocessing and dataset curation across widely used benchmarks for speech-based AD detection. We evaluate the speech quality of different datasets, the cross-dataset generalization of multiple deep learning models under matched and mismatched enhancement settings, and the behavior of several recent large audio-language models (LALMs). Experimental results show that across multiple supervised speech models, speech-enhanced datasets often improve in-domain performance while reducing robustness in cross-domain evaluation. Matched enhancement between training and test data alleviates, but does not eliminate, this degradation. LALMs show a similar sensitivity: enhanced datasets induce stronger class imbalance and prediction shifts than unprocessed data. These results suggest that speech preprocessing and dataset curation can substantially influence downstream AD detection behavior, indicating that ``cleaner'' speech datasets are not necessarily more reliable for real-world AD detection.
1 Introduction
Speech-based AD detection commonly uses Pitt-origin recordings and curated derivatives, but the effects of enhancement and curation on generalization remain insufficiently understood. The paper benchmarks these choices across multiple models and reports that enhanced datasets can improve in-domain performance while reducing cross-domain robustness.
- Benchmark context: Pitt-origin and its derivatives are widely used benchmarks built from the Cookie Theft picture-description task, with derivatives applying preprocessing and curation strategies.These strategies include speech enhancement, volume normalization, demographic balancing, and sample selection.
- Research gap: The paper addresses whether preprocessing and curation improve robust AD detection or instead alter model generalization and prediction behavior.The question matters because speech enhancement and dataset construction are often treated as beneficial preprocessing choices.
- Approach: The evaluation spans Pitt-origin and multiple derived datasets under in-domain, cross-domain, and matched or mismatched enhancement conditions.It includes acoustic-feature, self-supervised speech-representation, and pretrained speech-encoder models, as well as recent LALMs.
- Main findings: Speech-enhanced datasets can improve in-domain evaluation performance while frequently reducing robustness in cross-domain evaluation, even under matched enhancement settings.The reported effect extends across multiple modeling paradigms rather than being limited to one architecture.
2 Benchmark Datasets and Related Work
The benchmark landscape centers on Pitt-origin and derived datasets built from Cookie Theft recordings, differing in enhancement, balancing, selection, and sample size. Spectral and DNSMOS analyses show that denoising changes background-noise characteristics, while prior work has increasingly shifted toward deep and foundation-model approaches.
- Benchmark datasets: Pitt-origin contains four tasks, with Cookie Theft providing spontaneous speech from participants with AD and healthy controls and therefore serving as the most widely used task.The other three tasks are semantic fluency, sentence construction, and story recall, collected only from participants with AD.
- Benchmark datasets: Pitt is a denoised version of Pitt-origin with identical recordings and sample size, while ADReSS and ADReSSo add enhancement and age-gender matching.ADReSSo expands the sample size compared with ADReSS.
- Benchmark datasets: ADReSS-M uses the same speech samples and quantity as ADReSSo but does not apply speech preprocessing, whereas the independent Lu Corpus uses different populations and recording conditions.Lu also uses the Cookie Theft task and contains unprocessed spontaneous speech.
- Speech quality: Mel spectrograms show more prominent background noise in unprocessed Pitt-origin and ADReSS-M recordings than in denoised Pitt, ADReSS, and ADReSSo recordings.The denoised datasets exhibit cleaner spectral patterns and reduced background interference, indicating distributional differences introduced by preprocessing.
- Speech quality: DNSMOS results show substantially higher BAK scores for denoised datasets than for Pitt-origin and ADReSS-M, while SIG scores remain relatively similar.The three reported dimensions are speech quality (SIG), background-noise quality (BAK), and overall quality (OVRL).
- Related work: Recent AD detection research has moved from traditional machine learning baselines toward deep architectures, transformers, ASR-based systems, and large language or audio-language models.Earlier challenges used support vector machines, while later work explored random forests, paralinguistic features, pause information, and semantic features.
3 Methodology
The methodology examines how enhancement and distribution shifts affect AD detection across conventional speech models and LALMs. It combines multiple enhancement paradigms with in-domain, cross-domain, and matched-condition evaluations.
- Preprocessing and distribution shift: Speech enhancement can modify acoustic characteristics through artifacts, spectral smoothing, and signal distortion, potentially affecting downstream models.The paper also describes enhancement as a source of distribution shift from naturally recorded speech.
- Research question: The study asks whether enhancement, preprocessing, and curation improve robustness or instead reduce generalization across datasets and recording conditions.This question is motivated by unreleased enhancement procedures and the mismatch between processed benchmarks and uncontrolled clinical recordings.
- Supervised models: Three supervised architectures are evaluated: eGeMAPS acoustic features, XLS-R self-supervised speech representations, and an SLS model built on pretrained speech encoders.These models span increasing modeling complexity and represent distinct speech-modeling paradigms.
- Large audio-language models: Five large audio-language models are evaluated under multiple inference settings to assess enhancement effects beyond conventional supervised architectures.The evaluated models include Kimi-Audio, Qwen3-Omni, Qwen2-Audio, Audio Flamingo 3, and ultravox-v0_5-llama-3_2-1b.
- Enhancement evaluation: Five representative enhancement models cover waveform-based, complex-domain, Transformer-based, and semantic-conditioned enhancement paradigms.These enhancement methods are applied to the downstream AD detection architectures.
4 Speech Enhancement Impact on Deep Learning Models
Speech enhancement and curation can improve in-domain AD detection, but often weaken cross-dataset generalization. Matching enhancement between training and test data helps partially, yet unprocessed training data generally remains more robust.
- Experimental setup: The study evaluates three AD detection architectures across Pitt-origin and derived datasets using in-domain, cross-domain, and enhancement-matching conditions.The architectures are eGeMAPS, XLSR-based self-supervised speech modeling, and SLS-based modeling; Lu serves as an independent cross-domain test set.
- 4.1 Cross-Dataset Generalization: ADReSS and ADReSSo improve in-domain Macro-F1 relative to Pitt-origin, potentially because curation reduces distributional complexity and facilitates in-domain overfitting.Pitt-origin and denoised Pitt show comparable in-domain performance, with Pitt slightly better.
- 4.1 Cross-Dataset Generalization: Pitt-origin-trained models achieve the strongest Lu generalization, while denoised-data training shows clear degradation across domains.ADReSSo performs relatively strongly with SLS, but this advantage is not consistent for eGeMAPS and XLS-R.
- 4.2 Speech-Enhanced Test Sets: Enhanced-data models perform better on similarly enhanced test sets, whereas unprocessed-data models perform best on unenhanced tests; matched enhancement therefore aids alignment but remains variable.The exact preprocessing pipelines for Pitt, ADReSS, and ADReSSo are unavailable, limiting reproducibility of enhancement conditions.
- 4.3 Matched Enhancement Settings: Across five enhancement methods, matched settings outperform mismatched settings, but all enhancement-related settings remain below the raw baseline in most cases.Similar results for eGeMAPS and XLS-R indicate that enhancement-related robustness effects are not limited to one architecture.
5 Speech Enhancement Impact on Large Audio-Language Models
The paper evaluates recent LALMs for AD detection under audio-only and audio-plus-transcript settings. Enhanced datasets produce class-imbalanced prediction behavior that persists across inference conditions and is not eliminated by transcript information.
- Experimental setup: Five recent LALMs are evaluated under multiple inference settings, with Kimi-Audio selected for detailed zero-shot and two-shot analysis.Two-shot inference provides one labeled AD example and one labeled Control example without parameter updates or fine-tuning.
- 5.1 Audio-Only Settings: Most LALMs struggle in zero-shot audio-only AD-versus-Control classification, while Kimi-Audio is statistically significant across most datasets except Lu.Qwen2-Audio tends to predict Control for nearly all samples, whereas Audio Flamingo 3 tends to predict AD for nearly all samples.
- 5.1 Audio-Only Settings: On enhanced Pitt, ADReSS, and ADReSSo, Kimi-Audio shows substantially higher AD-positive than Control-positive F1 scores, unlike the more balanced unprocessed datasets.The imbalance persists under both zero-shot and two-shot settings, indicating that a small number of labeled examples does not resolve it.
- Overall findings: Across audio-only experiments, speech enhancement does not consistently improve LALM discrimination and may introduce systematic prediction bias that reduces cross-dataset robustness.The multimodal results likewise indicate substantial downstream prediction effects when textual information is available.
- 5.2 Audio + Transcript Settings: Audio-plus-transcript input preserves enhancement-induced class imbalance on denoised Pitt, while predictions on unprocessed Pitt-origin remain more balanced.Two-shot inference also fails to substantially improve this issue.
6 Conclusion
The paper systematically examines how speech preprocessing and dataset curation affect AD detection across evaluation settings and modeling paradigms. Speech-enhanced data often improve in-domain performance but reduce cross-dataset robustness, while LALMs also show preprocessing-sensitive prediction behavior.
- The study evaluates speech preprocessing and dataset curation across in-domain, cross-domain, matched, and mismatched enhancement settings.The experiments include multiple speech enhancement pipelines and diverse AD detection architectures.
- Speech-enhanced datasets often improve in-domain performance while reducing cross-dataset robustness across supervised AD detection models and LALMs.
- Even with matched enhancement applied to training and test data, models trained on unprocessed Pitt-origin speech generally achieve stronger generalization performance.
- Preprocessing sensitivity persists across acoustic-feature models, pretrained speech representation models, and large audio-language models.
- Speech enhancement and dataset curation substantially influence downstream AD detection behavior and cross-dataset generalization.The findings indicate that cleaner speech does not necessarily produce more robust AD detection systems.
7 Limitations
The study is limited by its focus on publicly available English AD speech datasets and by its emphasis on speech enhancement rather than text-based detection. Pitt-origin also contains substantial environmental noise, leaving noise robustness unresolved.
- Because few publicly available English AD speech datasets are independent of Pitt, the paper uses the English Lu Corpus for cross-domain testing.Other accessible English dementia datasets were excluded because they include MCI or other cognitive-decline groups without distinguishing diagnoses.
- The analysis focuses on speech enhancement and model generalization rather than conducting an in-depth study of text-based AD detection.Text is used only as an auxiliary diagnostic tool for LALMs.
- Pitt-origin contains substantial environmental noise despite its generally stronger generalization performance than several derived datasets.
- Robustly extracting AD-related pathological features from noisy speech remains an important unresolved problem for future work.
Use of AI Assistants
The paper reports using GPT-5.5 and Claude Opus 4.7 for code assistance and for language-related writing assistance.
- GPT-5.5 and Claude Opus 4.7 were used for code assistance.
- The same models were also used purely for language-related assistance in writing the paper.
Licensing
The supplied material describes DementiaBank licensing and the paper’s datasets, evaluation figures, model architectures, and LALM classification settings. Access and use are constrained by the DementiaBank consortium requirements and dataset-specific benchmark construction.
- Licensing: DementiaBank use is generally governed by CC BY-NC-SA 3.0, requiring attribution, restricting commercial use, and requiring share-alike for derivatives.Access is password-protected and limited to approved consortium members.
- Dataset construction: Pitt-origin and Pitt have the same size, ADReSS-M and ADReSSo have the same size, and ADReSS is smaller than both.
- Dataset characterization: The dataset comparisons include duration distributions and age-and-gender metadata for Pitt-origin, Pitt, ADReSS, ADReSSo, and ADReSS-M.
- Speech quality: Table 10 reports Pitt-origin DNSMOS scores under different enhancement methods, covering speech quality, background-noise quality, and overall speech quality.
- LALM evaluation: The LALM evaluations report audio-only and audio-plus-transcript settings, with AD+ and Control+ denoting F1 scores for each positive class.
- Model architectures: The evaluated model architectures include SLS-based, XLSR-based, and eGeMAPS-based models, alongside audio classification prompts for LALMs.The SLS model fuses weighted representations from Transformer layers of a frozen pretrained XLS-R model, while the eGeMAPS model uses 25-dimensional acoustic features.
F.1 Generalization Test On Raw Data
This section presents XLSR-based generalization tests using raw and speech-enhanced data, including same-domain and Lu-dataset evaluation. It also compares matched and mismatched enhancement settings and reports AD- and Control-positive F1 scores.
- XLSR-based models are evaluated on raw data using same-domain tests and testing on the Lu dataset.Same-domain evaluation uses the test split of the training dataset.
- Speech-enhanced data are evaluated with the same-domain protocol, with significance assessed against a binomial majority-class baseline.Asterisks indicate performance significantly different from the baseline at p < 0.05.
- Matched and mismatched speech-enhancement conditions compare enhanced training with raw testing, identical enhancement for training and testing, raw training with enhanced testing, and a raw-data baseline.The comparison explicitly distinguishes mismatched, matched, raw-trained, and baseline configurations.
- The training set is identified as Pitt-origin.
- The reported all-results figures use F1 with AD or Control designated as the positive class.The figures also mark results significantly different from the binomial majority-class baseline.
G.1 Generalization Test On Raw Data
This section presents eGeMAPS-based generalization tests on raw and speech-enhanced data. It compares enhancement matching conditions and reports AD- and Control-positive F1 scores against a majority-class baseline.
- eGeMAPS-based models are evaluated on raw data with same-domain tests and testing on the Lu dataset.Same-domain evaluation uses the test split of the dataset used for training.
- Speech-enhanced data are evaluated using the same-domain test definition, with significance compared against the binomial majority-class baseline.Asterisks denote significant differences at p < 0.05.
- The enhancement comparison includes mismatched, matched, raw-trained, and raw-training/raw-testing baseline configurations.Matched means applying the same enhancement to training and test data.
- The training set is identified as Pitt-origin.
- The all-results figure reports F1 with AD or Control treated as the positive class.Significance markers indicate differences from the binomial majority-class baseline.