Source-linked AI summary
Why Alzheimer's Speech Screening Fails to Generalize: Bridging the Deployment Gap via Cross-Corpus Evidence Anchoring
Zijian Lu, Sizhe Liu, Yin Zhang, Jixuan Deng, Xinrong Lin, Xinchen Yuan, Chicheng Jin, Yiping Zuo, Yuanchao Li
TL;DR
Speech-based cognitive screening faces a cross-corpus transfer gap: marker directions and model performance may vary across unseen domains. The paper audits markers and fuses selected evidence anchors with XLM-R in leave-one-corpus-out evaluation. Fusion balances mean performance and worst-domain robustness, while the authors identify the four-corpus scope as a limitation.
Problem
It remains unclear whether speech screening signals transfer reliably across unfamiliar tasks, languages, and recording protocols, despite their potential for non-invasive screening.
Source: §1 · p. 2Method
The paper audits marker reliability across corpora, selects evidence anchors during training, and fuses their scores with XLM-R under leave-one-corpus-out evaluation.
Source: §1 · p. 2, §3.3 · p. 5, §6.1 · p. 10Results
Balanced fusion achieves a mean speaker AUC of 0.785, 0.019 above GroupDRO, while anchor-heavy fusion reaches a worst-case domain AUC of 0.615, 0.111 above GroupDRO.
Source: §6.2 · p. 12, §7 · p. 13Takeaways & Limitations
The results support evidence auditing and worst-case domain testing for speech-based cognitive screening rather than relying only on stronger deep classifiers or source-loss reweighting.
Source: §7 · p. 14Evaluation is limited to four English and Mandarin corpora, one generic domain-generalization baseline, a simple fusion architecture, and fixed deep representations.
Source: §7 · p. 14
Abstract
from arXiv · showhide
Speech-based screening is a promising, non-invasive approach for detecting Alzheimer's disease and related cognitive risks. However, models trained on a single domain often generalize poorly to unseen languages, tasks, or recording protocols. This paper investigates this deployment gap using a leave-one-corpus-out evaluation across four distinct datasets. Among 70 interpretable speech and language features, 59 exhibit direction conflicts between healthy control and cognitive risk groups across corpora, with pause, silence, and speech rate showing high protocol sensitivity. Furthermore, while the XLM-R text baseline achieves strong average performance, its Area Under the ROC Curve (AUC) drops to 0.520 on the weakest held-out domain. A standard GroupDRO baseline reaches a 0.766 mean speaker AUC and a 0.504 worst-domain AUC under the same protocol. To address this, we propose a fusion method that integrates XLM-R text baseline scores with evidence anchors selected during training. Balanced fusion achieves a 0.785 mean speaker AUC, while anchor-heavy fusion raises the worst-case speaker AUC to 0.615. This work highlights the need to audit feature transferability and report worst-case domain robustness in cognitive speech screening.
1 Introduction
Speech can support non-invasive cognitive screening, but evidence from familiar domains may not transfer reliably across tasks, languages, and recording protocols. The paper audits cross-corpus markers and develops a framework that fuses selected evidence anchors with XLM-R scores.
1 Introduction
- Speech is a low-cost, non-invasive screening signal that can be collected remotely and tracked longitudinally.Speech analysis can use both interpretable markers and machine-learning models.Source: §1 · p. 1
- Prior screening approaches use interpretable markers such as pauses and speech rate or deep representations such as XLM-R and wav2vec 2.0.The paper frames both approaches as vulnerable to cross-domain transfer problems.Source: §1 · p. 2
- Models trained within one corpus may fit disease evidence alongside task artifacts and cohort bias, while performance averages can conceal severe failures on unseen corpora.Users encounter their own deployment setting, motivating worst-case domain robustness as a primary evaluation criterion.Source: §1 · p. 2
- 59 of 70 interpretable markers show direction conflicts between healthy controls and cognitive-risk groups across four corpora, while balanced fusion gives the best mean AUC and anchor-heavy fusion raises weakest-domain performance.Pause, silence, and speech rate are particularly protocol-sensitive; the fusion integrates audited anchors with XLM-R text-baseline scores.Source: §1 · pp. 2–3
- The proposed framework uses heterogeneous datasets in leave-one-corpus-out evaluation, audits marker reliability, selects robust evidence anchors, and fuses their scores with XLM-R.Each held-out corpus simulates a new deployment environment.Source: §1 · p. 2
2 Related Work
Prior work establishes speech markers and deep representations as useful for cognitive screening, while leaving transfer across corpus protocols an open concern. This section reviews marker sensitivity, held-out evaluation, and deep-model deployment risks.
2.1 Speech and Language Markers for Cognitive Screening
- DementiaBank and TalkBank provide core transcribed clinical speech resources, mostly from picture-description tasks, while reviews report substantial variation across datasets and evaluation protocols.Disfluency and interactional cues are also sensitive to transcription source and task format.Source: §2.1 · p. 3
- Pause-based markers depend on voice-activity thresholds, microphones, tasks, speaking time, and segmentation, while lexical markers are sensitive to transcription, tokenization, and prompts.Within-corpus significance and reliable direction transfer to an unseen domain are distinct questions.Source: §2.1 · p. 3
2.2 Shared Tasks and Cross-Corpus Evaluation
- ADReSS and ADReSSo standardized balanced splits and baselines, and later shared tasks extended evaluation to multilingual MCI and Chinese Alzheimer’s recognition.Because each corpus remains tied to its collection protocol, prior work motivates evaluation on held-out environments rather than in-distribution averages.Source: §2.2 · p. 3
2.3 Deep Representations
- BERT and XLM-R provide linguistic representations, while wav2vec 2.0 and wavLM provide self-supervised speech representations.Multimodal ADReSSo systems report complementary gains from acoustic, lexical, disfluency, and pause features.Source: §2.3 · pp. 3–4
- Deep models can overfit to corpus identity, transcription artifacts, channel conditions, and prompt structure, and self-supervised models show inconsistent behavior across corpora.The paper treats XLM-R as an audited text stream and combines it with cross-corpus evidence anchors.Source: §2.3 · p. 4
3 Data and Evaluation Protocol
The study evaluates speech-based cognitive-risk screening across four heterogeneous corpora using training-only pipelines and held-out-corpus testing. It compares interpretable and frozen deep representations, then reports a source-corpus GroupDRO baseline and control experiments.
3.1 Corpora
- Four corpora span Mandarin and English, picture-description tasks, and connected clinical speech; the binary target groups healthy controls against cognitive risk, combining Alzheimer’s disease, MCI, and other impairment labels.Multiple samples per speaker are averaged into a risk score before speaker-level AUC calculation.Source: §3.1 · p. 4
3.2 Features and Models
- The interpretable feature pool contains 70 speech and language features across temporal, acoustic, lexical, and discourse dimensions, with 24 canonical features forming the traditional baseline.The corpora include NCMMSC, Pitt Cookie, TAUKADIAL Mandarin, and Mandarin Chou.Source: §3.1 · p. 4, §3.2 · p. 5
- Deep baselines use frozen 768-dimensional XLM-R transcript embeddings and 1024-dimensional wav2vec 2.0 representations from its 12th and 24th layers, with no encoder fine-tuning.Continuous features use training-fitted winsorization, median imputation, and z-score normalization; the downstream model is weighted L2-regularized logistic regression.Source: §3.2 · p. 5
3.3 Leave-One-Corpus-Out Protocol
- Leave-One-Corpus-Out evaluation holds out one corpus as an external test set and trains on the other three, with all pipeline components fit only on training corpora.The primary measures are macro-averaged mean and worst-case held-out speaker AUC; GroupDRO weights source-corpus groups without using the held-out corpus.Source: §3.3 · p. 5
- Four controls test whether classifiers capture transferable indicators: shuffled training labels, metadata and length inputs, corpus-identity-only inputs, and a probe predicting corpus from interpretable features.The corpus-identity probe quantifies distributional distinctiveness among the four datasets.Source: §3.3 · pp. 5–6
4 Evidence Transferability Audit
The audit tests whether speech and language markers preserve their direction across corpora and task families. Most markers reverse across datasets, while lexical and POS measures are more consistent than temporal measures; isolated indicators also transfer weakly.
4.1 Audit Definition
- The audit defines direction transferability using within-corpus standardized Risk-versus-HC effect directions, treating a reversal across deployment settings as non-transferable regardless of localized effect magnitude.A positive standardized effect indicates elevated marker expression in the Risk group.Source: §4.1 · p. 6
4.2 Marker Direction Conflicts
- 59 of 70 markers show cross-dataset direction conflicts, including 20 of 24 core markers; task-family reversals affect 29 of 70 markers and 7 of 24 core markers.Figure 1 quantifies conflict prevalence across marker scopes.Source: §4.2 · pp. 6–7, Figure 1 · p. 7
- Pronoun and function-word ratios rise consistently in Risk speech, while content-to-function-word and timestamp-coverage ratios stay lower; silence and speech-rate directions vary by corpus.Silence ratio decreases in NCMMSC but increases in the other three corpora; character speech rate rises in NCMMSC and Pitt Cookie but trends downward in TAUKADIAL Mandarin and Mandarin Chou.Source: §4.2 · p. 7
4.3 Evidence Type Structure
- The 70-marker audit exposes systemic instability rather than defining a classifier feature space; lexical and POS measures align across domains, while temporal measures often invert.Figure 2 groups standardized HC-versus-Risk contrasts across all four corpora by evidence type.Source: §4.3 · p. 7, Figure 2 · p. 8
- The pre-specified 24-indicator panel includes only features computable across all four corpora under uniform speaker-level aggregation, separating nine evidence families.The families span word class, temporal fluency, discourse structure, information content, speech quantity, lexical diversity, alignment quality, disfluency, and repetition.Source: §4.3 · p. 8
- The top single indicator reaches only 0.658 mean AUC, and many indicators fall below chance (0.50) on worst-case-domain AUC.Figure 3 reports macro-averaged mean AUC with 95% confidence intervals and worst-case domain AUC for each core indicator evaluated alone under LOCO.Source: §4.3 · pp. 8–9, Figure 3 · p. 9
5 Deep Representations Under External Shift
Under leave-one-corpus-out evaluation, pre-trained representations capture contextual information but do not inherently mitigate external domain shift. XLM-R is the strongest standalone deep baseline, yet performance varies sharply across held-out corpora.
5 Deep Representations Under External Shift
- Pre-trained representations capture rich contextual information but do not inherently mitigate external domain shift.The LOCO evaluation treats each corpus in turn as an unseen deployment environment; NCMMSC is the hardest fold in this evaluation.Source: §5 · pp. 9–10
- XLM-R text baseline reaches a mean AUC of 0.769, ranging from 0.520 on NCMMSC to 0.968 on Mandarin Chou.Both wav2vec 2.0 layers are strong on Mandarin Chou and TAUKADIAL Mandarin but near chance on NCMMSC and weak on Pitt Cookie.Source: Table 2 · p. 9, §5 · p. 10
- Because macro-averaged performance can mask collapse on some domains, the authors select XLM-R and integrate audited interpretable evidence.The variability in Table 2 motivates combining the deep baseline with an evidence-selection framework.Source: Table 2 · p. 9, §5 · p. 10
6 Our Proposed Transfer Protocol
The protocol selects compact training-only evidence anchors and fuses their speaker-level scores with XLM-R under leave-one-corpus-out evaluation. Balanced fusion leads mean AUC, while anchor-heavy fusion improves the worst-domain result; the selected evidence varies across folds.
6.1 Evidence Anchor Selection
- The selector ranks 24 core indicators by macro-averaged training-corpus AUC, chooses the top four, and fits weighted L2-regularized logistic regression without observing the held-out corpus.The audited indicators constrain the anchor search space, with selection and tuning confined to each fold’s training partition.Source: §6.1 · p. 10, §6 · p. 10
- 0.629 macro-averaged mean AUC and 0.588 worst-case AUC come from source-AUC anchor selection, outperforming all 24 indicators (0.521 mean, 0.418 worst) and all 70 markers (0.538 mean, 0.395 worst).Table 3 reports AUC for traditional-indicator selection under leave-one-corpus-out evaluation.Source: §6.1 · p. 10, Table 3 · p. 11
- Pronoun ratio is selected in all four folds, while the remaining anchor channels vary with the source-corpus mixture.Figure 4 illustrates the source-corpus anchor choices for each held-out corpus.Source: §6.1 · p. 11, Figure 4 · p. 11
6.2 Anchor Fusion
- The fusion combines speaker-level anchor and XLM-R scores; α is fixed before held-out evaluation, with 0.50 balancing mean AUC and 0.75 favoring the hardest corpus.The weight is applied to speaker-averaged risk scores; Figure 5 presents sensitivity across score normalizations.Source: §6.2 · pp. 11–12, Figure 5 · p. 12
- 0.785 mean AUC for balanced fusion is 0.019 above GroupDRO; anchor-heavy fusion reaches 0.615 worst-case domain AUC, 0.111 above GroupDRO.The anchor-heavy result is also 0.095 above the deep-only baseline in the hardest deployment environment.Source: Figure 6 · p. 12, §6.2 · p. 12
- Anchor and XLM-R scores have low mean absolute Spearman correlation across folds (ρ = 0.235; orthogonality index 0.765), and audit results pre-specify active evidence channels before fusion.Source: §6.2 · p. 13
- Shuffled-label, metadata-only, corpus-identity-marker, and transcript-length controls are near-random or negligible, while corpus identity remains highly predictable.Table 4 reports negative controls and corpus-separability checks under leave-one-corpus-out evaluation.Source: §6.2 · p. 13, Table 4 · p. 13
7 Discussion and Conclusion
The discussion frames speech screening as an external domain-transfer problem: traditional markers can reverse direction, and deep models can collapse on unseen domains. Evidence anchoring yields a tradeoff between mean performance and weak-domain robustness, within a limited evaluation scope.
7 Discussion and Conclusion
- Balanced fusion reaches 0.785 AUC, 0.019 above GroupDRO, while anchor-heavy fusion reaches 0.615 AUC, 0.111 above GroupDRO.The paper presents these as a tradeoff between mean performance and weak-domain robustness.Source: §7 · p. 13
- The evaluation covers four English and Mandarin corpora, one generic domain-generalization baseline, a simple fusion architecture, and fixed deep representations.Source: §7 · p. 14
- The authors conclude that deployable clinical speech screening requires evidence auditing and worst-case domain testing, not only stronger deep classifiers or source-loss reweighting.Source: §7 · p. 14