Source-linked AI summary
LLM-Anchored Paralinguistic Enrichment for Alzheimer's Disease Detection
Xiao Wei, Yuqin Lin, Yaru Cao, Jinyu Li, Bin Wen, Kai Li, Yueying Chen, Longbiao Wang, Jianwu Dang
TL;DR
Speech-based AD detection must integrate linguistic content with heterogeneous paralinguistic cues while accounting for unequal modality contributions. LAPE anchors paralinguistic enrichment to LLM-derived text representations and achieves state-of-the-art accuracy and F1 across ADReSS and ADReSSo under participant-level CV and LOSO.
Problem
Existing approaches incompletely cover heterogeneous paralinguistic cues and insufficiently coordinate local prosodic structure, utterance-level speech features, and linguistic content.
Method
LAPE uses LLM-derived linguistic representations as an organizing anchor, enriching them through prosodic event textualization, lexico-prosodic unitization and chunking, and text-anchored fusion.
Results
LAPE achieves state-of-the-art accuracy and F1 on ADReSS and ADReSSo under participant-level cross-validation and leave-one-subject-out evaluation.
Takeaways & Limitations
LAPE coordinates speech evidence across temporal scales while preserving text as the anchor for speech-based AD detection.
Takeaways & Limitations
Duration normalization relies on a fixed task-specific reference table, with linear extrapolation for words longer than the sparsely observed range.
Abstract
from arXiv · showhide
Speech-based automatic detection of Alzheimer's disease (AD) provides a non-invasive and scalable approach to early cognitive screening. AD affects both lexical-semantic organization and speech production, including atypical pauses and word elongations. However, existing methods have yet to fully integrate these paralinguistic cues with linguistic content. We propose LLM-Anchored Paralinguistic Enrichment (LAPE), which enriches LLM-derived linguistic representations with paralinguistic cues through three coordinated innovations. The first is prosodic event textualization, which enables the LLM to model pauses and elongations jointly with lexical content by encoding them as explicit markers with bounded duration-aware repetition. The second is lexico-prosodic unitization and chunking, which preserves event identity and magnitude in both modalities by pooling only consecutive word units. The third is text-anchored paralinguistic fusion, which integrates local and utterance-level speech features by using NormGate to normalize and dynamically scale them relative to text. We evaluate LAPE on ADReSS and ADReSSo using participant-level cross-validation and leave-one-subject-out evaluation. LAPE achieves state-of-the-art performance across all four primary settings. Code will be released upon acceptance.
Introduction
Speech-based AD detection benefits from combining linguistic content with paralinguistic speech-production cues, but existing approaches leave timing coverage and multimodal coordination incomplete. LAPE addresses these gaps by anchoring structured paralinguistic enrichment to LLM-derived linguistic representations.
- Spontaneous speech offers accessible, non-invasive assessment while jointly reflecting lexical-semantic abilities and speech-production characteristics.
- AD detection requires both linguistic information and paralinguistic cues capturing speech production, including timing, articulation, and voice characteristics.
- Existing protocols leave pauses and elongations implicit or encode only pauses, while fusion methods insufficiently coordinate local prosodic structure with utterance-level speech features.
- LAPE uses LLM-derived linguistic representations as the organizing anchor for structured paralinguistic cues in speech-based AD detection.
1. Prosodic event textualization. We enrich transcripts
LAPE textualizes pauses and elongations as explicit markers alongside lexical content, using bounded duration-aware repetition so the LLM can model timing jointly with words.
- Prosodic event textualization makes pauses and elongations explicit in text, enabling joint modeling with lexical content.Bounded duration-aware marker repetition represents event magnitude while keeping the encoding explicit.
2. Lexico-prosodic unitization and chunking.
LAPE preserves prosodic event identity and magnitude across text and speech by organizing both modalities into units and pooling only consecutive word units.
- Chunk aggregation preserves event identity and magnitude in both modalities by pooling only consecutive word units.
3. Text-anchored paralinguistic fusion.
LAPE uses text as the anchor for integrating local and utterance-level speech features, with NormGate normalizing and dynamically scaling speech-derived blocks relative to text.
- NormGate normalizes and dynamically scales local and utterance-level speech features relative to text.
Related Work
Prior AD speech-detection methods separately enrich transcripts with pauses or fuse speech and text, but they do not jointly preserve local prosody, utterance-level speech, and unequal modality contributions.
- Punctuation-based pause encodings can conflate speech timing with syntactic or discourse boundaries.
- Explicit transcript schemes mainly encode inter-word pauses, while word-level elongation is usually represented as a continuous duration or acoustic feature.
- LAPE addresses these gaps by jointly textualizing pauses and elongations with dedicated event markers and anchoring speech evidence to LLM-derived text.
- Utterance-level fusion offers broad coverage but collapses local timing structure, whereas finer-grained methods establish speech–text relations before fusion.
- Existing fusion methods generally fail to jointly preserve local prosodic structure, incorporate utterance-level speech features, and account for unequal modality contributions.
Methodology: LAPE
LAPE augments transcripts with duration-aware prosodic markers, organizes aligned text and speech into event-preserving units and chunks, and uses text-anchored gating to integrate local and utterance-level speech evidence.
- Text-Anchored Paralinguistic Fusion: LAPE uses LLM-derived text as the anchor while integrating local and utterance-level speech evidence through unitization, chunking, and dynamic scaling.
- Prosodic Event Textualization: Pause and elongation events are encoded as dedicated markers with bounded duration-aware repetition alongside lexical content.
- Prosodic Event Textualization: Word timestamps define inter-word pauses and word durations, with pause markers preceding words and elongation markers attached after host words.
- Lexico-Prosodic Unitization and Chunking: Frozen Kimi and Whisper encoders provide contextualized token and frame states organized around timestamp-derived word and event units.
- Lexico-Prosodic Unitization and Chunking: Each augmented word group shares one contextualized text span, while its speech units select localized word-internal or inter-word frame intervals.
- Lexico-Prosodic Unitization and Chunking: Consecutive word units are mean-pooled into lexical chunks, while pause, elongation, and boundary units remain singletons so repeated markers preserve event magnitude.
Experimental Setup
LAPE is evaluated on ADReSS and ADReSSo using participant-level cross-validation and leave-one-subject-out evaluation, with cross-setting baselines and the reported implementation configuration.
- Datasets and Evaluation: The study evaluates LAPE on ADReSS and ADReSSo using participant-level five-fold cross-validation and leave-one-subject-out evaluation.ADReSS includes audio and manual transcripts, whereas ADReSSo is speech-only; ADReSSo uses Whisper largev3 transcripts and word timestamps.
- Baselines: MVG-GAT and CogniAlign serve as cross-setting baselines representing adaptive multimodal fusion and token-level audio–text alignment with cross-attention.Additional compact table labels include FFeat-SVM and LLaMA2-PL.
- Training: Training uses AdamW with gradient clipping, early stopping, fixed optimization settings, and grid search over hidden-layer sizes and batch size.The learning rate is 5e−4, weight decay is 1e−4, dropout is 0.3, label smoothing is 0.1, and LOSO reserves 15% of remaining training data for validation.
Experimental Results
LAPE ranks first across ADReSS and ADReSSo under participant-level CV and LOSO, with complementary prosodic markers, chunking, and NormGate contributing to performance.
- Main Results: 98.18% and 94.01% under CV, and 95.37% and 91.57% under LOSO, make LAPE first across ADReSS and ADReSSo.Its lead over the strongest listed baseline spans 2.21–8.37%, averaging 5.55%.
- Transcription Protocol Analysis: Joint pause-and-elongation textualization improves over Basic by 5.45%, 5.56%, 1.82%, and 3.62% across the four settings.The joint protocol exceeds the better single-event protocol by 2.74% on average.
- Aggregation Analysis: Chunk aggregation ranks first throughout and exceeds the better fine-grained alternative by 2.73%, 2.78%, 0.68%, and 2.41%.Chunking pools consecutive word units while keeping pause, elongation, and boundary units separate.
- Component Ablation: Text-only models outperform speech-only models by 9.27% and 12.03% on ADReSS and 3.59% and 7.22% on ADReSSo under CV and LOSO.Adding local Whisper features still yields gains in all four settings, averaging 1.07%.
- Component Ablation: NormGate raises accuracy over the no-gate variant by 0.91%, 2.78%, 0.62%, and 1.21%, averaging 1.38%.Its mean gain is larger under LOSO than CV, 2.00% versus 0.77%.
Conclusion
The paper presents LAPE as an LLM-anchored framework that enriches linguistic representations with paralinguistic cues for speech-based AD detection.
- Conclusion: LAPE combines prosodic event textualization, lexico-prosodic unitization and chunking, and text-anchored fusion of local and utterance-level speech features.It achieves state-of-the-art accuracy and F1 on ADReSS and ADReSSo under both CV and LOSO.
Generative AI Use Disclosure
The authors disclose that ChatGPT was used only to polish author-written text, with all suggestions reviewed and revised by the authors.
- Generative AI Use Disclosure: ChatGPT was not used to generate scientific content or references, and the authors retain responsibility for the manuscript.