PaperlayerView exact revision on arXiv

Source-linked AI summary

Explainable and Generalisable LLM-based Cognitive Decline Detection with Spontaneous Speech

Ziyun Cui, Wen Wu, Chuan Shi, Shuguang Yang, Xueying Gui, Yan Zheng, Qiong Yang, Haiyan Zhao, Wei-Qiang Zhang, Ji Wu, Yelei Li, Nan Li, Chao Zhang

arXiv:2609.34217v1eess.AScs.CL

TL;DR

Speech-based cognitive screening could offer an accessible alternative to resource-intensive assessments, but existing systems face limitations in interpretability, ASR reliability, and language and task coverage. This paper presents a bilingual raw-speech model that jointly classifies cognitive status and generates explanations; its SpeechLLM models outperformed the compared baselines, transferred to held-out tasks, and produced explanations clinicians rated as relevant and speech-grounded.

  • Problem

    Existing speech-based AD approaches face a performance-interpretability trade-off, ASR-related error propagation, and concentration on English-only, single-task datasets.

    Source: Introduction · p. 3

  • Method

    A bilingual SpeechLLM processes raw speech using joint acoustic-semantic representations and jointly performs cognitive-status classification and natural-language explanation generation.

    Source: Introduction · p. 3

  • Results

    Across six dataset/task conditions, SpeechLLM models outperformed all three baselines; on NCMMSC-AD, our model reached 0.857 accuracy versus Gemini-few-shot’s 0.392.

    Source: Introduction · p. 4, Discussion · p. 9

  • Takeaways & Limitations

    Clinicians rated generated explanations as clinically relevant and largely consistent with speech evidence, and joint training transferred to held-out PUTH-AD tasks without task-specific fine-tuning.

    Source: Introduction · p. 4, Discussion · pp. 9–10

    No fully independent external cohort was used for validation, and score-threshold PUTH-AD labels may contain diagnostic-label noise.

    Source: Discussion · p. 10

Abstract

from arXiv · show

Alzheimer's disease (AD) and mild cognitive impairment (MCI), which may precede AD, manifest early through subtle linguistic and acoustic alterations. Traditional diagnostics, however, are often resource-intensive and lack scalability for mass screening. To address these challenges, we introduce a novel bilingual speech large language model framework for automated, explainable cognitive screening. Unlike conventional pipelines that rely on error-prone automatic speech recognition, our system directly processes raw speech to learn joint acoustic-semantic representations, preserving critical prosodic cues often lost in transcription. Utilising our newly collected PUTH-AD dataset alongside multiple open-source corpora, we implemented a multi-task learning objective that simultaneously performs cognitive status classification and generates clinician-understandable natural language explanations. Our system achieved the highest average accuracy and AUROC across six dataset/task conditions, comparing three representative baselines. The system demonstrated cross-task transfer to held-out PUTH-AD task subsets, maintaining classification accuracy on an entirely unseen cognitive task without task-specific fine-tuning. Furthermore, clinician evaluation confirms that the generated explanations are both clinically relevant and largely consistent with the underlying speech evidence, supporting their potential utility in clinical interpretation. This study provides a scalable, objective, and explainable framework for speech-based cognitive screening, combining cognitive status classification with natural language explanations that clinicians can assess and verify, bridging the gap between advanced AI and clinical utility.

Introduction

Speech offers an accessible route to cognitive screening, but existing speech-based AD systems face trade-offs in interpretability, transcription reliability, and language/task coverage. The paper introduces a bilingual raw-speech framework that jointly classifies cognitive status and generates clinician-understandable explanations.

Motivation

  • Speech may support scalable, low-cost screening because it reflects cognitive and linguistic processing, while conventional assessments are resource-intensive and difficult to scale.Speech can be collected non-invasively with ubiquitous devices and may reveal pauses, hesitations, word-finding difficulty, and other early changes.Source: Introduction · p. 2

Research gap

  • Existing systems trade classification performance against interpretability, rely on error-prone ASR, and often use English-only, single-task datasets.ASR-transcript accuracy has been reported as markedly lower than accuracy with manual reference transcripts, which are unavailable in real-world deployment.Source: Introduction · p. 3

Approach

  • The paper introduces a bilingual system that processes raw speech to learn joint acoustic-semantic representations and jointly classifies cognitive status and generates explanations.The representations preserve prosodic cues such as pauses, speech rate, and intonation alongside linguistic content.Source: Introduction · pp. 3–4

Contributions

  • The contributions include outperforming SSL and general-purpose language-model baselines, transferring across speech tasks, and producing explanations clinicians judged relevant and largely consistent with speech evidence.The paper also presents the PUTH-AD dataset, collected from 166 participants completing four distinct tasks.Source: Introduction · pp. 3–4

Results

Across six dataset/task conditions, Kimi-Audio had the highest average accuracy, and multi-task training improved its average accuracy and AUROC over classification-only training. Joint training also transferred to held-out tasks, while class-wise and clinician evaluations characterized performance and explanation quality.

Dataset Statistics

  • The three corpora span English and Chinese; PUTH-AD includes 166 participants across four tasks, and its class distribution includes proportionally fewer AD and more MCI cases.Training explanations comprised 1,393 Gemini-2.5-Flash-generated samples across datasets and tasks.Source: Dataset Statistics · pp. 4–5

Overall Classification Performance

  • 0.666 average accuracy was highest across six conditions; on ADReSS, Kimi-Audio scored 0.893 versus SALMONN 0.882, SSL-Model 0.831, Text-only-LLM 0.802, and Gemini-few-shot 0.637.Among PUTH-AD tasks, Personal Recall had the highest mean accuracy (0.638) and AUROC (0.770).Source: Table 1 · p. 5, Overall Classification Performance · p. 6

Multi-Task Learning Ablation Analysis

  • Compared with classification-only training, multi-task training raised Kimi-Audio’s average accuracy from 0.622 to 0.666 and average AUROC from 0.749 to 0.777; the largest accuracy gain was on ADReSS, from 0.815 to 0.882.Source: Multi-Task Learning Ablation Analysis · p. 6
  • Compared with generation-only training, multi-task training raised average accuracy from 0.471 to 0.602 for SALMONN and from 0.500 to 0.666 for Kimi-Audio.SALMONN’s explanation quality degraded on Chinese data, whereas Kimi-Audio’s average BERT-Score changed from 0.750 to 0.744 and hit rate from 0.610 to 0.604.Source: Multi-Task Learning Ablation Analysis · pp. 6–7

Joint Training on Multiple Datasets Improves Generalisation

  • Joint training raised held-out PUTH-AD accuracy on FPD from 0.507 to 0.550 and CTD from 0.553 to 0.576, but lowered it on AFT from 0.553 to 0.538.Hard-voting accuracy reached 0.700 ± 0.029 under joint training, versus 0.630 ± 0.039 for the strongest individual task; this improvement was not observed with separate training.Source: Joint Training on Multiple Datasets… · p. 7

Class-wise Performance Analysis

  • For the pooled joint-training test predictions, HC had the highest F1, with F1, sensitivity, and specificity of 0.87, 0.88, and 0.89; sensitivity was lower for MCI and AD.The authors attribute the lower MCI and AD sensitivity to limited sample numbers for those classes.Source: Class-wise Performance Analysis · p. 7

Clinical Evaluation of Generated Explanations

  • Kimi-Audio explanations received mean clinician scores of 4.36 for clinical relevance and 4.71 for evidence faithfulness, with higher relevance for correctly classified samples.Gemini-generated training explanations scored 4.56 for relevance and 4.70 for faithfulness; clinician revisions had BERTScore 0.952 and hit rate 0.935.Source: Clinical Evaluation of Generated… · p. 8

Discussion

The discussion presents raw-speech SpeechLLMs as stronger than the compared baselines, finds that joint classification-and-explanation training generally supports classification, and reports transfer to held-out speech tasks. It also identifies limits in sample size, labels, and external validation.

Discussion

  • Personal Recall accuracy exceeded the three structured PUTH-AD tasks by 8.5 to 13.1 percentage points with Kimi-Audio.The authors suggest its open-ended narrative format may expose differences in linguistic complexity, coherence, and discourse planning.Source: Discussion · pp. 8–9
  • SpeechLLMs outperformed all three baselines; transcript-based LLMs captured linguistic cues but lost acoustic and prosodic information potentially important for detection.The discussion attributes the gains over SSL pipelines to end-to-end audio processing with joint acoustic-semantic representations.Source: Discussion · p. 9
  • Multi-task training generally maintained or improved accuracy over classification-only training, while generation-only training performed poorly for classification.The authors suggest explanation supervision may regularise representations, but state that fluent explanations alone do not provide sufficiently precise diagnostic supervision.Source: Discussion · p. 9
  • Joint training improved accuracy on the held-out Family Party Description task, which was excluded from training.The discussion also reports improved BERT-Scores across datasets, including unseen data, and ensemble accuracy rising from 0.634 to 0.700.Source: Discussion · pp. 9–10
  • The explanations and held-out-task results suggest potential use in workflows where clinicians assess AI rationales and speech tasks vary across settings.The discussion frames explanations as natural-language rationales clinicians can evaluate and verify.Source: Discussion · p. 10
  • No fully independent external cohort was used, and limitations also include small samples, score-defined PUTH-AD labels, and Gemini-generated explanation targets.The authors note that these targets may carry unsupported observations, overconfident statements, or teacher-model biases despite clinician evaluation.Source: Discussion · p. 10

Methods

The methods combine PUTH-AD and open-source speech datasets, then train SpeechLLMs for cognitive-status classification and explanation generation. Evaluations use cross-subject splits, multiple baselines, and clinician review of generated explanations.

Datasets and Speech Tasks

  • PUTH-AD includes 166 participants completing four tasks: animal fluency, two picture descriptions, and a three-minute personal-recall narrative.The collection app guided participants through task instructions and recordings on mobile devices; recordings were made in quiet hospital or community settings.Source: Datasets and Speech Tasks · pp. 10–11
  • PUTH-AD labels HC, MCI, and AD reflect MoCA and HKBC score thresholds, with the more impaired category assigned when scales disagreed.These screening-based labels are not formal AD diagnoses; no independent neurological adjudication or biomarker confirmation was performed.Source: Datasets and Speech Tasks · p. 12
  • The study also used English ADReSS data and Mandarin NCMMSC-AD data, alongside de-identified secondary datasets collected under their providers’ requirements.ADReSS contains 330 participants with binary labels; NCMMSC-AD has 399 segments from 175 participants with three-class labels.Source: Datasets and Speech Tasks · p. 12

Model Architecture

  • The framework uses dual output heads for classification and explanation generation, jointly optimised with classification and generation losses.SALMONN and Kimi-Audio were the two 7-billion-parameter SpeechLLM backbones; DoRA adapted attention query and key matrices while most pretrained parameters remained frozen.Source: Model Architecture · pp. 13–14

Baseline Models

  • Baselines comprised an ASR-plus-SSL feature pipeline, few-shot Gemini audio classification, and a fine-tuned text-only LLM using ASR transcripts.The text-only baseline shared Kimi-Audio’s classification head design, DoRA strategy, random seeds, and data splits.Source: Baseline Models · p. 14

Statistical Analysis

  • Trainable models used fixed subject-level train, development, and test partitions and five random-seed runs; accuracy and AUROC summarized classification performance.Explanation quality was assessed with BERT-Score and keyword hit rate, and subject-level bootstrap 95% confidence intervals were calculated.Source: Statistical Analysis · pp. 15–16

Implementation Details

  • SpeechLLMs were fine-tuned with DoRA for 10 epochs, using development-set criteria to select checkpoints for test evaluation.Kimi-Audio inference required approximately 7 seconds per sample and 34 GB of GPU memory.Source: Implementation Details · p. 16

Clinical Evaluation

  • Five neurologists evaluated 327 Chinese-language test samples for explanation relevance and faithfulness on a five-point scale.Each sample was assessed by one neurologist; explanations were also compared with clinician-revised Gemini explanations using BERT-Score and hit rate.Source: Clinical Evaluation · p. 17

Code Availability

The authors provide a public code repository with materials for reproducing the reported analyses.

Code Availability

  • The study code is publicly available under the Apache-2.0 license.The repository is identified as https://github.com/cuiziyun/kimia-explainable-AD.Source: Code Availability · p. 18
  • The repository includes the complete environment specification and software versions.These materials accompany the study code.Source: Code Availability · p. 18
  • Configuration files, prompt templates, and key parameters are included to reproduce the reported analyses.Source: Code Availability · p. 18

Funding

The work received support from the OPPO Research Fund and the National Natural Science Foundation of China, with different funder roles reported.

Funding

  • The work was supported by the OPPO Research Fund and National Natural Science Foundation of China Grant No. 62501336.Source: Funding · p. 18
  • OPPO participated in the study design and manuscript preparation.Source: Funding · p. 18
  • The National Natural Science Foundation of China had no role in study design, data analysis, manuscript preparation, or the publication decision.Source: Funding · p. 18

Supplementary Information

The supplementary material details how explanation targets are prompted and illustrates both successful and failed generations. It also documents the few-shot baseline and the scope of the supplementary result tables.

Generation Examples

  • Successful examples identify word-finding difficulty, repetition, and weak self-monitoring in AFT speech, and coherent, detailed discourse in an FPD healthy-control sample.The AFT explanation describes impairment cues; the FPD explanation notes the absence of repetition and word-finding difficulty.Source: Generation Examples · p. 30, Prompts · pp. 32–33
  • Two representative failures show explanations can misread confused descriptions or normal informal speech, so outputs should remain auxiliary rather than clinical conclusions.One case was AD predicted as HC; another was HC predicted as MCI.Source: Generation Examples · p. 30

Prompts

  • The Gemini explanation-target prompt uses two stages: detailed acoustic and linguistic analysis, followed by a concise account retaining the two or three most important cognitive-relevant points.The prompts also supply clinical and task-specific criteria and example outputs.Source: Prompts · pp. 30–31, Figure 3 · p. 34
  • The Gemini few-shot baseline performs direct classification using six in-context NCMMSC-AD examples, two for each of HC, MCI, and AD.It outputs a cognitive-status category rather than a rationale.Source: Prompts · p. 31, Figure 4 · p. 35
  • Supplementary Tables 1–3 cover separate dataset/task results, training-objective ablations, and separate versus joint training, respectively.Table 1 reports classification and explanation metrics; Table 2 compares training variants and Table 3 assesses held-out PUTH-AD tasks.Source: Table 1 · p. 37, Table 2 · p. 38, Table 3 · p. 39
Loading 2609.34217v1…