Source-linked AI summary
Detecting Linguistic Characteristics of Alzheimer's Dementia by Interpreting Neural Models
Sweta Karlekar, Tong Niu, Mohit Bansal
TL;DR
Language changes can indicate cognitive impairment in AD, but diverse speech patterns make feature-based classification challenging. The paper trains neural models on DementiaBank and interprets their learned linguistic patterns. Its POS-tagged CNN-LSTM reaches 91.1% accuracy, a new benchmark for the task.
Problem
AD speech cues vary across scenarios and disease stages, making feature-based approaches difficult to adapt to the linguistic and world knowledge needed for classification.
Source: §1 · p. 1Method
Results
91.1% accuracy on POS-tagged data set a new benchmark for the AD classification task.
Source: §5 · p. 3Takeaways & Limitations
The analyses recover previously reported AD speech patterns and provide new insights through automatic speech-pattern extraction.
Source: §1 · p. 2The authors propose adding surrounding interview context to distinguish temporarily unaffected speech in AD-positive interviews from continually unaffected control speech.
Source: §5 · p. 4
Abstract
from arXiv · showhide
Alzheimer's disease (AD) is an irreversible and progressive brain disease that can be stopped or slowed down with medical treatment. Language changes serve as a sign that a patient's cognitive functions have been impacted, potentially leading to early diagnosis. In this work, we use NLP techniques to classify and analyze the linguistic characteristics of AD patients using the DementiaBank dataset. We apply three neural models based on CNNs, LSTM-RNNs, and their combination, to distinguish between language samples from AD and control patients. We achieve a new independent benchmark accuracy for the AD classification task. More importantly, we next interpret what these neural models have learned about the linguistic characteristics of AD patients, via analysis based on activation clustering and first-derivative saliency techniques. We then perform novel automatic pattern discovery inside activation clusters, and consolidate AD patients' distinctive grammar patterns. Additionally, we show that first derivative saliency can not only rediscover previous language patterns of AD patients, but also shed light on the limitations of neural models. Lastly, we also include analysis of gender-separated AD data.
1 Introduction
Language changes can signal cognitive impairment in AD, but diverse, scenario-dependent speech patterns and disease-stage variation challenge hand-engineered classification. The paper introduces neural classifiers and interpretation methods to identify AD-related linguistic patterns.
1 Introduction
- The task demands linguistic and world knowledge because AD cues vary by scenario and disease stage, limiting the adaptability of feature-based approaches.The example combines an uncommon word with a self-correction that may signal confusion and memory loss.Source: §1 · p. 1
- The authors present CNN, LSTM-RNN, and joint CNN-LSTM models, with the best model using only existing automatic POS tags and establishing a new independent benchmark.The POS tags are commonly used and are already present in the dataset.Source: §1 · p. 1
- Activation Clustering, First Derivative Saliency, and automatic grammatical-pattern discovery are used to interpret what linguistic features the neural models learn.The two visualization techniques are credited to Girshick et al. (2014) and Simonyan et al. (2013), respectively.Source: §1 · p. 1
- The analyses recover previously reported speech patterns and offer new insights into AD characteristics through automatic pattern extraction.The authors state these findings could potentially help improve the accuracy and speed of medical diagnoses.Source: §1 · p. 2
- The paper also evaluates model performance on gender-separated data to examine whether the identified features are gender-specific.The introduction describes this as an analysis of separate gender subsamples.Source: §1 · p. 2
2 Related Works
Prior language-based AD detection relied mainly on hand-crafted transcript features, while neural approaches raised interpretability concerns. The paper combines neural classification with visualization methods to examine both model effectiveness and reasoning.
2 Related Works
- Earlier language-based AD detection relied mainly on hand-crafted transcript features, sometimes supplemented by acoustic data.The cited examples include Orimaye et al. (2017, 2015) and K¨onig et al. (2015); Rudzicz et al. (2014).Source: §2 · p. 2
- Hand-picked features depend on expert knowledge, may generalize poorly across disease stages or types, and can become outdated as language and culture evolve.Some nuanced features may also be difficult for humans to detect, especially in early AD.Source: §2 · p. 2
- Neural language models offer an alternative, but their reasoning is difficult to interpret, motivating visualization of effectiveness and limitations.Activation Clustering highlights sentence-level patterns, while First Derivative Saliency highlights word importance for the classification label.Source: §2 · p. 2
3 Classification Models
The paper compares CNN and LSTM-RNN classifiers with a hybrid CNN-LSTM that combines convolutional features with sequential modeling. Each architecture produces probabilities over two classes.
3 Classification Models
- The CNN embeds each sentence, applies convolution filters over varying word-window sizes, max-pools the features, and outputs two-class probabilities.The convolution and max-pooling approach is attributed to Collobert et al. (2011).Source: §3 · p. 2
- The LSTM-RNN processes embedded words sequentially and sends its final sentence-level state through a fully connected layer to produce class probabilities.The model is motivated by CNNs’ limited specialization for long-range sequential correlations, citing Pascanu et al. (2013a).Source: §3 · p. 2
- The CNN-LSTM places an LSTM layer over the CNN representation, combining the CNN convolution stage with the LSTM sequence stage.The authors explored the hybrid because the standalone models had comparable results to a previous best approach and complementary strengths.Source: §3 · p. 2
4 Experimental Setup
Experiments use DementiaBank transcripts and automatic linguistic annotations, with utterances as samples. The study compares models under tagged and untagged setups and tunes hyperparameters on a development set.
4 Experimental Setup
- DementiaBank provides transcripts and audio from AD and control interviews, including the Boston Cookie Theft description and a story-recall task.For Cookie Theft, participants describe an image; the corresponding figure identifies the task prompt.Source: §4 · p. 3, Figure 2 · p. 3
- Transcripts include automatic POS, tense, and repetition annotations, and the study splits them into utterances while excluding those without POS tags.The authors note that balancing the tagged and untagged setups reduced the data but supported fair comparison.Source: §4 · p. 3
- The CNN uses a 2-D convolution with filter sizes [3, 4, 5], the LSTM-RNN has two layers, and the CNN-LSTM uses [3, 4, 5, 6] filters and one LSTM hidden layer.Hyperparameters for each model were tuned on the development set.Source: §4 · p. 3
5 Results
With POS-tagged data, the CNN-LSTM achieved 91.1% accuracy, establishing a new benchmark. Error analysis found non-AD samples harder to classify, in part because utterance-level labels can include apparently unaffected speech from AD-positive interviews.
5 Results
- 91.1% accuracy: the POS-tagged CNN-LSTM set a new benchmark, while untagged CNN, LSTM-RNN, and CNN-LSTM models achieved 82.8%, 83.7%, and 84.9%, respectively.The models were compared after downsampling to remove utterances without POS tags.Source: §5 · p. 3, Table 1 · p. 3
- The authors note that prior results are mostly not directly comparable because studies used different metrics, datasets, or test sets.Orimaye et al. (2016) reported 87.5% accuracy using only 36 transcripts, yielding a different test set with little data and high variance.Source: §5 · p. 3
- Almost all AD-positive results were classified correctly, but non-AD errors may reflect AD-positive interviews containing partially unaffected utterances labeled as AD-positive.In a random sample of 10% of misclassified non-AD examples, 36.3% were short utterances found in both classes, and 63.7% could be classified either way without surrounding context.Source: §5 · p. 3
- Bidirectional models achieved 84.7% accuracy for LSTM-RNN, 86.2% for CNN-RNN, and 91.1% for tagged CNN-RNN.These are the reported accuracies for the bidirectional variants.Source: §5 · p. 3
- The authors propose using surrounding interview samples to distinguish temporarily unaffected speech in AD-positive patients from continuously unaffected control speech.The proposal follows the ambiguity of utterances that lack enough context for classification.Source: §5 · p. 4
6 Analysis and Visualization
The analysis examines gender differences and interprets the CNN-LSTM’s linguistic cues using activation clusters and first-derivative saliency. The analyses recover known AD speech patterns, identify task-dependent POS patterns, and illustrate how mixed cues can confuse classification.
6.1 Gender Differences
- 86.6% accuracy on male data and 86.2% on female data accompanied the same top ten POS tags, with no statistically significant gender difference detected.The analysis used gender-separated data; the female subset was down-sampled to match the male subset’s size and class ratio.Source: §6.1 · p. 4, §6.2 · p. 4
6.2.1 Rediscovering Existing Strategies
- Activation clusters recovered known AD patterns: short answers and speech bursts, repeated clarification requests, and utterances beginning with interjections.Examples include brief responses separated by pauses, past-tense clarification questions, and openings such as “oh,” “well,” “so,” and “right.”Source: §6.2.1 · p. 4
6.2.2 Automatic Cluster Pattern Analysis
- Automatic cluster analysis found distinct top POS tags across clusters, with similar tag profiles recurring within tasks; unaffected Cookie-task clusters commonly featured nouns, determiners, and participles.Non-AD clusters for the Cookie task shared the same top four POS tags; the reported tag frequencies are scaled by the number of tags in each cluster.Source: §6.2.2 · pp. 4–5
- AD cluster POS profiles differed by task: Recall clusters featured interjections and verbs, while the Cookie AD cluster featured adjectives and adverbs.For Cookie, the representative AD cluster shared only two of its most-used POS tags with a non-AD cluster; the three non-AD clusters lacked adjectives and adverbs among their top tags.Source: Table 3 · p. 4, §6.2.2 · p. 5
6 Analysis and Visualization
- First-derivative saliency estimates word influence on sentence classification by taking gradients of the final scores with respect to input word embeddings.The analysis presents this alongside activation clustering as a strategy for interpreting the CNN-LSTM’s linguistic cues.Source: §6 · p. 4, §6.3 · p. 5
6.3 First Derivative Saliency Heat Maps
- Saliency maps highlighted “uh” and “um” and an initial “and” in a correctly classified AD example, while a control example emphasized structural words such as “the,” “that,” and “is.”The Figure 3 caption identifies the left and middle maps as correctly classified AD and control examples, respectively.Source: §6.3 · p. 5, Figure 3 · p. 5
- In one misclassified example, utterance length and heavy determiner use contributed to the error despite repeated “um”s and an initial coordinating conjunction indicating AD.The authors suggest more conversational context and advanced neural methods as future steps for handling samples with both AD and control features.Source: §6.3 · p. 5
7 Conclusion
The conclusion reports a new neural-model benchmark for AD classification and interpretation of linguistic features, while identifying additional analyses for future work.
7 Conclusion
- The CNN-LSTM achieved a new benchmark accuracy for neural AD classification, and visualization techniques illustrated linguistic features captured by the models.The authors identify conversational context, multi-class classification across AD stages, and application to other neurological diseases as potential future work.Source: §7 · p. 5
A.1 Training Details
The appendix reports model-specific training configurations, alongside shared optimizer and hyperparameter-tuning choices.
CNN
- The CNN used filter sizes [3, 4, 5] with 128 filters per size, batch size 128, and dropout 0.80.The models shared an Adam optimizer with learning rate 1e−4, gradient clipping at 2.0, and development-set hyperparameter tuning.Source: §A.1 · p. 7
LSTM
- The LSTM used two layers with 128 hidden units, batch size 32, and dropout 0.70.The models shared an Adam optimizer with learning rate 1e−4, gradient clipping at 2.0, and development-set hyperparameter tuning.Source: §A.1 · p. 7
CNN-LSTM
- The CNN-LSTM used 100 CNN filters per size [3, 4, 5, 6], 300-dimensional embeddings, a 300-unit LSTM, and dropout 0.65 on both CNN and recurrent layers.The models shared an Adam optimizer with learning rate 1e−4, gradient clipping at 2.0, and development-set hyperparameter tuning.Source: §A.1 · p. 7
A.2 Dataset Details
The dataset section describes DementiaBank’s utterance-level samples, class distribution, and train/dev/test split.
A.2 Dataset Details
- The dataset comprised 14,362 patient utterances: 11,458 from Alzheimer’s-diagnosed interviewees and 2,904 from controls, giving an AD-positive majority baseline accuracy of 79.8%.The samples came from 1,017 Alzheimer’s transcripts and 243 control transcripts; a 80/10/10 train/dev/test split was used for each setting.Source: §A.2 · p. 7