Source-linked AI summary
To BERT or Not To BERT: Comparing Speech and Language-based Approaches for Alzheimer's Disease Detection
Aparna Balagopalan, Benjamin Eyre, Frank Rudzicz, Jekaterina Novikova
TL;DR
Traditional AD diagnosis is costly and time consuming, motivating speech-based detection methods. This paper compares domain-informed hand-crafted features with fine-tuned BERT models on the ADReSS dataset and also evaluates feature-based MMSE regression. Fine-tuned BERT achieved the strongest reported AD detection result, while feature-based regression produced the reported MMSE outcome.
Problem
AD diagnosis is time consuming and expensive, motivating speech-based approaches for detecting cognitive impairment.
Method
The paper compares domain-knowledge speech features with fine-tuned BERT transcript classifiers on ADReSS and evaluates feature-based models for MMSE regression.
Results
83.3% test accuracy was achieved by fine-tuned BERT versus 81.3% for the best feature-based model, while feature-based L2-regularized regression achieved MMSE RMSE 4.56.
Takeaways & Limitations
Fine-tuned BERT models outperformed hand-crafted feature engineering for AD classification, and combining both representations is proposed for future work.
Takeaways & Limitations
Hand-crafted feature approaches require time-consuming engineering and risk missing highly relevant features.
Abstract
from arXiv · showhide
Research related to automatically detecting Alzheimer's disease (AD) is important, given the high prevalence of AD and the high cost of traditional methods. Since AD significantly affects the content and acoustics of spontaneous speech, natural language processing and machine learning provide promising techniques for reliably detecting AD. We compare and contrast the performance of two such approaches for AD detection on the recent ADReSS challenge dataset: 1) using domain knowledge-based hand-crafted features that capture linguistic and acoustic phenomena, and 2) fine-tuning Bidirectional Encoder Representations from Transformer (BERT)-based sequence classification models. We also compare multiple feature-based regression models for a neuropsychological score task in the challenge. We observe that fine-tuned BERT models, given the relative importance of linguistics in cognitive impairment detection, outperform feature-based approaches on the AD detection task.
1. Introduction
The paper investigates speech-based AD detection as a faster, less costly alternative to traditional diagnosis, comparing domain-informed features with fine-tuned BERT models on ADReSS. It also evaluates feature-based models for MMSE regression.
- Traditional AD diagnosis is time consuming and expensive, while spontaneous speech can provide clinical information relevant to cognition.Prior speech-analysis methods reported 82%-93% accuracy on related datasets.
- The study uses the demographically matched ADReSS speech dataset to develop models for binary AD classification and MMSE score regression.
- The domain-knowledge approach extracts linguistic transcript features and acoustic audio features for both tasks.The features are informed by previous clinical and machine-learning research on cognitive impairment detection.
- The transfer-learning approach fine-tunes pretrained BERT text-classification models on speech transcripts to evaluate high-level language representations for AD detection.
- 83.3% test accuracy was achieved by fine-tuned BERT, compared with 81.3% for the best feature-based model, while MMSE prediction reached an RMSE of 4.56 with feature-based L2-regularized regression.
- The paper contrasts the two approaches and discusses reasons for their differing classification performance.
2. Background
Prior work uses acoustic and linguistic speech features for AD detection and MMSE prediction, while this paper frames feature engineering and transfer learning as complementary modeling approaches with different trade-offs.
- Earlier speech-based AD studies used acoustic measures and transcript-derived linguistic features, reporting 82%-93% classification accuracy.
- Previous work predicted MMSE scores from picture-description speech features with mean absolute error between 2.91 and 3.83.
- Pre-engineered features can make model decisions more interpretable and may require fewer resources with conventional machine-learning models.
- Feature engineering is time consuming and may omit highly relevant features.
- BERT uses bidirectional contextual representations and attention mechanisms, but fine-tuned models are not directly interpretable like feature-based models.
3. Dataset
The study uses the ADReSS Challenge dataset, a balanced collection of English speech samples and transcripts from participants with and without AD, elicited through a standardized picture-description task.
- The dataset contains 156 speech samples from 78 non-AD and 78 AD English-speaking participants.
- Speech was elicited using the Cookie Theft picture from the Boston Diagnostic Aphasia exam.
- ADReSS is matched for participant age and gender and divided into standard train and test sets.
- MMSE scores are available for all but one participant in the training set.
4. Feature Extraction
The feature-based pipeline uses participant speech from manually transcribed picture descriptions and combines transcript and audio information into manually engineered linguistic, acoustic, and semantic features.
- Only participant speech segments are used, and segments from each picture description are combined for acoustic-feature extraction.
- 509 manually engineered features are extracted from transcripts and associated audio files to encode domain knowledge.
- The feature set includes 297 lexico-syntactic features, covering parsing, speech graphs, lexical norms, lexical richness, part-of-speech patterns, and utterance similarity.
- The acoustic category contains 187 features, including MFCCs, fundamental frequency, zero-crossing statistics, and pause measures.
- The semantic category contains 25 features representing information-content units relevant to memory impairment in picture descriptions.
5. Experiments
The experiments compare conventional models using engineered linguistic, semantic, and acoustic features with transcript-level fine-tuned BERT for AD classification and feature-based regression. Evaluation uses cross-validation and held-out ADReSS data, with task-specific metrics for classification and MMSE prediction.
- Domain knowledge-based approach: Conventional models classify transcript-level linguistic, semantic, and acoustic features for AD detection and MMSE regression.The models include SVM, neural network, random forest, and naïve Bayes for classification, plus linear and ridge regression for MMSE prediction.
- Model optimization: Feature selection and model hyperparameters are optimized through cross-validation grid search.Classification uses top-k features selected by ANOVA F-values, while regression selects features using correlations with MMSE and jointly tunes ridge regularization.
- Transfer learning-based approach: Fine-tuned BERT initializes a transcript-level binary classifier from pretrained language representations.A classification layer maps the final BERT representations to AD versus non-AD labels.
- Evaluation: 10-fold cross-validation compares BERT with feature-based classifiers, while LOSO cross-validation is used for the other models.BERT is evaluated with 10-fold cross-validation because GPU-memory constraints prevent LOSO cross-validation; reported 10-fold metrics average three random-seed runs.
- Evaluation: Classification is evaluated primarily with accuracy, alongside precision, recall, specificity, and F1 for AD.Accuracy is emphasized because the train and test sets are balanced.
- Evaluation: Regression performance is reported with RMSE and MAE, with MMSE predictions constrained to the 0–30 score range.Training results use LOSO cross-validation, and selected models also produce RMSE results on the ADReSS test set.
6. Results
BERT performs best overall for AD classification, including on the unseen test set, while engineered-feature models achieve the reported MMSE regression result. The BERT advantage in cross-validation is not statistically significant for accuracy, and feature selection substantially improves SVM performance.
- AD vs non-AD Classification: BERT outperforms all domain knowledge-based models across the reported classification metrics in 10-fold cross-validation.SVM is the strongest feature-based model, but BERT’s accuracy is not significantly higher than SVM’s (H = 0.4838, p > 0.05).
- AD vs non-AD Classification: +30% accuracy is achieved over challenge baselines by the best model under LOSO cross-validation.Feature selection increases the SVM classifier’s accuracy by about 13%.
- AD vs non-AD Classification: BERT outperforms the best feature-based classifier, SVM, on the unseen held-out challenge test set.The held-out test-set trend follows the cross-validated accuracy pattern.
- MMSE Regression: 4.56 RMSE is achieved by ridge regression with 25 selected features during LOSO cross-validation on the training set.This is a 2.7-point decrease from the challenge baseline; the same ridge regressor reaches 4.56 RMSE on the ADReSS test set, 1.6 below baseline.
- MMSE Regression: Feature selection reduces ridge-regression error by up to 1.5 RMSE points and 0.86 MAE.The reported improvements concern ridge regression in the MMSE prediction task.
7. Discussion
The discussion attributes BERT’s classification strength to the prominence of linguistic information in this dataset, while regression analyses likewise identify linguistic features as especially distinguishing for MMSE severity. The tables document feature differences, classification comparisons, and regression outcomes across evaluation settings.
- Analysing AD Detection Performance Differences: 87 features differ significantly between AD and non-AD speech at p < 0.05, including 79 text-based and 8 acoustic features.The acoustic features include measures related to long pauses, while the text features are lexico-syntactic and semantic.
- Analysing AD Detection Performance Differences: BERT outperforms the best domain knowledge-based model across classification metrics on train and test settings, although train-set accuracy is not significantly higher.No significance testing is possible for the single score available per model on the held-out test set.
- Analysing AD Detection Performance Differences: BERT may perform well because its language representations capture important lexico-syntactic and semantic information in transcripts.The paper presents this as a hypothesis based on the observed importance of linguistic features, not as a demonstrated causal mechanism.
- Analysing AD Detection Performance Differences: Table 3 reports 10-fold cross-validation results averaged across three random-seed runs, with BERT’s accuracy higher but not significantly different from SVM’s.The comparison uses the ADReSS training set.
- Analysing AD Detection Performance Differences: Table 4 reports LOSO cross-validation results averaged across three random-seed runs, with SVM significantly outperforming NN in accuracy.The reported Kruskal-Wallis result is H = 4.50, p = 0.034.
- Analysing AD Detection Performance Differences: 10-fold and LOSO cross-validation can yield different classification trends, potentially because the small dataset changes training-set size across folds.The discussion contrasts N_train = 98 for 10-fold cross-validation with N_train = 107 for LOSO cross-validation.
- Regression Weights: All ten highly weighted MMSE-prediction features are linguistic, and their regression-weight signs match their MMSE correlations.This indicates that these features retain consistent relationships with MMSE even alongside other regressors.
- Analysing AD Detection Performance Differences: Table 5 presents results on the unseen held-out ADReSS test set in the challenge baseline’s reporting format.The table marks the best result in bold.
8. Conclusions
The paper compares domain-knowledge-based feature engineering with fine-tuned BERT classification for AD detection. It concludes that fine-tuned BERT models outperform hand-crafted feature engineering, while suggesting feature fusion as future work.
- Fine-tuned BERT classification models outperform explicit domain-knowledge-based feature engineering for AD detection.The paper identifies combining BERT and hand-crafted features as a possible future direction.
- The comparison covers two widely used approaches: explicit feature engineering and transfer learning with fine-tuned BERT.
- Feature-fusion models combining BERT representations with hand-crafted features could potentially improve cognitive impairment detection.
A. List of features
The paper reports test-set performance through organizer-provided scores and documents the feature inventories in separate lexico-syntactic, acoustic, and semantic tables.
- Test-set performance scores for each prediction set are reported in Tables 5 and 6.
- The feature inventory is organized into lexico-syntactic, acoustic, and semantic features with brief descriptions and subtype counts.
- Table 7 summarizes the extracted lexico-syntactic features and gives the number of features in each subtype.
- Table 8 summarizes the extracted acoustic features and gives the number of features in each subtype.
- Table 9 summarizes the extracted semantic features and gives the number of features in each subtype.
B. Hyper-parameter Settings
The models use tuned conventional classifiers and neural-network settings, with hyperparameters selected by cross-validation. A t-SNE visualization uses 13 significantly different features and shows strong class separation in the feature representations.
- Hyperparameter tuning: Hyperparameters were tuned with grid search and 10-fold cross-validation on the ADReSS challenge training set.
- Random forest: The random forest uses 200 decision trees, considers √features at each split, and requires two samples for internal-node splits and leaves.
- Gaussian Naive Bayes: The Gaussian Naive Bayes classifier uses balanced priors and a variance smoothing coefficient of 1e −10.
- SVM and neural network: The SVM uses an RBF kernel with γ = 0.001 and regularization parameter 100, while the neural network has two layers of 10 units.
- SVM and neural network: The neural network uses ReLU hidden layers, Adam optimization, 200 epochs, and a batch size equal to the training-set size in each fold.
- t-SNE visualization: The t-SNE plot uses 13 features significantly different between classes to visualize class separation.
- t-SNE visualization: The feature-based representations show strong class separation, suggesting that a non-linear model could achieve good AD detection performance.
D. Test Performance Metrics
Test-set evaluation sends predictions from selected classification and regression models to the challenge organizer. The supplied passages also identify tables summarizing lexico-syntactic, acoustic, and semantic feature inventories.
- Test-set performance metrics are obtained through a procedure described before reporting the results.
- The team submits predictions from five AD classification models: SVM, NN, RF, NB, and BERT.
- The team also submits predictions from five linear regression models for the prediction task.
- Table 7 summarizes lexico-syntactic features and reports the number of features in each subtype.
- Table 8 summarizes acoustic features and reports the number of features in each subtype.
- Table 9 summarizes semantic features and reports the number of features in each subtype.