Source-linked AI summary
MFCC-based Recurrent Neural Network for Automatic Clinical Depression Recognition and Assessment from Speech
Emna Rejaibi, Ali Komaty, Fabrice Meriaudeau, Said Agrebi, Alice Othmani
TL;DR
The paper addresses automatic depression detection and severity assessment from speech despite limited Speech Depression Recognition data and imperfect clinical identification. It proposes an MFCC-based recurrent framework with data augmentation and transfer learning, achieving 76.27% validation accuracy with 0.4 root mean square error, while added visual or acoustic features improve accuracy to 95.6% or 86%.
Problem
Limited depression speech data and only half-time clinical identification motivate automatic, speech-based depression assessment.
Method
An MFCC-based LSTM recurrent neural network processes speech for depression classification and severity prediction, with augmentation and emotion-task transfer learning addressing limited data.
Results
76.27% validation accuracy and 0.4 root mean square error were achieved for depression assessment; adding visual features reached 95.6% and adding other acoustic features reached 86%.
Takeaways & Limitations
MFCC-based high-level features contain relevant depression information, while additional visual or acoustic features further improve classification results.
Takeaways & Limitations
Visual features raise patient privacy concerns, whereas additional acoustic features increase computation time and visual integration makes real-time application more difficult.
Abstract
from arXiv · showhide
Clinical depression or Major Depressive Disorder (MDD) is a common and serious medical illness. In this paper, a deep recurrent neural network-based framework is presented to detect depression and to predict its severity level from speech. Low-level and high-level audio features are extracted from audio recordings to predict the 24 scores of the Patient Health Questionnaire and the binary class of depression diagnosis. To overcome the problem of the small size of Speech Depression Recognition (SDR) datasets, expanding training labels and transferred features are considered. The proposed approach outperforms the state-of-art approaches on the DAIC-WOZ database with an overall accuracy of 76.27% and a root mean square error of 0.4 in assessing depression, while a root mean square error of 0.168 is achieved in predicting the depression severity levels. The proposed framework has several advantages (fastness, non-invasiveness, and non-intrusion), which makes it convenient for real-time applications. The performances of the proposed approach are evaluated under a multi-modal and a multi-features experiments. MFCC based high-level features hold relevant information related to depression. Yet, adding visual action units and different other acoustic features further boosts the classification results by 20% and 10% to reach an accuracy of 95.6% and 86%, respectively. Considering visual-facial modality needs to be carefully studied as it sparks patient privacy concerns while adding more acoustic features increases the computation time.
1. Introduction
The paper motivates speech-based depression recognition as a non-invasive response to substantial clinical burden and limitations in current assessment. It positions MFCC-based deep recurrent modeling as a focused approach for automatic depression assessment.
- Clinical motivation: Depression is persistent and can reach extreme severity, including suicide risk, while affecting 350 million people globally.The reported annual economic burden is 210 billion US Dollars, driven mainly by absenteeism and reduced workplace productivity.
- Assessment gap: Clinical evaluations rely on self-reports and questionnaires such as the PHQ, whose score describes depression severity from 0 to 23.Doctors correctly identify depression only half the time even when patients self-report symptoms.
- Speech-based assessment: Speech is attractive for depression recognition because it can be measured cheaply, remotely, non-invasively, and non-intrusively.The study therefore focuses on detecting depression using speech recordings only.
- Feature motivation: MFCCs have shown high efficiency for depression detection compared with other audio features in shallow-based approaches.The paper uses MFCCs as the basis for a deep recurrent approach to assess depression.
2. Related work
Prior work spans hand-crafted acoustic features and deep-learning architectures, with speech, visual cues, and multimodal fusion used for depression assessment. Reported results vary by feature set, classifier, architecture, segmentation, and participant characteristics.
- Approach families: Deep-learning-based approaches are reported to outperform hand-crafted-feature approaches in automatic depression recognition.The literature commonly organizes depression assessment into preprocessing, feature extraction, dimension reduction, and classification.
- Deep learning: 1.46 root mean square error was reported for PHQ-8 severity prediction using a DCNN-DNN model on a group of depressed men.The reviewed deep architectures include DCNN, DCNN-DNN, and LSTM.
- Experimental choices: Depression prediction varies with segmentation, including 20s windows shifted by 1s and 60ms windows shifted by 10ms.Studies also compare feature fusion, score fusion, and decision fusion methods across classifiers.
- Hand-crafted features: 81.61% accuracy was achieved by an SVM with GMM using decision fusion, whereas GMM with feature fusion achieved 48.26%.Five classifiers were compared, including GMM, SVM variants, MLP, and HFS.
- Deep learning: A recurrent neural network previously achieved 52% for detecting depression and 70% for detecting non-depression.The literature also reports that gender affects model performance, with substantially different errors for depressed women and men.
3. Proposed Method
The proposed method preprocesses participant speech, extracts and normalizes MFCC features, expands limited training data through augmentation and transfer learning, and feeds sequences into an LSTM-based RNN. The model supports both binary depression detection and severity prediction.
- Method overview: The method uses MFCCs and an LSTM-based recurrent neural network to assess depression and predict its severity levels.MFCC features are extracted from preprocessed speech and fed into the deep neural network.
- Preprocessing: Clinical interviews are separated by speaker so that only participants’ speech segments are retained for analysis.Interviewer audio segments are no longer used.
- Low-level features: MFCC coefficients are extracted from preprocessed recordings using 2.5s windows at 500ms intervals, then transformed into cepstral feature vectors.The supplied method description also reports DFT, logarithmic amplitude spectra, Mel-scale components, and KL transformation.
- Data normalization: Z-score normalization subtracts the mean from each Mel-scale coefficient and divides by the standard deviation.Normalization is applied across coefficients rather than separately for each 60ms audio frame.
- High-level modeling: The MFCC-based RNN receives an n by 60 matrix, where n is the number of extracted 60ms audio frames and 60 is the number of MFCCs.The architecture uses successive LSTM layers followed by Dense layers.
- Prediction tasks: The network uses separate output designs for binary PHQ-8 classification and PHQ-8 score prediction.The binary output uses two sigmoid-activated cells, while score prediction uses 24 softmax-activated neurons.
- Data expansion: Data augmentation diversifies preprocessed speech inputs, while transfer learning addresses limited depression data by pretraining on emotion recognition and fine-tuning on depression recognition.The augmentation methods include noise injection, pitch, shift, and speed augmentation.
4.1. Datasets
The study uses DAIC-WOZ as its primary depression corpus, with RAVDESS for emotion-transfer pretraining and AVi-D for cross-dataset evaluation.
- DAIC-WOZ provides 189 clinical-interview recordings labeled with PHQ-8 scores and binary depression status, although 182 recordings are used technically.Recordings average 15 minutes and use a 16 kHz sampling rate.
- DAIC-WOZ is gender-balanced but class-imbalanced, with non-depressed participants occurring three times as often as depressed participants.After preprocessing, audio segments are split into 80% training, 10% validation, and 10% testing.
- RAVDESS supplies speech-emotion recordings from 24 actors expressing eight emotions across speaking and singing tasks.Its recordings average five seconds.
- AVi-D contains 300 recordings from 292 participants performing Northwind reading and Freeform question-answering tasks for generalization evaluation.Approximately 150 recordings are available per task.
4.2. Implementation details
The implementation processes 60 ms audio frames into MFCC sequences and feeds them through an LSTM-based recurrent architecture with task-specific output layers. Training uses regularization, adaptive optimization, augmentation, and transfer-learning configurations.
- The model receives an n-by-60 MFCC matrix from 60 ms audio frames and uses three LSTM layers followed by dense layers.The three LSTM layers have 40, 30, and 20 output units.
- The network applies tanh and hard-sigmoid LSTM activations, recurrent dropout, batch normalization, weight initialization, bias regularization, and dense-layer dropout.The recurrent dropout and dense-layer dropout are reported as 0.2%.
- The binary task uses a two-cell sigmoid output for PHQ-8 depression status, whereas severity prediction uses 24 softmax neurons for PHQ-8 scores.
- Optimization uses Adam with an adaptive learning rate decreasing from 10^-3 to 10^-10, while RMSE is used instead of cross-entropy for comparison with prior work.
- The selected batch size is 130, and prediction takes 2 ∗10^-3 seconds despite training requiring five days.
- The transfer-learning setup pretrains on emotion recognition, freezes the first three LSTM layers, and retrains the final dense layers for depression assessment.
4.3. Experimental results
Experiments evaluate binary depression assessment and 24-class severity prediction, including baseline, augmentation, and imbalance-aware analyses. Performance improves with augmentation, while class imbalance limits depression detection.
- The framework evaluates PHQ-8 binary classification and prediction of 24 depression-severity classes.Results are reported per sample derived from a 60 ms audio frame.
- 67.61% validation accuracy and 0.5 RMSE are achieved by the baseline MFCC-based RNN on depression assessment.Validation accuracy slightly exceeds training accuracy at the end of training.
- 0.168 RMSE is achieved for severity prediction, three times better than depression-assessment RMSE and ten times better than the study benchmark.
- 74% validation accuracy and 0.42 RMSE are reached after data augmentation, representing a 6.39% accuracy increase and a 0.08 loss reduction.The improvement is attributed to increasing and diversifying the input data.
- After augmentation, 66% of samples are correctly identified as non-depression, while depression recall is 26% and its F1 score reaches 38%.The non-depression F1 score reaches 84%.
4.3.3. Transfer Learning evaluation
Transfer learning improves depression assessment beyond data augmentation by pretraining the MFCC-based RNN on an independent emotion-recognition task. The resulting evaluation reports higher accuracy and depression F1 performance.
- Transfer Learning evaluation: Depression F1 score reaches 46% after transfer learning, increasing by 8% from the 38% obtained with data augmentation.The recall of depression increases by 9%, and 69% of samples predicted as depression are correctly classified.
- Transfer Learning evaluation: The model is pretrained on RAVDESS emotion recognition and then fine-tuned on DAIC-WOZ depression assessment.The first three LSTM layers are frozen while the final dense layers are retrained.
- Transfer Learning evaluation: Pretraining improves target-task performance without increasing the number of depressed-participant inputs.The paper attributes this to learning more complex and abstract features in early layers.
4.3.4. Gender e
The MFCC-based RNN was evaluated under noisy validation conditions and across participant groups. Noise caused a modest performance reduction, while further increases in noise did not produce additional deterioration.
- Noise robustness: 8%: Adding Gaussian noise to the dataset slightly reduced MFCC-based RNN performance.Noise was added to portions of, and then the whole, 20% validation set.
- Noise robustness: The model’s depression classification ability declined more than its non-depression classification ability under noise.
- Noise robustness: Performance remained stable when the amount of validation noise increased further.
- Noise robustness: The proposed network performed better than the baseline trained without data augmentation or knowledge transfer.
4.3.6. Generalization of the MFCC-based RNN model to other dataset
The MFCC-based RNN was trained on DAIC-WOZ using PHQ-8 binary labels and tested on AVi-D using BDI-II binary labels. Generalization accuracy declined, although depression identification slightly improved in one cross-dataset comparison.
- Cross-dataset evaluation: 56%: The best classification accuracy in the AVi-D generalization experiment was achieved on the Freeform task.
- Cross-dataset evaluation: 35%: The depression F1 score reached this value for both tasks combined on the new dataset.
- Cross-dataset evaluation: 20.43%: Generalization accuracy dropped relative to the transfer-learning result.
- Cross-dataset evaluation: 12.69%: The model correctly identified this share of depressed samples when classifying BDI-II binary labels across both tasks.
- Cross-dataset evaluation: Different recording conditions and depression-test thresholds constrained direct transfer from DAIC-WOZ to AVi-D.The AVi-D Freeform and Northwind tasks involved actors who talked and sang.
4.3.8. Discussion
The discussion weighs the efficient, non-intrusive MFCC speech framework against the performance gains and costs of adding visual or additional acoustic features. Both extensions improved validation results, but they introduced privacy or computational trade-offs.
- Baseline framework: 76.27%: The MFCC-based RNN achieved this overall validation accuracy with an RMSE of 0.4 for depression assessment.
- Multi-modal experiment: 19.33%: Adding visual facial action units increased validation accuracy to 95.6% and reduced validation loss to 0.18.
- Multi-modal experiment: The visual-feature concatenation doubled the depression F1 score to 94% and increased the non-depression F1 score by 11% to 96%.
- Multi-features experiment: 86%: Adding 53 extra acoustic features increased validation accuracy by almost 10%.
- Multi-features experiment: Adding acoustic features increased the depression F1 score by 29% to 75%, while computational complexity also increased.
- Trade-offs: Visual features improved performance but were described as intrusive and invasive, making real-time application more difficult.
5. Conclusion and future works
The study proposes an MFCC-based recurrent neural network for depression detection and severity assessment from speech. It combines LSTM-based high-level feature extraction with data augmentation and knowledge transfer to address limited training data and overfitting.
- Proposed approach: The proposed system extracts and normalizes MFCC features before feeding them into successive LSTM layers in a deep recurrent neural network.
- Data strategy: Training-data augmentation and knowledge transfer from a related task are used to address limited data and overfitting.