Source-linked AI summary
Measuring Depression Symptom Severity from Spoken Language and 3D Facial Expressions
Albert Haque, Michelle Guo, Adam S Miner, Li Fei-Fei
TL;DR
The paper addresses the difficulty and time required to assess depressive symptom severity amid barriers to mental health care. It proposes a multimodal sentence-based model using clinical-interview audio, facial scans, and text, and evaluates it against word-level embeddings and prior work.
Problem
Depressive symptom assessment is time-intensive, while stigma, cost, and limited treatment availability contribute to unmet mental health needs.
Source: §1 · p. 1Method
Results
The experiments compare the sentence-based model with word-level embeddings and prior work on depressive symptom severity measurement.
Source: §1 · p. 2Takeaways & Limitations
The authors present a combination of speech recognition, computer vision, and natural language processing that they hope will inspire AI tools for mental health disorders beyond depression.
Source: §5 · p. 4The work is intended to augment clinical methods, not issue a formal depression diagnosis, and its interviews were human-to-computer rather than human-to-human.
Source: §5 · p. 4
Abstract
from arXiv · showhide
With more than 300 million people depressed worldwide, depression is a global problem. Due to access barriers such as social stigma, cost, and treatment availability, 60% of mentally-ill adults do not receive any mental health services. Effective and efficient diagnosis relies on detecting clinical symptoms of depression. Automatic detection of depressive symptoms would potentially improve diagnostic accuracy and availability, leading to faster intervention. In this work, we present a machine learning method for measuring the severity of depressive symptoms. Our multi-modal method uses 3D facial expressions and spoken language, commonly available from modern cell phones. It demonstrates an average error of 3.67 points (15.3% relative) on the clinically-validated Patient Health Questionnaire (PHQ) scale. For detecting major depressive disorder, our model demonstrates 83.3% sensitivity and 82.6% specificity. Overall, this paper shows how speech recognition, computer vision, and natural language processing can be combined to assist mental health patients and practitioners. This technology could be deployed to cell phones worldwide and facilitate low-cost universal access to mental health care.
1 Introduction
Depression symptom assessment is important but time-intensive, while stigma, cost, and treatment availability leave many people without care. The paper proposes measuring symptom severity from multimodal clinical-interview data to support screening and access.
1 Introduction
- 60% of mentally-ill adults do not receive treatment, amid barriers including social stigma, financial cost, and lack of accessible treatment options.More than 300 million people worldwide are depressed.Source: §1 · p. 1
- Clinicians assess verbal and non-verbal symptoms during interviews, and the clinically validated PHQ measures symptom severity across personal dimensions.Severity assessment is time-intensive and important for initial diagnosis and tracking improvement.Source: §1 · p. 1
- The authors envision smartphone-based multimodal sensing or text messages for timely, cost-effective symptom screening and actionable feedback for individuals and clinicians.The proposed feedback may include visual, audio, and linguistic signals.Source: §1 · p. 2
- The proposed method uses interview audio, 3D facial keypoints, and speech transcription to predict a PHQ score or major depressive disorder label.It uses a causal convolutional network to summarize sentences into an embedding for symptom-severity prediction.Source: §1 · p. 2
2 Dataset
The study uses DAIC-WOZ clinical interviews with PHQ-8 scores, audio, and 3D facial scans. The authors describe privacy protections in this dataset and flag privacy considerations for future applications to other datasets.
2 Dataset
- DAIC-WOZ provides audio, 3D facial scans, and PHQ-8 scores from semi-structured interviews conducted through a remote-controlled digital avatar.The avatar asks questions aimed at identifying depressive symptoms and provides conversational feedback.Source: §2 · p. 2
- The study reports data collected from 189 clinical interviews involving 142 patients; following prior work, its reported results are from the validation set.Source: §2 · p. 2
- The dataset excludes protected health information, but researchers applying the method to other datasets should account for possible PHI.Names, specific dates, and locations were removed; the 68-pixel facial scans retain motion information without enough detail to identify individuals.Source: §2 · p. 2
3 Model
The model combines audio, 3D facial scans, and text in a sentence-level embedding, then uses a causal convolutional network to predict depression outcomes. Sentence-level modeling is intended to capture longer-term multimodal signals than word- or phoneme-level embeddings.
3 Model
- The proposed sentence-level embedding is designed to capture long-term acoustic, visual, and linguistic elements that word- and phoneme-level embeddings model only over narrow temporal contexts.Those contexts are often a few hundred milliseconds at most.Source: §3 · p. 2
- The model takes audio, 3D facial scans, and text, and feeds a multimodal sentence embedding to a depression classifier and PHQ regression model.The model is described as a causal CNN [5].Source: Figure 2 · p. 3
- Causal convolutional networks are selected to model long interview sequences, which can be extended by patients’ stutters and pauses between words.The passage notes prior findings that C-CNNs outperform RNNs on long sequences [5].Source: §3 · p. 3
4 Experiments
The experiments compare the proposed approach with prior work and examine the effects of sentence-level embeddings and modalities. The supplied passages describe the evaluation design but do not report comparative scores.
4 Experiments
- The experiments evaluate PHQ-score regression and binary major depressive disorder classification, then study the effects of modalities and sentence-level embeddings.The classification label typically corresponds to a PHQ score of at least ten.Source: §4 · p. 3
4.1 Automatically Measuring the Severity of Depressive Symptoms
- The prior-work comparison contrasts a method that accepts sentences without interview-context metadata and uses raw audio, visual, and text inputs with approaches relying on context or engineered features.The passage notes that context may help but can leave too few training examples per contextual class.Source: §4.1 · pp. 3–4
4.2 Ablation Study
- The ablation compares averaged hand-crafted or pretrained sentence embeddings with learned sentence embeddings built using LSTM or causal convolutional networks.The learned sentence-level representations take word-level features as input.Source: §4.2 · p. 4
5 Discussion
The discussion bounds the method's scope: it uses avatar-mediated interviews and PHQ severity scores rather than human-to-human interviews or formal diagnosis. It also identifies bias in pretrained embeddings and proposes future work on longitudinal information and interpretability.
5 Discussion
- The study uses human-to-computer interviews, and PHQ symptom severity is not equivalent to a formal depression diagnosis.The authors frame the method as augmenting existing clinical methods, not issuing a formal diagnosis.Source: §5 · p. 4
- Pre-existing embeddings may contain bias from their training data, and mitigating that bias is outside this work's scope.The authors identify culturally sensitive diagnosis and treatment as a reason bias mitigation matters.Source: §5 · p. 4
- Future work could use longitudinal symptom scores across sessions and investigate predictions with visualizations of facial confidence and audio-segment usefulness.Source: §5 · p. 4
- The conclusion presents a multimodal approach combining speech recognition, computer vision, and natural language processing, with the hope it inspires tools beyond depression.Source: §5 · p. 4
A.1 Data Format
The data-format description specifies the audio, video, and facial-keypoint recordings, the available train-validation split, and text normalization applied to transcripts.
A.1 Data Format
- Audio was recorded at 16 kHz, video at 30 frames per second, and 68 three-dimensional facial keypoints were extracted using OpenFace.Source: §A.1 · p. 7
- The dataset split used 107 training patients and 35 validation patients; although a test set exists, its labels are not public.Source: §A.1 · p. 7
- Transcripts were normalized by expanding slang, converting text to lowercase, and spelling out numbers.Examples include “bout” to “about” and “24” to “twenty four.”Source: §A.1 · p. 7
A.2.1 Experiment 1: Automatically Measuring the Severity of Depressive Symptoms
The experiment uses audio, visual, and linguistic inputs with a causal convolutional network; the appendix specifies the feature formats and training setup.
A.2.1 Experiment 1: Automatically Measuring the Severity of Depressive Symptoms
- The model takes log-mel spectrograms with 80 mel filters, 68 three-dimensional facial keypoints, and Word2Vec embeddings as inputs.Source: §A.2.1 · p. 7
- The causal CNN has 10 layers, kernel size 5, and 128 hidden nodes per layer.Dropout is 0.5; classification uses binary cross entropy, while regression uses mean squared error.Source: §A.2.1 · p. 7
- Training used Adam for 100 epochs with batch size 16, with learning rates of 1e-3 for classification and 1e-5 for regression.The model was trained on a single Nvidia V100 GPU.Source: §A.2.1 · p. 7
A.2.2 Experiment 2: Ablation Studies
The ablation appendix describes alternative input representations and network variants used for Table 2, including pretrained or handcrafted embeddings and learned sentence-level models.
A.2.2 Experiment 2: Ablation Studies
- Table 2 includes log-mel, MFCC, facial-keypoint, Word2Vec, Doc2Vec, and Universal Sentence Embedding representations, with vector lengths of 13, 300, or 512 where specified.Word2Vec and Doc2Vec vectors are 300-dimensional; Universal Sentence Embeddings are 512-dimensional, and MFCCs yield 13 values.Source: §A.2.2 · p. 7
- The listed LSTM baseline consists of 10 layers with 128 hidden units, and the causal CNN is the same as the model outlined in Appendix A.2.1.Source: §A.2.2 · p. 7
- Public code implementations were used for the core LSTM and causal CNN architecture components.Source: §A.2.2 · p. 7