Source-linked AI summary
The Verbal and Non Verbal Signals of Depression -- Combining Acoustics, Text and Visuals for Estimating Depression Level
Syed Arbaaz Qureshi, Mohammed Hasanuzzaman, Sriparna Saha, Gaël Dias
TL;DR
Depression is widespread and difficult to detect because many symptoms are covert. The paper introduces an attention-based network that fuses acoustic, text, and visual modalities to regress PHQ-8 depression severity on DAIC-WOZ. All three modalities gave the most accurate estimates, while the approach improved over state-of-the-art by 7.17% RMSE and 8.08% MAE.
Problem
Depression is a serious, widespread illness, and its covert symptoms make automatic detection challenging.
Method
An attention-based fusion network combines seven acoustic, text, and visual modality encodings for PHQ-8 severity regression.
Results
Fusion of all modalities gave the most accurate estimates, and the approach improved over state-of-the-art by 7.17% RMSE and 8.08% MAE.
Takeaways & Limitations
The results support combining acoustic, text, and visual information, with verbal input playing an important role in depression regression.
Abstract
from arXiv · showhide
Depression is a serious medical condition that is suffered by a large number of people around the world. It significantly affects the way one feels, causing a persistent lowering of mood. In this paper, we propose a novel attention-based deep neural network which facilitates the fusion of various modalities. We use this network to regress the depression level. Acoustic, text and visual modalities have been used to train our proposed network. Various experiments have been carried out on the benchmark dataset, namely, Distress Analysis Interview Corpus - a Wizard of Oz (DAIC-WOZ). From the results, we empirically justify that the fusion of all three modalities helps in giving the most accurate estimation of depression level. Our proposed approach outperforms the state-of-the-art by 7.17% on root mean squared error (RMSE) and 8.08% on mean absolute error (MAE).
1 Introduction
Depression is a widespread, serious illness whose covert symptoms make detection challenging. The paper introduces an attention-based multimodal network and reports that combining acoustic, text, and visual inputs improves depression-level estimation and state-of-the-art performance.
- Motivation: Depression affects more than 300 million people worldwide, while many symptoms remain covert and difficult to detect automatically.The paper motivates automated detection through depression’s global burden and diagnostic difficulty.
- Main contributions: The attention-based network fuses seven modality encodings and predicts depression severity on the PHQ-8 scale.The encodings comprise two acoustic, one text, and four visual modalities.
- Main contributions: Combining acoustic, text, and visual modalities produced the most accurate depression-level estimates among the tested combinations.The experiments evaluated multimodal combinations for depression-level regression.
- Main contributions: Verbal input played a predominant role in regression, consistent with therapists’ experience.The conclusion likewise identifies the text modality as important in the regression process.
2 Related work
Prior work examined acoustic, visual, text, and multimodal cues for automatic depression detection. This paper proposes attention-based early fusion to learn modality significance while predicting PHQ-8 severity on DAIC-WOZ.
- Existing multimodal research: Earlier studies used individual nonverbal descriptors or combined facial, head-movement, and vocal-prosody cues for depression detection.These studies explored which acoustic and visual behaviors provide useful indicators.
- Existing multimodal research: Other work fused visual, acoustic, and text features on DAIC-WOZ, establishing multimodal depression detection as an active research direction.The cited studies used feature fusion, model fusion, or decision-level strategies.
- Proposed approach: Attention-based early fusion combines acoustic, visual, and text modalities into fused representations for PHQ-8 regression on DAIC-WOZ.The approach is motivated by differences in the modalities’ discriminative characteristics.
3 Methodology
CombAtt encodes multiple time-series modalities, combines them through tensor and attention fusion, and regresses a patient’s PHQ-8 depression score. Its attention mechanism weights modality representations before final regression.
- 3 Methodology: CombAtt uses modality encoders, a fusion subnetwork, and a regression subnetwork to estimate PHQ-8 scores.The encoders produce modality representations, the fusion subnetwork combines them, and the regression subnetwork outputs the estimated score.
- 3.1 Modality encoders: LSTM encoders transform each time-series modality into an encoding vector that is passed to the fusion network.The LSTM output serves as the encoding vector for the corresponding modality.
- 3.2.1 Tensor fusion layer: The tensor fusion layer explicitly models unimodal and bimodal dynamics from pairs of modality encodings before attention fusion.It forms flattened multimodal vectors through an outer-product construction and has no learnable parameters.
- 3.2.2 Attention fusion subnetwork: The architecture uses attention because modalities may differ in relevance for estimating depression level.The attention generation subnetwork lets the deep regression network determine its parameters without constraints on the generated attention values.
- 3.2.2 Attention fusion subnetwork: The attention fusion subnetwork projects modality inputs to a common dimension, generates attention values, and combines them into a fusion vector.The resulting fusion vector is fed to the PHQ-8 regression subnetwork.
4 DAIC-WOZ depression dataset
The DAIC-WOZ depression dataset consists of interview sessions with audio, visual, voice, and transcript data, alongside PHQ-8-related participant information.
- DAIC-WOZ contains 189 interview sessions, although incomplete or interrupted interviews were discarded.
- Each session includes raw audio, facial landmarks, facial HoG features, head pose, eye gaze, facial action units, voice features, and a transcript.
- Voice features include COVAREP and formant time series recorded at 100Hz, with VUV flags identifying voiced segments.
- The dataset records participant and interviewer speech with timestamps, speaker labels, and exact utterances in the transcript.
- Training and development splits provide PHQ-8 labels, scores, participant gender, and questionnaire responses, while the test split provides interview IDs and gender.
5 Experiments and results
The experiments and results section is organized around preprocessing, experimental setup, and experimental results.
- Section 5 covers data preprocessing, experimental setup, and experimental results.
- Data preprocessing is treated as the first subsection of the experiments and results section.
- Experimental setup and experimental results form the remaining subsections of the section.
5.1 Data preprocessing
Preprocessing is modality-specific: facial features are geometrically normalized and enriched, speech is filtered and segmented by speaker turns, and transcripts are encoded as temporal vectors.
- Facial representations are Z-axis centered, normalized by average distance to the origin, and augmented with pairwise point distances.The resulting facial feature vector has size 2482 at each time step.
- Speech preprocessing removes unvoiced rows, isolates participant turns using transcript timestamps, discards turns shorter than one second, and zero-pads the sequences.
- Participant utterances are time-ordered, contractions are expanded, and each utterance is represented by a 512-D pretrained Universal Sentence Encoder vector.
5.2 Experimental setup
The experimental setup uses separately encoded acoustic, text, and visual streams, tensor fusion, attention-based modality weighting, and deep regression to estimate PHQ-8 scores.
- CombAtt uses modality encoders, fusion components, and regression subnetworks whose hyperparameters were selected empirically after searching a broad space.
- The network encodes facial landmarks, head pose, eye gaze, action units, COVAREP, formants, and transcripts with recurrent encoders producing modality-specific vectors.
- Visual and acoustic encodings are paired into three tensor-fusion groups, whose outputs are combined with the transcript encoding before attention fusion.
- The attention module receives selected modality representations, produces softmax attention values, and uses the fused output for regression.
- The experiments compare acoustic-visual and text-visual modality combinations to assess modality fusion and attention for PHQ-8 estimation.
- The regression network concatenates fused modalities with participant gender and predicts an integer PHQ-8 score from 0 to 24.
5.3 Experimental results
CombAtt is evaluated against modality combinations and prior state-of-the-art approaches using RMSE, MAE, and EVS. The full acoustic-text-visual fusion configuration performs best and significantly surpasses prior systems.
- The experiments compare modality combinations and state-of-the-art systems using RMSE, MAE, and explained variance score.
- CombAtta×t×v outperforms every other modality configuration, showing that combining acoustic, text, and visual inputs gives the best depression-level estimation.These results are statistically significant compared with all other configurations.
- The attention mechanism improves fusion by accounting for the weight of each modality in the regression process.
- 7.17% lower RMSE and 8.08% lower MAE are achieved over VFSCsemantic, the best-performing state-of-the-art approach.CombAtt outperforms all compared state-of-the-art approaches, with statistically significant improvements.
6 Conclusion
The paper introduces CombAtt, an attention-based fusion network for estimating PHQ-8 depression scores. Experiments support multimodal fusion, the importance of text, and superiority over prior state-of-the-art approaches, while leaving substantial room for improvement.
- CombAtt estimates PHQ-8 scores with an attention-based fusion network combining acoustic, text, and visual modalities.
- Experiments show that fusing all modalities gives the closest depression-level estimates, text is especially important, and CombAtt outperforms previous state-of-the-art approaches.
- The reported performance still leaves substantial room for improvement relative to a constant mean-score baseline.The cited baseline achieves RMSE=5.73 and MAE=4.74.