Source-linked AI summary
Multi-level Attention network using text, audio and video for Depression Prediction
Anupama Ray, Siddharth Kumar, Rutvik Reddy, Prerana Mukherjee, Ritu Garg
TL;DR
Depression assessment lacks straightforward diagnostic tests and is affected by subjective responses. The paper proposes a multi-level attention network that fuses audio, video, and text while learning feature and modality relevance. The all-feature fusion model outperforms the baseline by 17.52% in RMSE.
Problem
Subjective assessment and the lack of straightforward diagnostic tests motivate automated depression prediction from behavioural cues.
Method
A multi-level attention network fuses audio, video, and speech-transcription features while selecting important intra- and inter-modality features.
Results
17.52% improvement over the baseline is achieved by the all-feature fusion network.
Takeaways & Limitations
Attention provides feature- and modality-importance information while supporting depression-severity prediction across individual and fused models.
Takeaways & Limitations
The visual modality is limited to eye gaze, facial action units, and head pose, while body posture was unavailable from the used video features.
Abstract
from arXiv · showhide
Depression has been the leading cause of mental-health illness worldwide. Major depressive disorder (MDD), is a common mental health disorder that affects both psychologically as well as physically which could lead to loss of lives. Due to the lack of diagnostic tests and subjectivity involved in detecting depression, there is a growing interest in using behavioural cues to automate depression diagnosis and stage prediction. The absence of labelled behavioural datasets for such problems and the huge amount of variations possible in behaviour makes the problem more challenging. This paper presents a novel multi-level attention based network for multi-modal depression prediction that fuses features from audio, video and text modalities while learning the intra and inter modality relevance. The multi-level attention reinforces overall learning by selecting the most influential features within each modality for the decision making. We perform exhaustive experimentation to create different regression models for audio, video and text modalities. Several fusions models with different configurations are constructed to understand the impact of each feature and modality. We outperform the current baseline by 17.52% in terms of root mean squared error.
1 INTRODUCTION
The paper addresses subjective, unreliable depression assessment by using behavioural cues and proposes multi-level attention across audio, video, and text. Its best all-feature fusion improves on the baseline by 17.52%.
- Motivation: Subjective questionnaire responses and the absence of straightforward diagnostic tests make early depression assessment difficult.The passage notes that individual symptom presentation can vary substantially, contributing to misdiagnosis.
- Motivation: Behavioural cues such as facial expression and speech prosody motivate automated prediction of depression and related disorders.
- Approach: The proposed framework applies attention at several layers to select important audio, video, and speech-transcription features for depression-level prediction.
- Results: 17.52% improvement over the baseline is achieved by the best-performing all-feature fusion network.
- Evaluation: The paper evaluates individual modality features and multimodal combinations to examine feature and modality contributions.
2 RELATED WORK
Prior work uses speech, facial, textual, and multimodal behavioural signals for distress and depression analysis. This paper positions multi-level attention as a way to estimate feature and modality importance while improving computational efficiency.
- Speech: Speech analysis is motivated by clinical use of prosody and by speech’s noninvasive, non-intrusive recording properties.
- Visual analysis: Depression-related visual research examines facial expressions and temporal video patterns as indicators of emotional and mental-health states.
- Text analysis: Text-based approaches use social-media content, sentiment, lexical features, and behavioural patterns to classify distress or depression levels.
- Multimodal approaches: Existing multimodal work includes fusion techniques, computational-linguistic models, context-aware features, and deep neural networks.
- Positioning: The proposed approach applies attention within modalities and across modalities to learn relative feature and modality importance.
- Positioning: Multiple attention levels are reported as improving results over the closest prior attention-based approach while reducing computational expense and test time.
3 PROPOSED ATTENTION NETWORK FOR HYBRID FUSION
The network creates modality-specific context features with attention, fuses them through feedforward and recurrent layers, and regresses depression severity. Experiments use clinical interview recordings from E-DAIC and its defined train, development, and test partitions.
- Network architecture: Attention over each modality selects important features to create a modality-specific context representation.
- Network architecture: Three modality context representations pass through feedforward networks before fusion in a stacked BLSTM.
- Network architecture: A further attention layer combines the fused representation with the stacked Bi-LSTM output before regression.
- Dataset: E-DAIC contains audio-video clinical interviews for psychological distress conditions including anxiety, depression, and post-traumatic stress disorder.
- Dataset: The dataset has 275 subjects, with 163 used for training and 56 each for development and testing; test labels are unavailable.
4 METHODOLOGY
The methodology builds modality-specific models and combines audio, video, and text through multi-level attention-based fusion networks. Attention selects influential intra-modality features and weights modality contributions for depression-score regression.
- Model construction: The study creates separate regression models for audio, video, text, and multiple within- and across-modality fusion configurations.The methodology includes experiments on individual features and fused modality combinations.
- Text modality: Text processing uses speech-to-text transcripts, pretrained Universal Sentence Encoder embeddings, zero padding, and a two-layer stacked BLSTM.Sentence embeddings have dimension 512, with 400 timesteps used for constant tensor size.
- Audio modality: Audio models use low-level descriptors and functionals, including MFCC, eGeMAPS, Bag-of-AudioWords, and deep DenseNet features.The models cover both acoustic descriptors and higher-level functional or learned representations.
- Video modality: Video experiments compare low-level descriptors with their functionals and use low-level features because they retain more information for sequence modeling.The video features include gaze, pose, and facial action units in the fusion experiments.
- Attention-based fusion: Multi-layer attention learns feature importance within modalities and modality contribution during fusion, supporting early fusion while addressing computational and overfitting concerns.Fusion networks combine attention outputs through BLSTM, feed-forward, and regression components.
5 RESULTS
The unimodal experiments show that text performs best overall, audio features consistently improve on the baseline, and video improves on comparison systems but remains weaker than text and speech.
- Text modality: Text achieved the best results among modalities, reaching RMSE 4.37 on development and RMSE 4.73 with MAE 4.02 on the challenge test set.The test results also included CCC 0.67, PCC 0.676, r2 0.457, and SCC 0.651.
- Audio modality: 29.80% improvement over baseline was obtained with the audio MFCC model, while eGEMAPS, BoAW-MFCC, and BoAW-eGE improved by 29.04%, 10.44%, and 14.46%.These comparisons use RMSE on the challenge development split.
- Visual modality: Video features outperformed the baseline and state of the art, with Bag of Visual Words performing best among visual features and exceeding the baseline by 4.8%.Relative to [32], pose, gaze, and facial action units improved by 9.3%, 6.6%, and 8.7%, respectively.
- Cross-modality comparison: Video results remained worse than those obtained from text and speech modalities despite improving over the baseline and state of the art.
6 RESULTS FROM FUSED MODALITIES
The all-feature, multi-level attention fusion model achieves the strongest fused-modality result against the baseline. Comparisons with prior work vary by fusion combination and are not fully conclusive because the datasets differ.
- All-modality fusion: 17.52% improvement over baseline was achieved by the all-feature fusion network using multiple levels of attention.The model fuses audio, video, and text features for PHQ-8 regression.
- Comparison with prior work: Audio-text and video-text fusion outperformed [32] by 5.8% and 9.19%, respectively, whereas the all-feature fusion network was slightly worse.The paper states that this comparison is not conclusive because the datasets differ slightly.
- Attention interpretation: The attention mechanism weighs features within each modality and learns their relationship with PHQ-8 scores for regression decisions.
7 DISCUSSIONS AND FUTURE WORK
The multi-level attention model improved depression prediction across individual and fusion models, while assigning the greatest weight to text and similar weights to audio and video. Its video contribution remains constrained by the limited behavioral features available compared with face-to-face clinical observation.
- Discussion: The attention network assigned the highest weight to text and nearly equal weights to audio and video.The authors relate text’s higher weighting to clinicians’ emphasis on speech content.
- Limitations: Video received lower importance because the model used limited features such as eye gaze, facial action units, and head pose.Clinical interviews can additionally capture body posture and electrophysiological signals.
- Results: 17.52%: All-feature fusion with multi-level attention outperformed the baseline.The model fused audio, video, and text modalities for depression-severity prediction.
- Results: Multi-level attention improved both individual and fusion models compared with the baseline and state-of-art.Attention also reduced computational complexity and training and test time.