Source-linked AI summary
Deep Learning for Depression Recognition with Audiovisual Cues: A Review
Lang He, Mingyue Niu, Prayag Tiwari, Pekka Marttinen, Rui Su, Jiewei Jiang, Chenguang Guo, Hongyu Wang, Songtao Ding, Zhongmin Wang, Wei Dang, Xiaoying Pan
TL;DR
Depression diagnosis is difficult to standardize and can be time-consuming and subjective, while prior reviews did not adequately cover deep-learning methods using combined audiovisual cues. This paper reviews databases, objective markers, and deep neural-network approaches for automatic depression estimation, then discusses challenges and future directions. Across the reviewed literature, multi-modal fusion reports the best performance on each database, although such fusion is complicated; the review also identifies limited data, unmodeled personal attributes, feature-combination challenges, and insufficient theoretical foundations for multi-modal fusion.
Problem
Depression diagnosis is time-consuming and subjective, and prior reviews inadequately covered combined audiovisual cues and deep-learning approaches.
Method
The paper comprehensively reviews databases, objective markers, and deep neural-network methods for automatic depression estimation from audiovisual cues.
Results
Multi-modal fusion yielded the optimal ADE performance on every database reviewed, including RMSE 7.03 and MAE 5.21 on AVEC2013 and AVEC2014.
Takeaways & Limitations
The review supports continued development of objective ADE tools while highlighting the complexity of fusing complementary audio and video information.
Takeaways & Limitations
ADE deep-learning methods are constrained by limited public depression databases and insufficient theoretical foundations for multi-modal fusion.
Abstract
from arXiv · showhide
With the acceleration of the pace of work and life, people have to face more and more pressure, which increases the possibility of suffering from depression. However, many patients may fail to get a timely diagnosis due to the serious imbalance in the doctor-patient ratio in the world. Promisingly, physiological and psychological studies have indicated some differences in speech and facial expression between patients with depression and healthy individuals. Consequently, to improve current medical care, many scholars have used deep learning to extract a representation of depression cues in audio and video for automatic depression detection. To sort out and summarize these works, this review introduces the databases and describes objective markers for automatic depression estimation (ADE). Furthermore, we review the deep learning methods for automatic depression detection to extract the representation of depression from audio and video. Finally, this paper discusses challenges and promising directions related to automatic diagnosing of depression using deep learning technologies.
1. Introduction
Depression poses a substantial social and clinical burden, while diagnosis remains subjective, time-consuming, and difficult to standardize. This review addresses gaps in prior surveys by examining deep-learning methods using combined audiovisual cues, relevant databases, challenges, and future directions.
- Motivation: Depression creates serious burdens, yet its diagnosis remains time-consuming, subjective, and difficult to standardize.Clinical assessments depend on patient self-report and clinician interviews, experience, and motivation.
- Motivation: Automatic Depression Estimation systems use audiovisual cues to estimate depression severity, but the most efficient biomarker remains unclear.These systems draw on techniques from machine learning, speech recognition, and computer vision.
- Research evolution: ADE feature extraction has progressed from hand-crafted to shallow and deep learning, alongside database evolution from single-modal to multi-modal data.The review frames this development across the period from 2013 to 2021.
- Review gap: Prior reviews separately covered audio or visual cues and traditional approaches, leaving deep-learning audiovisual methods inadequately discussed.The authors identify an in-depth review of multi-modal audiovisual depression recognition as missing.
- Review scope: The paper reviews deep neural-network ADE methods, introduces databases and objective markers, and discusses challenges and future research directions.Its structure covers depression assessment, databases, deep ADE architectures based on audiovisual cues, additional issues, and conclusions.
2. Current Diagnostic Methods
Depression diagnosis relies on clinical definitions, interviews, self-report scales, and emerging objective behavioral markers, but remains time-consuming, subjective, and difficult to characterize consistently.
- 2.1. The Definition of Depression: Depression is represented in Valence-Arousal space as a low-arousal, negative-valence state in the third quadrant.The VA model describes valence as positive versus negative and arousal as emotional intensity.
- 2.1. The Definition of Depression: Clinical classification includes multiple depressive disorders, including Major Depressive Disorder, Persistent Depressive Disorder, and other specified or unspecified forms.The DSM-based list also includes DMDD, PDD, substance- or medication-induced disorder, and depression due to another medical condition.
- 2.2. Diagnosing Depression: Diagnosis is difficult because clinical assessment can be time-consuming and subjective, while symptom presentation and depression profiles vary substantially.Primary-care assessment is complicated by misidentification and by patients who do not directly show depressive manifestations.
- 2.2. Diagnosing Depression: HAMD uses a 20–30 min clinician interview, whereas BDI uses a 5–10 min self-reported questionnaire to rate 21 depression-related symptoms.Both methods assign scores intended to characterize depression severity.
- 2.2. Diagnosing Depression: SRSIs can reach 80% to 90% specificity and sensitivity but remain vulnerable to reporting bias, individual variability, and weak differentiation among depression subtypes.They also do not fully account for the clinical meaning of observed symptoms.
- 2.3. Objective Makers for Depression Assessment: Objective markers from speech, facial expression, posture, gaze, and related behaviors are being investigated to support depression-severity assessment.Speech features include prosodic, source, acoustic, and vocal-tract dynamics, while visual cues include facial patterns and pupil responses.
3. Depression Databases
Depression databases combine audio, video, physiological, and text modalities, but collection requires careful clinical and experimental design and available datasets remain limited.
- 3. Depression Databases: The reviewed databases were collected across hospitals, clinics, laboratories, family interactions, and treatment studies using varied audiovisual recording settings.Collection may involve cameras, microphones, sensors, interviews, emotion elicitation, or spontaneous interaction tasks.
- 3. Depression Databases: Clinical labels were obtained using standards and instruments including DSM-IV, HAMD, MINI, QIDS-SR, HRSD, and depression rating scores.The specific diagnostic and severity-assessment procedures differed across databases.
- 3. Depression Databases: The databases include diverse participant groups, such as depressed, non-depressed, high-risk suicidal, remitted, dysthymic, and agitated or retarded subjects.Examples include family-interaction datasets involving adolescents and parent–child discussions.
- 3.2. Reviewed Databases: All reviewed databases contained limited numbers of data samples, reflecting the difficulty of collecting sensitive depression data.The paper identifies limited data availability as a challenge for training discriminative deep-learning models.
4. Deep Audiovisual Depression Recognition
Deep audiovisual depression recognition systems typically preprocess inputs, learn modality-specific deep representations, and estimate depression severity through classification or regression. Reviewed approaches use temporal modeling, attention, feature fusion, and multimodal integration, with reported performance gains but continuing data and personalization challenges.
- ADE commonly includes preprocessing, deep feature extraction, and classification or regression across audio, video, transcript, and physiological modalities.Audio preprocessing may include resampling and spectrogram generation with a Hanning window.
- Deep ADE Networks for Audio Modality: Audio models combine DCNNs for raw-waveform patterns with LSTMs for short- and long-term representations from Mel-scale filter banks.DepAudioNet also uses random sampling during training to balance positive and negative samples.
- Deep ADE Networks for Audio Modality: Time-frequency channel attention and vectorization achieved RMSEs of 8.32 on AVEC2013 and 9.25 on AVEC2014.The architecture combines time-frequency convolutional blocks with DenseNet blocks to learn informative spectrogram patterns.
- Deep ADE Networks for Audio Modality: Speech recognition architectures fuse speaker-recognition and speaker-emotion features, then use FVCM to model correlations and covariances across time-delayed channels.The review identifies feature fusion and coordination measurement as two contributions of the architecture.
- Deep ADE Networks for Visual Cues: Visual models use attention, multiscale facial regions, and spatiotemporal architectures to learn local, global, and temporal depression patterns.C3D and RNN components model tight and loose facial regions, while other systems aggregate local and global features with attention or pooling.
- Deep ADE Networks for Multi-modal Fusion: Multimodal fusion yielded the best reported performance on each database, including RMSE 7.03 and MAE 5.21 on AVEC2013 and AVEC2014, and RMSE 5.40 and MAE 4.35 on DAIC.The review notes that multimodal fusion is complicated because complementary information must be combined.
5. Open Issues and Promising Directions
The review identifies data availability, transparency, model standardization, multimodal integration, and sample imbalance as open issues for advancing ADE toward clinical use. It proposes broader data access, public code, interdisciplinary collaboration, and methods addressing skewed training distributions.
- The review targets ADE frameworks suitable for hospitals, psychiatric centers, and other non-laboratory applications.It emphasizes clinical prototypes and improving availability, extensibility, and capacity outside laboratory settings.
- Database availability: Depression databases remain scarce, sensitive, and small, limiting deep-learning research and motivating effective data augmentation.AVEC2013, for example, contains only 50 samples in each training, development, and test set.
- Transparency of data and algorithms: Sensitive datasets and limited raw-data access hinder reproducibility, so studies should share raw data or provide secure computing access.The review specifically cites DAIC-WOZ restrictions on distributing raw video clips.
- Transparency of data and algorithms: Publicly available code is needed because no commonly accepted deep-learning architecture currently exists for ADE.Shared implementations would allow researchers to validate algorithm efficiency and support clinical application.
- Interdisciplinary collaboration: Audio-only and video-only deep models capture modality-specific patterns, motivating collaboration across psychology, physiology, computer science, and machine learning.The review presents interdisciplinary collaboration as important for advancing ADE.
- Imbalanced distribution of training samples: Imbalanced training samples can impair minority-category performance and substantially affect validation and test results.In AVEC2014, the 0–9 class has more participants while the 30–44 class has fewer.
6. Conclusions
The review finds that deep learning has established promising foundations for automatic depression estimation, while clinical deployment still requires progress in data, feature learning, multimodal fusion, and diagnostic specificity.
- Deep learning-based ADE methods show promising performance and provide foundations for future clinical application.
- Future ADE systems must distinguish major depressive disorder from other depression types and learn effectively from few training samples.
- Additional data collection, broader methodological exploration, and clinical-use-case implementation remain necessary before widespread clinical application.
- Multimodal databases combining audio, video, text, and physiological signals could augment training data for ADE models.
- Future methods should account for individual attributes and data attributes when extracting informative, discriminative features for ADE.
- Robust ADE architectures should model complementary information between hand-crafted and deep-learned features, as well as across modalities.