Source-linked AI summary
Topic Modeling Based Multi-modal Depression Detection
Yuan Gong, Christian Poellabauer
TL;DR
Depression detection from long interviews must preserve temporally specific multimodal cues and their conversational context rather than averaging across the entire recording. The paper organizes audio, video, and semantic features by interview topic and applies feature selection; it reports significantly better performance than context-unaware methods and the challenge baseline for all metrics.
Problem
Long interview recordings contain useful short-term details whose context can be lost when features are averaged, while depression requires an interview-level decision.
Method
The approach segments interviews by topic, places separate audio, video, and semantic features into topic-specific vector slots, and applies two-step feature selection.
Results
The proposed approach performs significantly better than context-unaware methods and the challenge baseline for all metrics.
Takeaways & Limitations
Topic modeling can provide the basis for context-aware multimodal analysis of long depression-detection interviews.
Takeaways & Limitations
The evaluation cannot provide baseline test-set results because the 2017 AVEC challenge limited the number of permitted test attempts.
Abstract
from arXiv · showhide
Major depressive disorder is a common mental disorder that affects almost 7% of the adult U.S. population. The 2017 Audio/Visual Emotion Challenge (AVEC) asks participants to build a model to predict depression levels based on the audio, video, and text of an interview ranging between 7-33 minutes. Since averaging features over the entire interview will lose most temporal information, how to discover, capture, and preserve useful temporal details for such a long interview are significant challenges. Therefore, we propose a novel topic modeling based approach to perform context-aware analysis of the recording. Our experiments show that the proposed approach outperforms context-unaware methods and the challenge baselines for all metrics.
1 INTRODUCTION
Depression detection from long, multimodal interviews must preserve short-term behaviors together with their conversational context. The paper proposes topic-organized multimodal features and feature selection to address this challenge.
- Background: The AVEC task predicts depression levels from audio, video, and text in 7–33-minute interviews, requiring one decision for each entire interview.The DAIC-WOZ corpus contains 189 subjects, while the training set has only 107 samples, including 30 depression samples.
- Challenges: Averaging short-term features across an interview can discard informative details such as sighs, laughter, or anger and their conversational context.Examples include sighing when asked about sleep quality or laughing while discussing a journey.
- Method: The proposed scheme segments interviews by topic, generates audio, video, and semantic features per segment, and stores each topic in a separate feature-vector slot.This organizes details across utterances while keeping the feature space smaller than utterance-level organization.
- Contribution: Topic-specific feature combinations allow the model to distinguish the same behavior in different contexts, such as smiling while discussing family versus greeting someone.The scheme therefore supports finer analysis of reactions to specific topics.
- Related Work: The work extends topic modeling beyond text by applying it to context-aware audio, video, and text analysis.The authors describe this as the first effort, to their knowledge, to combine topic modeling with all three modalities.
2.1 Topic Modeling
The paper uses simple topic modeling tailored to the structured interviewer dialogue, then finds that topics are sparsely distributed across interviews.
- 2.1 Topic Modeling: Topic modeling exploits Ellie’s fixed, question-led topic openings instead of requiring sophisticated general-purpose algorithms.The authors build and manually clean a sentence dictionary, then cluster sentences that start the same topic.
- 2.1 Topic Modeling: The extracted topic dictionary is used to identify topics discussed in each interview.Topic-starting sentences are clustered after removing redundant or non-topic-starting utterances.
- 2.1 Topic Modeling: Only 14 topics cover over 80% of interviews, indicating that topics are sparsely distributed.The topic-cover-rate distribution is shown in Figure 2.
2.2 Features
The model computes audio, video, and semantic features separately for each topic, combining temporal statistics with subject-speech information.
- 2.2 Features: Audio, video, and semantic features are all computed in a topic-wise fashion for the multimodal model.Audio and video features come from the AVEC organizers, while semantic features are extracted by the authors.
- 2.2 Features: Each topic contributes 237 audio features from COVAREP and formants summarized by mean, maximum, and minimum.The audio features are extracted every 10 ms before topic-level statistical aggregation.
- 2.2 Features: Each topic contributes 60 video features from 20 facial action units summarized by mean, maximum, and minimum.The action-unit features are extracted using OpenFace.
- 2.2 Features: Each topic contributes 93 LIWC semantic features based on word-presence frequencies in the subject’s speech.Additional semantic features are extracted for selected key topics, including sleep quality and PTSD diagnosis history.
2.3 Topic-wise Feature Mapping
The proposed representation maps multimodal information into topic-specific feature slots, preserving topic context while organizing the interview into a fixed feature vector.
- 2.3 Topic-wise Feature Mapping: Each topic receives a separate feature-vector slot to support context-aware analysis.Speech provides semantic features, while timestamps synchronize the corresponding audio and video features.
- 2.3 Topic-wise Feature Mapping: The mapping records topic-specific subject speech and timestamps before placing all modalities into the corresponding topic slot.The topic-wise mapping preserves the association between speech content and synchronized audio-video features.
2.4 Feature Selection
Because topic-wise feature generation creates many correlated features, the method uses correlation-aware filtering followed by model-dependent selection to reduce redundancy and overfitting risk.
- 2.4 Feature Selection: The 32,462-dimensional representation contains many features that may be unhelpful, making feature selection essential.The authors aim to retain a smaller feature set to reduce potential overfitting.
- 2.4 Feature Selection: The first selection stage uses correlation-based feature subset selection to favor predictive features with low mutual redundancy.CFS is model-independent and evaluates feature subsets using predictive ability and inter-feature redundancy.
- 2.4 Feature Selection: The second stage uses model-dependent selection to determine an appropriate feature count for the specific model and dataset.This stage addresses overfitting risk that model-independent CFS cannot assess directly.
- 2.4 Feature Selection: Features within the same topic are often highly correlated, so selecting only by individual feature scores could retain redundant information.The two-step design responds to stronger within-topic correlations than in context-unaware feature generation.
2.5 Regression Model Building
The study addresses class imbalance and evaluates several regression models through grid search.
- Random oversampling addresses the imbalance in DAIC-WOZ, where 30 of 107 training subjects are depressed.
- The grid search evaluates random forest, stochastic gradient descent, and support vector regression models.Random forest varies tree count, while support vector regression tests linear, polynomial, and RBF kernels.
3 EXPERIMENTAL SETUP
The experiments use multiple test strategies, metrics, and baselines to evaluate context-aware depression regression while limiting development-set overfitting.
- Test Strategy: The study uses 10-fold stratified cross-validation, development-set testing, and official test-set evaluation.Cross-validation combines training and development data; the latter two strategies use parameters selected through cross-validation.
- Test Strategy: Cross-validation performs splitting before random oversampling and model-dependent feature selection, but CFS selection uses the full combined dataset.The authors describe this as the fairest available testing procedure while acknowledging that CFS can produce an over-optimistic estimate.
- Evaluation Metrics: RMSE is the challenge target, while MAE and Pearson correlation coefficient provide additional regression measures.Model selection and feature selection are optimized using RMSE.
- Baselines: The comparisons include a mean-prediction baseline, the official challenge baseline, and a context-unaware baseline.The challenge baseline averages frame-wise audio-video regression outputs across the interview, whereas the context-unaware baseline otherwise matches the proposed pipeline without topic modeling.
4 EVALUATION AND DISCUSSION
The proposed method performs best across reported evaluation settings and benefits from context-aware, two-step feature selection, although its estimates have stated evaluation limitations.
- Overall Performance: The proposed method achieves the best performance for all metrics and test strategies, outperforming the context-unaware and challenge baselines.Its test-set performance remains much higher than the challenge baseline despite the stated CFS evaluation issue.
- Regression Model and Feature Number Analysis: The selected model uses 46 features and an SGD regressor chosen through cross-validation grid search.
- Selected Feature Analysis: RMSE 4.99 is achieved with the proposed two-step feature selection, compared with RMSE 5.60 for the individual-score baseline on the test set.The two-step procedure is designed to retain label-correlated features while reducing redundancy among correlated features.
- Selected Feature Analysis: The selected feature set spans LIWC, key topic semantic, COVAREP audio, and AUs video features.
- Evaluation Limitations: Test-set results for baseline approaches cannot be provided because the challenge limited the number of test attempts.The challenge baseline paper also lacks test results for CC and F1-score and does not report cross-validation results.
- Regression Model and Feature Number Analysis: 46 features yield RMSE 4.99 on the test set, compared with RMSE 5.22 for 41 features.The authors select 46 features despite the lowest cross-validation RMSE occurring at 41, treating that result as a likely fluctuation.
5 CONCLUSIONS
The paper presents topic modeling as a context-aware way to preserve temporal information in multimodal depression detection. Experiments report better performance than context-unaware and challenge baselines across all metrics.
- The proposed approach segments interviews by topic, builds multimodal topic-specific features, and applies feature selection to preserve useful temporal details.
- The proposed approach performs significantly better than the context-unaware method and challenge baseline for all metrics.
- The selected features include varied temporal information related to depression that the authors describe as difficult for humans to model.