Source-linked AI summary
From Sparse Representations to Behavioral Insights for Multimodal Depression Assessment
Guimin Hu, Zihao Song, Jiachen Luo, Jiayuan Xie, Ruichu Cai
TL;DR
Existing multimodal depression assessment methods use dense, opaque representations that make behavioral patterns difficult to interpret. BehavDep factorizes these representations, connects factors to behavioral concepts, and estimates video-level tendencies under user-level supervision; it achieves state-of-the-art assessment performance while supporting structured behavioral analysis.
Problem
Dense multimodal representations make it difficult to interpret the behavioral patterns underlying depression-assessment predictions.
Source: §1 · p. 1Method
BehavDep factorizes multimodal behavioral representations into sparse latent factors, associates them with behavioral concepts, and estimates video-level depression tendencies under user-level weak supervision.
Source: §1 · pp. 1–2Results
BehavDep achieves state-of-the-art assessment performance, including DAIC-WOZ F1-scores of 0.87 for Normal samples and 0.74 for Depressed samples.
Source: §1 · p. 2, §3.4 · p. 5Takeaways & Limitations
BehavDep provides a structured approach to analyzing multimodal behavioral representations, including complementary modality contributions and heterogeneous behavioral patterns across observations.
Source: §5 · p. 8BehavDep is an assistive research framework, not a clinical diagnostic system, and its tendency scores should not be treated as a diagnosis or definitive measure of mental health.
Source: Ethical Considerations · p. 9
Abstract
from arXiv · showhide
Multimodal depression assessment offers a promising approach to analyzing behavioral patterns associated with depression. However, existing methods often rely on dense and opaque multimodal representations, making it difficult to interpret the behavioral patterns underlying their predictions. In this work, we introduce BehavDep, a sparse factor-based framework that decomposes multimodal behavioral representations into sparse latent factors and associates them with behaviorally meaningful concepts through a semantic bridge. To address the mismatch between user-level annotations and heterogeneous video-level behaviors, BehavDep further learns video-level depression tendency scores under weak supervision and aggregates information across multiple observations for user-level assessment. Extensive experiments demonstrate that BehavDep achieves the best overall assessment performance while revealing complementary modality contributions, heterogeneous behavioral patterns across observations, and prediction responses to concept-level editing. These results show that BehavDep provides a structured and interpretable approach to analyzing multimodal behavioral representations for depression assessment.
1 Introduction
Behavioral cues offer a basis for computational depression assessment, but dense multimodal representations make it difficult to inspect the patterns behind predictions. BehavDep decomposes these representations into sparse factors, grounds them in behavioral concepts, and estimates video-level depression tendencies under user-level supervision.
1 Introduction
- Dense multimodal representations entangle modalities and behavioral patterns, limiting insight into how learned representations organize behaviors associated with depression assessment.Observable cues such as facial expressiveness, speech dynamics, and verbal communication also provide a potential basis for computational assessment.Source: §1 · p. 1
- Because behavioral patterns may differ across observations while labels are user-level, BehavDep estimates video-level depression tendencies to characterize variation across observations.Together, the components provide a structured approach to analyzing multimodal representations and associated behavioral patterns.Source: §1 · pp. 1–2
- BehavDep decomposes dense representations into sparse latent factors without predefined semantic meanings, then associates factors with depression-related behavioral concepts.Source: §1 · p. 2
- BehavDep achieves state-of-the-art performance while enabling interpretable analysis of multimodal behavioral patterns and their variation across observations.Source: §1 · p. 2
2 Related Work
Prior work uses multimodal behavioral signals for depression assessment, while dense representations complicate analysis of captured patterns. Related mental-health AI work also uses implicit representations, and the link between behavioral concepts and multimodal representations remains underexplored.
2.1 Multimodal Depression Understanding
- Multimodal depression methods use facial, vocal, and linguistic cues, but dense representations make it difficult to disentangle patterns across modalities and observations.Source: §2.1 · p. 2
2.2 AI for Mental Health Assessment
- Mental-health AI includes instruction-tuned language models and concept-based interpretability approaches, but the relationship between concepts and multimodal behavioral representations remains underexplored.Source: §2.2 · p. 2
3 Method
BehavDep factorizes multimodal video representations, links sparse factors to behavioral concepts, and learns video-level depression tendencies under user-level supervision. Experiments report strong benchmark results alongside analyses of concept activations, longitudinal variation, and component contributions.
3.2.1 Sparse Behavioral Representation
- BehavDep uses a temporal Transformer encoder and masked pooling to create video representations, then applies a TopK sparse autoencoder to expose latent factors.TopK activation retains the highest-ranked dimensions, encouraging sparse representations.Source: §3.2.1 · p. 3
- A clinically grounded concept set gives sparse factors semantic associations, with concept activations representing each video’s behavioral patterns.Examples include reduced facial expressiveness, monotonous speech, and social withdrawal.Source: §3.2.1 · p. 3, §3.2.2 · p. 3
3.2.2 Concept-Aware Depression Assessment
- Under user-level weak supervision, BehavDep estimates video-level depression tendencies and combines them with an attention-weighted concept profile for user assessment.The soft-label bank blends historical and current estimates to provide more stable video-level supervision.Source: §3.2.2 · p. 4
3.4 Main Results
- 5.05 and 4.62 percentage points are BehavDep’s accuracy and F1-score gains over Transformerhan on MUD3, where it leads all evaluation metrics.The reported recall is also highest at 83.33%.Source: Table 1 · p. 5, §3.4 · p. 5
- BehavDep achieves the best performance for both DAIC-WOZ classes, with F1-scores of 0.87 for Normal and 0.74 for Depressed samples.Against Mental-Perceiver, its F1-score improves by 1.2% for Normal samples and 24.0% for Depressed samples.Source: Table 2 · p. 5, §3.4 · p. 5
3.5 Behavioral Concept Analysis
- Concept analysis identifies low pitch variability, low eye contact, soft speaking, and reduced expressiveness as highly activated concepts for a depressed user.The factor–concept associations provide semantic interpretations of the learned sparse factors.Source: §3.5 · p. 5, Figure 2 · p. 5
3.7 Ablation Study
- Removing the SAE reduces F1-score from 81.97% to 76.52%, and removing concept guidance yields 76.35%; removing either modality also degrades performance.Removing modality-specific concepts causes additional performance drops.Source: Table 3 · p. 6, §3.7 · p. 7
3.6 Behavioral Pattern Analysis
- Video-level tendency scores vary across a user’s observations, while concept activations show that behaviors can remain consistent or vary over time.The authors describe this analysis as exploratory because video-level scores are indirectly learned from user-level supervision.Source: §3.6 · p. 6, §3.5 · p. 6
4 Behavioral Concept Editing Analysis
The analyses examine how concept edits affect predictions, which modalities and cues contribute to assessment, and how concept patterns vary across users and observations. Joint edits shift predictions more than isolated edits, while the case study shows heterogeneous concept profiles among users with depression.
4 Behavioral Concept Editing Analysis
- Joint edits across behavioral dimensions shift predictions from Depressive to Normal more consistently than isolated edits, which usually cause modest changes without changing the outcome.The pattern suggests that predictions respond to accumulated evidence from complementary concepts rather than one dominant cue.Source: §4 · p. 7
4.1 Modal Contribution Analysis
- Audio contributes the largest share, with long pauses and reduced speech dynamics among the most influential cues; visual information provides complementary evidence.The analysis links modality-level contributions with feature-level importance.Source: §4.1 · pp. 7–8
4.2 User Study
- In the three-person case study, the non-depressed user has consistently low concept activations, while the two users with depression show different concept profiles across videos.One profile concentrates importance on several concepts; the other distributes it more broadly, illustrating heterogeneity despite a shared label.Source: §4.2 · p. 8, Figure 6 · p. 8
5 Conclusion
BehavDep decomposes multimodal behavioral representations into sparse factors linked to meaningful concepts, and experiments report improved assessment performance and behavioral insights. The longitudinal analysis is exploratory, and its scores and concepts are not clinically validated.
5 Conclusion
- BehavDep provides a structured approach to transforming opaque multimodal representations into interpretable behavioral evidence for depression assessment.It decomposes dense representations into sparse latent factors and connects them with behaviorally meaningful concepts.Source: §5 · p. 8
- Experiments report improved assessment performance, complementary modality contributions, heterogeneous and temporally varying patterns, and prediction responses to concept-level editing.Source: §5 · p. 8
Limitations
- The longitudinal behavioral analysis is exploratory, not a clinically validated characterization of depression-related behavior.Source: Limitations · p. 9
- Video-level tendency scores are indirectly learned from user-level supervision and may be affected by the distribution of observations in each user’s video history.Source: Limitations · p. 9
- Associated behavioral concepts are not clinically verified indicators; the model-generated characterizations are intended as references for fine-grained annotation, not clinical conclusions.The authors suggest finer-grained annotations and expert evaluation to further validate the relationship between identified patterns and depression-related assessments.Source: Limitations · p. 9
Ethical Considerations
The paper describes data privacy handling and frames BehavDep as an assistive research framework, not a clinical diagnostic system. It also identifies deployment risks and calls for human oversight and ethical safeguards.
Ethical Considerations
- Personally identifiable information is removed or anonymized before analysis, and data are processed in accordance with applicable ethical guidelines.Source: Ethical Considerations · p. 9
- BehavDep is an assistive research framework, not a clinical diagnostic system, and its outputs should not replace professional assessment.Weakly supervised video-level tendency scores are not clinical diagnoses or definitive measures of mental health status.Source: Ethical Considerations · p. 9
- The authors identify risks of algorithmic bias, model uncertainty, and misinterpretation, and call for human oversight and attention to privacy, fairness, and informed consent.They also note the consequences of erroneous predictions and the need to comply with relevant ethical standards.Source: Ethical Considerations · p. 9
A Depression Concept Set
The Depression Concept Set draws on clinically recognized depressive symptoms and literature on depressive symptoms, affective behaviors, and observable markers. It organizes concepts across acoustic, visual, and cross-modal categories to connect multimodal signals with higher-level interpretations.
A Depression Concept Set
- The Depression Concept Set is based on clinically recognized depressive symptoms described in the DSM-5 and literature on depressive symptoms, affective behaviors, and observable depression markers.Source: A Depression Concept Set · p. 10
- Its acoustic concepts include speech timing, energy, prosody, fluency, and engagement, while visual concepts include facial expressiveness, eye contact, gaze, posture, and psychomotor slowing.The acoustic list includes slow speech, long pauses, reduced verbal output, low vocal energy, and monotone speech.Source: A Depression Concept Set · pp. 10–11
- Cross-modal concepts capture higher-level patterns such as psychomotor retardation, emotional flattening, behavioral withdrawal, and social disengagement.Source: A Depression Concept Set · p. 11
- The literature-guided bank is intended to bridge low-level multimodal signals and higher-level clinical interpretations through semantically meaningful concepts.Source: A Depression Concept Set · p. 11
B Baselines
The baseline set spans traditional machine-learning approaches, multimodal neural architectures, and recent depression-assessment models. Its listed examples include temporal modeling, multimodal fusion, general-purpose multimodal learning, and audio-focused systems.
B Baselines
- The evaluation compares BehavDep with traditional machine-learning approaches, multimodal neural architectures, and recent depression-assessment models.Source: B Baselines · p. 11
- Dep uses Transformer-based temporal features with cross-attention and concatenation; TAMFN uses temporal attention-based cross-modal fusion; BiLSTM, Trans, and ConvLSTM model temporal or heterogeneous information.BiLSTM aggregates temporal information with attention, while Trans aggregates across time and ConvLSTM uses sub-attention.Source: B Baselines · p. 11
- Mental-Perceiver uses audio and text, PerceiverIO provides general-purpose multimodal attention and flexible querying, Qwen2-Audio-Instruct supports audio-language tasks, and AFABNet fuses acoustic features.Source: B Baselines · p. 11
C Observation-level Behavioral Heterogeneity
BehavDep scores depression tendencies per video, revealing uneven behavioral evidence across observations and variation within users. Concept importance combines model attention with semantic alignment, while PHQ-9 scores alone do not show the behavioral evidence underlying assessment.
C Observation-level Behavioral Heterogeneity
- 58.5% of 3,192 video segments score below 0.1, while 14.1% score above 0.5, indicating depression tendencies concentrate in a subset of observations.The distribution challenges the assumption that all videos from a depressed user contribute equally to assessment.Source: C Observation-level Behavioral… · p. 12
D Intra-User Variability of Depression Tendency
- Predicted video-level tendencies vary substantially within representative users, motivating aggregation across multiple observations rather than relying on one video.Different observations from the same user may contain substantially different levels of depression-related behavioral evidence.Source: D Intra-User Variability of Depression… · p. 12
E Computation of Concept Importance
- Concept importance combines model attention to each concept with semantic alignment between the sparse representation and that clinical concept.The combined measure represents each concept's overall contribution to depression prediction.Source: E Computation of Concept Importance · pp. 12–13
F Patient Health Questionnaire-9 (PHQ-9)
- PHQ-9 assesses depressive symptoms through nine scored items, but its aggregate score alone does not reveal the behavioral evidence underlying assessment.Clinical interpretation also considers observable indicators such as affective expression, speech, cognitive responses, and social behaviors.Source: F Patient Health Questionnaire-9 (PHQ-9) · p. 13
G Implementation Details
Implementation converts PHQ-9 scores to normalized severity values and uses a clinical cutoff for binary labels. Training protocols and hyperparameters are held consistent across experiments.
G Implementation Details
- Subjects are classified using the standard PHQ-9 diagnostic threshold, with identical training protocols and hyperparameter settings across experiments unless specified otherwise..Source: G Implementation Details · p. 13
- PHQ-9 totals are min-max normalized from the 0–27 range, with 0 and 1 representing minimum and maximum possible symptom severity.The original total score is denoted XPHQ9 and the normalized value Xnorm.Source: G Implementation Details · pp. 13–14
- The normalized PHQ-9 threshold for binary labels is 0.370, based on the commonly used screening cutoff of 10.The normalized score ranges from 0 to 1; it represents symptom severity rather than a direct probability of depression.Source: G Implementation Details · pp. 13–14
H Case Study
A case study illustrates how BehavDep links an individual prediction to observable behavioral evidence. Two mental health professionals found the extracted evidence aligned with the observed behaviors.
H Case Study
- Two mental health professionals independently reviewed the extracted evidence and found it well aligned with observed user behaviors.The case study leaves the final assessment to human professionals.Source: H Case Study · p. 14
- 60.5% depressive-state probability is supported primarily by acoustic concepts, including speech latency, long pauses, and reduced fluency.The report summarizes these cues as reduced speech energy and psychomotor slowing.Source: H Case Study · p. 14
- BehavDep traces its prediction to specific behavioral concepts and evidence, making the output easier to interpret and inspect.The report presents the prediction alongside influential concepts and their corresponding behavioral evidence.Source: H Case Study · p. 14