Source-linked AI summary
Depression and Self-Harm Risk Assessment in Online Forums
Andrew Yates, Arman Cohan, Nazli Goharian
TL;DR
The paper addresses self-harm-risk assessment in support forums and depression identification in general forums. It introduces a shared neural architecture and the large RSDD dataset, achieving substantially better performance than strong existing methods on both tasks. These results support using language-based models to identify depressed users and estimate self-harm risk, while the datasets and applications remain subject to privacy and representativeness constraints.
Problem
The paper addresses the need to identify posts indicating self-harm risk in support forums and depressed users in general forums.
Method
The paper combines posts with a shared neural architecture and introduces RSDD, matching over 9,000 diagnosed users with over 107,000 control users.
Results
The method substantially outperformed strong existing methods for depression detection and self-harm-risk estimation across the two datasets.
Takeaways & Limitations
The approach supports identifying depressed users and estimating the self-harm risk posed by individual posts.
Takeaways & Limitations
RSDD captures only self-reporting users, may not represent the broader population, and cannot verify whether reported diagnoses are truthful.
Abstract
from arXiv · showhide
Users suffering from mental health conditions often turn to online resources for support, including specialized online support communities or general communities such as Twitter and Reddit. In this work, we present a neural framework for supporting and studying users in both types of communities. We propose methods for identifying posts in support communities that may indicate a risk of self-harm, and demonstrate that our approach outperforms strong previously proposed methods for identifying such posts. Self-harm is closely related to depression, which makes identifying depressed users on general forums a crucial related task. We introduce a large-scale general forum dataset ("RSDD") consisting of users with self-reported depression diagnoses matched with control users. We show how our method can be applied to effectively identify depressed users from their use of language alone. We demonstrate that our method outperforms strong baselines on this general forum dataset.
1 Introduction
Depression, suicide, and self-harm create an important public-health challenge, while online forums provide support and language data for studying these concerns. The paper proposes a framework for assessing self-harm risk and identifying depression across support and general forums.
- 350 million people are estimated to suffer from depression worldwide, and suicide and self-harm are major related public-health concerns.
- Social media enables people with mental-health problems to express their experiences, seek support, and disclose concerns.
- Thousands of daily support-forum posts make manual self-harm-risk identification difficult, while people in acute distress need prompt attention.
- Depression identification in general social media can support research on mental health and language use and monitoring users’ states.
- The framework combines users’ posts, introduces the RSDD dataset, and addresses depression detection and self-harm-risk estimation.
2 Related Work
Prior work studies mental-health and self-harm discourse using relatively small datasets and predominantly engineered-feature or linear-classifier approaches. This paper instead develops a shared neural architecture that combines post representations into user-level activity representations and evaluates ordinal-risk modeling.
- Many previous approaches rely on LIWC, topic models, manual lexicons, or other domain-dependent features that may omit additional predictive signals.
- Related self-harm research commonly uses linear classifiers with feature engineering, including sparse and manually designed features.
- The proposed CNN-based architecture learns representations of users’ posts and combines them into an overall representation of user activity.
- The ordinal-loss model for green, amber, red, and crisis labels is more robust than the paper’s other variants.
3 Data
The paper constructs RSDD from users with self-reported depression diagnoses and matched controls, and uses ReachOut’s annotated Triage data for self-harm assessment. Ethical safeguards address anonymity and restricted dataset access.
- 3.1 Depression dataset construction.: Potential control users had no mental-health subreddit posts or depression-related terms, reducing the likelihood of including depressed users as controls.
- 3.1 Depression dataset construction.: Each diagnosed user was matched with 12 controls using subreddit-post probability distributions and activity-level restrictions.
- 3.1 Depression dataset construction.: RSDD is an order of magnitude larger than prior self-reported-diagnosis datasets, but captures only self-reporting users, may not represent the whole population, and cannot verify truthfulness.
- 3.2 Self-harm assessment.: The ReachOut Triage dataset contains 65,024 forum posts, including 1,227 annotated for self-harm risk across green, amber, red, and crisis labels.
- 3.2 Self-harm assessment.: The shared architecture processes inputs through convolutional layers, merges them into a user-activity vector, and classifies that representation through dense and output layers.
- 3.3 Ethical concerns.: The datasets involve sensitive mental-health data; RSDD uses public Reddit posts and restricted researcher access, while ReachOut data require anonymity and redact identifying information.
4 Methodology
The paper develops a shared neural architecture for classifying users from one or more posts, adapting its inputs, merging strategy, and output or loss functions to depression detection and self-harm risk assessment. Convolutional networks extract post features, which are merged into user representations for classification.
- Shared architecture: The architecture processes one or more posts with convolutional and pooling layers, merges their features into a user-activity representation, and applies dense layers for classification.Convolutional filters detect features in sliding windows; the resulting representation is processed with dropout before the final output layer.
- Depression detection: Depression detection processes each post with convolutional networks, averages each network’s output, and merges post representations with a second convolutional layer.The second convolutional layer was more stable than a second average-pooling or max-pooling layer in the authors’ experiments.
- Task inputs: Self-harm risk classification uses a target post together with prior posts in its thread, while depression detection uses Reddit users’ post histories.Thread history supplies contextual information for estimating the target post’s self-harm risk; depression detection excludes mental-health-keyword posts and mental-health-related subreddits.
- Output and loss functions: The models use categorical cross-entropy with softmax for classification, mean squared error with a linear output as an alternative, and metric-learning objectives for representation-based classification.Metric-learning variants compare user-activity representations with representations of the four self-harm severity labels; the Class Metric objective minimizes distance to the correct class.
- Metric-learning variants: Class Metric enforces a margin between the correct and randomly selected incorrect class, while Class Metric (Ordinal) scales that margin with the distance between ordinal classes.Classification uses similarity or distance between the learned activity representation and class representations.
5 Experiments
The experiments evaluate the neural models for depression detection and self-harm risk assessment against established baselines under task-specific validation and test protocols. The proposed models improve key depression and self-harm metrics, while training–test differences indicate that robustness varies across loss functions.
- Experimental setup: The experiments tune model hyperparameters using validation data and evaluate depression detection on a held-out Reddit test set.Self-harm models use 10-fold cross-validation on the ReachOut training posts, with 15% of each fold reserved for validation.
- Depression detection: Additional LDA-topic and χ2 feature-selection features do not improve performance beyond the evaluated feature sets.TF-IDF sparse weighting performs better than other weighting schemes in the reported experiments.
- Depression detection: The CNN outperforms depression-detection baselines by 41% in recall and 16% in F1 on diagnosed users, but has lower precision.The comparison includes FastText, multinomial Naive Bayes, and SVM baselines using bag-of-words and feature-rich representations.
- Self-harm risk classification: The proposed self-harm models perform up to 19% better than the best existing methods on non-green F1, the official CLPsych ’16 metric.They also improve up to 11% on all-category macro-averaged F1 and up to 5% on all-category accuracy; Categorical Cross Entropy performs best across these measures.
- Self-harm risk classification: Class Metric (Ordinal) is more robust across self-harm test and training evaluations, while Categorical Cross Entropy and MSE trade places between datasets.The authors report no statistically significant difference between Class Metric (Ordinal) and the best-performing method on either dataset.
6 Analysis
The analysis examines input-selection choices for Reddit depression classification, privacy-preserving interpretation of influential phrases, and performance on the CLPsych ’17 self-harm task. Results favor random post selection and show that distinguishing crisis-level labels remains difficult.
- 6.1 Posts per user and post length: F1 is maximized at nterm = 100 and npost = 1750, but npost = 1500 is used for efficiency because gains from 1250 to 1750 are small.Approximately 20% of users exceed 1500 posts, while approximately 40% of posts exceed 100 terms.
- 6.2 Post selection: Randomly selecting posts performs best across metrics at npost = 1500, improving precision substantially over earliest-post selection and slightly over latest-post selection.Selecting earliest posts performs worst regardless of npost.
- 6.3 Phrases contributing to classification: The analysis publishes only one top phrase per correctly identified depressed user to reduce re-identification risk, rather than publishing Reddit posts or paraphrases.The authors found that paraphrased posts could remain identifiable through Web searches.
- 6.3 Phrases contributing to classification: The model identifies phrases often associated with negative sentiment or outlook, while classification decisions depend on phrase occurrence across many posts by the same user.Terms directly related to depression or diagnoses are absent by dataset design.
- 6.4 CLPsych ’17 shared task: On CLPsych ’17, all methods perform substantially worse than on CLPsych ’16 for non-green, urgent, and overall F1, although method trends remain similar.Categorical Cross Entropy performs best on non-green and overall F1, with no statistically significant difference between Class Metric (Ordinal) and the best method.
- 6.4 CLPsych ’17 shared task: The decline is attributed to difficulty distinguishing amber, red, and crisis labels as crisis prevalence rose to 11% in the 2017 test data.Methods rarely classify posts as crisis; improving crisis identification is left for future work.
7 Conclusion
The paper presents a neural architecture and the RSDD dataset for depression detection and self-harm risk classification in social media. The approach substantially outperforms strong existing methods, while depression detection remains challenging in absolute metric terms.
- The paper describes a neural network architecture for self-harm risk classification and depression detection on social media posts.
- RSDD contains over 9,000 users with self-reported depression diagnoses matched with over 107,000 similar control users.The dataset is available under a data usage agreement.
- The approach substantially outperforms strong existing methods on RSDD in Recall and F1, although the absolute metrics show depression detection remains challenging.
- The approach substantially outperforms strong previously proposed methods for estimating self-harm risk on ReachOut.com support-forum posts.
- The work supports social-media self-harm risk identification, large-scale depression studies, and possible clinical-care notifications when patient activity suggests self-harm risk.Large-scale datasets can complement generally smaller existing mental-health datasets.