Source-linked AI summary
SMHD: A Large-Scale Resource for Exploring Online Language Usage for Multiple Mental Health Conditions
Arman Cohan, Bart Desmet, Andrew Yates, Luca Soldaini, Sean MacAvaney, Nazli Goharian
TL;DR
Mental-health research needs large-scale labeled language data from diagnosed users across multiple conditions. The paper constructs SMHD using high-precision self-report patterns and matched Reddit controls, then examines language and classification. SMHD provides a large multi-condition resource, with FastText performing best overall among the tested classifiers.
Problem
Large-scale labeled datasets of diagnosed users across multiple mental health conditions are needed to study mental-health language.
Method
The paper constructs SMHD from Reddit users identified by high-precision self-reported diagnosis patterns, matched with controls, and evaluates linguistic, psychological, and text-classification methods.
Results
SMHD covers diverse mental health conditions and matched controls, while FastText achieves the best classification performance overall, except for bipolar and eating disorder.
Takeaways & Limitations
SMHD supports research on language differences across conditions and on identifying diagnosed users through language, while making the resource available to the research community.
Takeaways & Limitations
Control-user availability is constrained because stringent matching excludes users who lack subreddit overlap, post volume, or mental-health-term restrictions, and controls are sampled without replacement.
Abstract
from arXiv · showhide
Mental health is a significant and growing public health concern. As language usage can be leveraged to obtain crucial insights into mental health conditions, there is a need for large-scale, labeled, mental health-related datasets of users who have been diagnosed with one or more of such conditions. In this paper, we investigate the creation of high-precision patterns to identify self-reported diagnoses of nine different mental health conditions, and obtain high-quality labeled data without the need for manual labelling. We introduce the SMHD (Self-reported Mental Health Diagnoses) dataset and make it available. SMHD is a novel large dataset of social media posts from users with one or multiple mental health conditions along with matched control users. We examine distinctions in users' language, as measured by linguistic and psychological variables. We further explore text classification methods to identify individuals with mental conditions through their language.
1 Introduction
The paper addresses the shortage of large-scale labeled mental-health language data by introducing SMHD, a Reddit dataset built from self-reported diagnoses and matched controls. It also studies language differences across conditions and classification of diagnosed users.
- Social media provides accessible mental-health-related language because users can discuss, seek advice, and disclose sensitive matters anonymously or pseudo-anonymously.
- SMHD uses high-precision matching patterns to identify self-reported diagnoses for nine mental health conditions without manual labeling.
- Reddit supplies long-form discussion without Twitter’s character limit, while the dataset includes up to two orders of magnitude more diagnosed individuals per condition than the compared Twitter resource.
- The study normalizes users’ language by removing specific mental-health signals and examines general-discussion language differences between conditions and controls.
- The dataset contains users with multiple conditions, enabling analysis of interacting mental conditions alongside language characteristics within each group.
- The authors explore classification methods for detecting users with mental health conditions through social-media language.
2 Related work
Prior work used social-media and other text sources to study mental-health language, but existing efforts were limited in scale, platform constraints, or condition coverage. This work extends that approach with long-form Reddit data, broader diagnosis patterns, and nine conditions.
- Earlier studies used Twitter diagnosis statements, crisis forums, essays, and crisis-center text messages to examine mental-health language and prediction.
- Twitter-based studies offered insight into mental-health language but were constrained by short messages, whereas long-form content can provide additional linguistic information.
- The work builds a large Reddit corpus from self-reported diagnoses, avoiding character limits and producing language more typical of general writing.
- Unlike prior work focused on depression, the study uses extended synonym-derived matching patterns and investigates nine mental health conditions.
3 Data
SMHD constructs a large Reddit dataset by identifying self-reported diagnoses for nine mental health conditions and matching diagnosed users with controls. The dataset preserves multi-condition labels, supports language comparisons, and includes privacy safeguards.
- Dataset construction: Diagnosed users were filtered using negative diagnosis patterns and a requirement for at least 50 non-mental-health posts.Mental-health posts were identified through relevant subreddits or mental-health-related language.
- Dataset construction: Controls were selected from users with no mental-health posts and at least 50 posts, matched using subreddit overlap and posting volume.Stringent exclusions reduced the available control pool, and matching was performed without replacement.
- Dataset characteristics: SMHD contains 20,406 diagnosed users and 335,952 matched controls, while SMHD-RC adds 16,542 diagnosed users with fewer than nine controls.SMHD-RC is intended for analyses comparing differences between mental-health conditions rather than diagnosed users and controls.
- Dataset construction: SMHD uses DSM-5 condition headings, expanded with synonyms, misspellings, vernacular terms, and abbreviations, to build diagnosis patterns.The patterns combine self-reported diagnosis matching with condition-keyword matching within a specified character distance.
- Dataset characteristics: 26.7% of diagnosed users reported multiple diagnoses; depression commonly co-occurs with other conditions, while schizophrenia most often co-occurs with bipolar disorder.Depression, ADHD, anxiety, and bipolar disorder are the most common diagnosed conditions in the dataset.
- Dataset statistics: Reddit provides long-form content, but controls average twice as many posts as diagnosed users and those posts are considerably shorter.Classification experiments can reduce this potential signal by truncating post length and number.
4 Analysis
The analysis compares diagnosed and control users through LIWC-based linguistic and psychological measures, finding condition-related differences across both language style and psychological attributes. Social, self-focused, authentic, and cognitive language patterns distinguish several diagnosed groups from controls.
- Language analysis: LIWC categories capture both linguistic style and psychological attributes for comparing each condition group with controls.The comparisons use user-level LIWC categories and Welch’s t-tests.
- Overall differences: Effect sizes for diagnosed–control language differences range from very small to medium, with 0.05<d<0.5.The reported effects span both linguistic style and psychological attributes.
- Linguistic style: Control users show greater clout than depression and anxiety users, while authentic language is stronger among most diagnosed groups.Clout indicates language associated with high social status; authenticity correlates with personal and first-person pronoun use.
- Social and cognitive processes: Most diagnosed groups use more first-person singular pronouns than controls, alongside stronger cognitive-process language.The analysis also reports more female references among bipolar, depression, and anxiety users, with a possible corpus gender bias noted.
- Social and cognitive processes: Social-process language is higher among depressed, anxiety, and PTSD users than controls.The category covers language related to human interaction and sharing.
5 Experiments
The experiments evaluate classical and neural classifiers on binary and multi-label, multi-class condition-identification tasks. FastText performs best overall, but performance is strongly related to training-set size and sampling choices.
- Experimental setup: Classifiers are evaluated in binary and multi-label, multi-class settings for identifying diagnosed users across conditions.Binary models distinguish diagnosed from control users; the multi-label setting predicts which conditions apply to a user.
- Classification results: FastText achieves the best F1 score across all conditions except bipolar and eating disorder, and leads in the multi-label, multi-class setting.SVM outperforms FastText on bipolar, while CNN outperforms it on eating disorder.
- Classification results: Bag-of-words models favor precision over recall, whereas FastText and CNN provide more balanced performance.The paper attributes the bag-of-words pattern to stronger weighting of indicative features and possible training-data overfitting.
- Data-size effects: All classifiers struggle on OCD, schizophrenia, and eating disorder, the three conditions with the fewest diagnosed users.Across conditions, model performance is strongly affected by the number of diagnosed users in the training set.
- Model comparison: FastText outperforms CNN, which samples only a portion of each user’s posts for efficiency.The result motivates developing sampling strategies that remain robust across all SMHD conditions.
6 Conclusion
The paper presents SMHD, a large Reddit dataset covering diverse mental health conditions with matched controls. It analyzes language differences, evaluates classification methods, and releases the dataset to support further research and reproducibility.
- Dataset: SMHD contains Reddit users with diverse mental health conditions and matched control users.The dataset is constructed with high-precision diagnosis patterns and carefully selected controls.
- Dataset: SMHD is reported as the largest dataset supporting a variety of mental health conditions and is up to two orders of magnitude larger than the largest similar resource.The paper compares its scale with previously published mental-health resources.
- Analysis and classification: The study examines linguistic and psychological differences between diagnosed and control groups and evaluates several text-classification methods.FastText is reported as the most effective approach overall.
- Impact: The released dataset is intended to foster further research and enhance reproducibility of proposed approaches.