Source-linked AI summary
MentalBERT: Publicly Available Pretrained Language Models for Mental Healthcare
Shaoxiong Ji, Tianlin Zhang, Luna Ansari, Jie Fu, Prayag Tiwari, Erik Cambria
TL;DR
Mental health disorders and suicidal ideation create a need for early detection from social content, but pretrained language models specialized for mental healthcare were unavailable. The paper trains and releases MentalBERT and MentalRoBERTa on mental-health Reddit discussions and evaluates them across detection and classification benchmarks. Domain-specific pretraining generally performs better than general-corpus pretraining, while English-only data and unresolved fairness, bias, uncertainty, and interpretability issues limit the scope.
Problem
Pretrained language models customized for mental healthcare were unavailable despite the need to detect mental disorders and suicidal ideation from social content.
Method
The paper trains and releases MentalBERT and MentalRoBERTa using continued masked-language pretraining on mental-health social-forum data, then fine-tunes them for downstream detection tasks.
Results
Domain-specific mental-health pretraining generally outperforms general-corpus pretrained models across most evaluated cases and improves downstream mental-health classification performance.
Takeaways & Limitations
The released models provide publicly available pretrained language resources for mental-health research and practical downstream studies.
Takeaways & Limitations
The study focuses only on English and leaves bias, fairness, uncertainty, and interpretability issues for future evaluation.
Abstract
from arXiv · showhide
Mental health is a critical issue in modern society, and mental disorders could sometimes turn to suicidal ideation without adequate treatment. Early detection of mental disorders and suicidal ideation from social content provides a potential way for effective social intervention. Recent advances in pretrained contextualized language representations have promoted the development of several domain-specific pretrained models and facilitated several downstream applications. However, there are no existing pretrained language models for mental healthcare. This paper trains and release two pretrained masked language models, i.e., MentalBERT and MentalRoBERTa, to benefit machine learning for the mental healthcare research community. Besides, we evaluate our trained domain-specific models and several variants of pretrained language models on several mental disorder detection benchmarks and demonstrate that language representations pretrained in the target domain improve the performance of mental health detection tasks.
1 Introduction
Mental health disorders and suicidal ideation motivate early detection from social content. Because no pretrained language models were customized for mental healthcare, the paper introduces domain-specific models trained on mental-health discussions.
- Motivation: Mental disorders affect people worldwide, while many people with severe disorders do not receive treatment.The passage reports that 1 in 4 people worldwide experience mental disorders to some extent and 3 out of 4 people with severe disorders do not receive treatment.
- Motivation: Social media provides anonymous spaces for discussing stigmatic topics and self-reporting mental health issues.Reddit and Twitter content has been used to study mental health problems and suicidal ideation.
- Motivation: Machine learning-based detection can support healthcare workers in early detection and assessment for proactive prevention.
- Research gap: Contextualized pretrained language models have encouraged development of domain-specific text representations for downstream NLP tasks.BERT uses bidirectional transformer-based encoders trained on large-scale corpora, motivating later domain-specific models.
- Contribution: The paper addresses the absence of pretrained language models customized for mental healthcare by training and releasing two masked language models.The models use corpus collected from social forums for mental health discussion.
- Contribution: The models are named MentalBERT and MentalRoBERTa and are evaluated on several mental health detection datasets.The paper also releases the pretrained models through Hugging Face’s model repository.
2 Methods and Setup
The study follows standard BERT and RoBERTa pretraining protocols using Reddit mental-health discussions, then fine-tunes the models for mental-disorder classification tasks. Its setup combines domain-adaptive masked-language pretraining with downstream neural classification.
- Scope: The study provides pretrained text embeddings and evaluates their usability in downstream tasks rather than proposing novel pretraining techniques.
- Language-model pretraining: MentalBERT and MentalRoBERTa follow bidirectional masked-language-modeling protocols based on BERT and RoBERTa.Both use base architectures; RoBERTa-based MentalBERT uses dynamic masking, and training continues in the target domain.
- Pretraining corpus: The pretraining corpus consists of anonymous Reddit posts from selected mental-health-related subreddits, without collecting user profiles.The selected communities include depression, SuicideWatch, anxiety, offmychest, bipolar, mentalillness, and mentalhealth.
- Pretraining corpus: The resulting training corpus contains 13,671,785 sentences.
- Downstream tasks: The models are fine-tuned for binary detection and multiclass classification of disorders including stress, anxiety, and depression.The final input feature is the [CLS] embedding from the last hidden layer, followed by an MLP classifier.
3 Results
The study evaluates pretrained language models across depression, suicidal ideation, stress, and other mental-disorder benchmarks using recall and F1. MentalRoBERTa performs best on most reported datasets, while domain-specific mental-health pretraining generally outperforms broader domain alternatives.
- Datasets: The benchmarks cover depression, suicidal ideation, stress, anxiety, bipolar disorder, and other mental-health categories across Reddit, Twitter, SMS-like sentences, and multiple forums.The datasets were selected to cover a relatively wide range of mental-health categories and social platforms.
- Evaluation: Recall and F1 are reported because mental-disorder detection commonly involves unbalanced classes and missed cases are important.F1 addresses class imbalance, while recall helps assess false negatives.
- Depression detection: MentalRoBERTa achieves the best depression-detection performance on eRisk and CLPsych, while MentalBERT ranks second on Depression_Reddit.These results are reported across three Reddit-collected depression datasets.
- Other disorders and suicidal ideation: MentalRoBERTa obtains the highest recall and F1 scores on T-SID, SAD, and Dreaddit, and the best F1 score on SWMH.MentalBERT has the highest F1 score on UMD, although its F1 is not competitive with other models there.
- Comparison and discussion: Mental-health-specific models perform better in most cases than general-corpus models, while biomedical and clinical pretraining is less helpful than target-domain pretraining.The discussion attributes the pattern to continued pretraining on the mental-health domain improving downstream classification performance.
4 Related Work
Prior work uses contextualized embeddings and NLP methods to represent behavioral, emotional, and textual signals in mental-health research. The cited benchmark tables provide results for depression and other mental-disorder classification tasks.
- Contextualized text embeddings: Self-supervised contextualized pretraining learns semantic and contextual information that benefits downstream tasks such as classification, sentiment analysis, and relation extraction.The related work situates mental-health models within broader pretrained-embedding research.
- NLP for mental healthcare: Mental-health NLP applies labeling and representation methods to identify emotions, mood, profiles, topics, sentiment traits, and possible mental-health problems.Mental-health detection includes categorizing social posts into disorder classes such as depression.
- Evaluation benchmarks: Table 2 reports depression-detection results, while Table 3 reports classification results for other mental disorders including stress, anorexia, and suicidal ideation.Bold text marks the best performance in both tables.
- NLP for mental healthcare: Suicidal-ideation detection is motivated by the reported connection between severe mental disorders and suicidal ideation without intervention.The cited work applies machine-learning methods, including deep representation learning, to early identification.
5 Conclusion and Future Work
The paper trains and releases MentalBERT and MentalRoBERTa as the first domain-specific language models for mental healthcare. Evaluation shows that continued pretraining on mental health-related corpus can improve downstream classification performance, while future work will extend coverage beyond English.
- 5 Conclusion and Future Work: MentalBERT and MentalRoBERTa are trained and released on Reddit mental-health domain data.The models are publicly available for reuse by the research community.
- 5 Conclusion and Future Work: The work is presented as the first effort to train domain-specific language models for mental healthcare.
- 5 Conclusion and Future Work: Continued pretraining with mental health-related corpus improves classification performance on downstream mental health detection tasks.The evaluated tasks include depression, stress, and suicidal ideation detection.
- 5 Conclusion and Future Work: Future work will collect multilingual mental health-related posts and train a multilingual language model.The current study focuses only on English because English corpora are relatively easy to obtain.
Social Impact
The released models support non-clinical automatic detection of mental disorders in online social content, potentially helping social workers identify people needing early prevention. Their predictions are not psychiatric diagnoses, and privacy, bias, fairness, uncertainty, and interpretability remain important concerns.
- Social Impact: The models facilitate non-clinical automatic detection of mental disorders in online social content.They may help social workers find potential individuals in need of early prevention.
- Social Impact: Model predictions are not psychiatric diagnoses.People with mental health issues are advised to seek professional help when possible.
- Social Impact: The study uses anonymous public posts, excludes user profiles, avoids identifying or interacting with users, and stores collected data securely.These measures are intended to minimize privacy impact during model training.
- Social Impact: Bias, fairness, uncertainty, and interpretability issues may arise during data collection and model training and require future evaluation.