Source-linked AI summary
Dreaddit: A Reddit Dataset for Stress Analysis in Social Media
Elsbeth Turcan, Kathleen McKeown
TL;DR
Existing computational stress research is concentrated in settings such as speech and short social-media genres, despite the importance and prevalence of stress. The paper introduces Dreaddit, a lengthy multi-domain Reddit corpus with human-labeled segments and supervised benchmarks, finding that lexical and domain-specific information is central to stress detection while writer framing and intentions remain difficult to capture.
Problem
Existing computational research studies stress mainly in speech or short genres such as Twitter, motivating broader text resources for stress identification.
Method
The paper constructs Dreaddit from lengthy Reddit posts across five categories, labels segments with Mechanical Turk, and evaluates traditional and neural supervised models.
Results
Stress detection is highly lexical: domain-specific embeddings outperform n-grams and pretrained embeddings, while the best logistic-regression model reaches an F-score of 79.8.
Takeaways & Limitations
Dreaddit provides a benchmark for interpretable stress detection, with performance benefiting from psychological resources and domain knowledge.
Takeaways & Limitations
Lexical features struggle with positive or retrospective experiences and stories centered on someone other than the poster, leaving writer framing and intentions for future work.
Abstract
from arXiv · showhide
Stress is a nigh-universal human experience, particularly in the online world. While stress can be a motivator, too much stress is associated with many negative health outcomes, making its identification useful across a range of domains. However, existing computational research typically only studies stress in domains such as speech, or in short genres such as Twitter. We present Dreaddit, a new text corpus of lengthy multi-domain social media data for the identification of stress. Our dataset consists of 190K posts from five different categories of Reddit communities; we additionally label 3.5K total segments taken from 3K posts using Amazon Mechanical Turk. We present preliminary supervised learning methods for identifying stress, both neural and traditional, and analyze the complexity and diversity of the data and characteristics of each category.
1 Introduction
The paper introduces stress detection as an important social-media research problem and presents Dreaddit, a labeled corpus and supervised-model benchmarks for studying it.
- Motivation: Social media provides large-scale text data for studying people’s daily worries and stresses.The paper highlights Twitter, Reddit, and Facebook as sources of observable stress-related data.
- Motivation: Stress is widespread, subjective, and associated with physical and mental health outcomes when excessive.Its definition and effects vary across people, while moderate stress can be positive.
- Contributions: The paper presents a corpus of social-media text for detecting the presence of stress.The authors position the corpus as a resource for applications including health, public mood, and disaster analysis.
- Contributions: Dreaddit contains lengthy posts from five categories, including stressful and non-stressful text and human annotations.The dataset is designed to capture different ways stress is expressed across categories.
- Contributions: The paper provides discrete and neural supervised models, benchmarks, and analyses of dataset content and model performance.These contributions support further work on stress detection and characterize the problem’s data.
2 Related Work
Prior work studies stress through physiological signals, speech, and short social-media genres, whereas Dreaddit focuses on stress as a general experience in lengthy Reddit text.
- Existing approaches: Earlier stress research often relies on physiological signals or speech, which require hardware or physical presence.The paper argues that social-media text can provide a purely textual setting for studying stress.
- Dreaddit’s position: Dreaddit differs by using lengthy, multi-domain Reddit posts rather than primarily short microblog messages.This setting supports analysis of stress expressions in more extended personal narratives.
- Existing approaches: Related computational studies use Twitter, Facebook, or microblog data to detect or characterize stress.Examples include LSTM, CNN, factor-graph, and questionnaire-based approaches.
- Dreaddit’s position: The corpus is presented as the first Reddit dataset focused on stress as a general experience rather than only a clinical concept.Prior Reddit work cited by the paper examined depression and clinical diagnoses such as general anxiety.
3 Dataset
Dreaddit collects lengthy Reddit posts from stress-relevant communities and annotates five-sentence segments for expressed stress, while preserving contextual and domain variation.
- Reddit data: Reddit organizes user posts and comments into topic-specific communities, making lengthy posts useful for studying stress nuances.The dataset selects communities where users are likely to discuss stressful topics.
- Reddit data: Figure 1 illustrates stress expressions in an r/anxiety post by highlighting possible indicators in the original social-media text.The example is reproduced as found in the dataset.
- Reddit data: The dataset covers interpersonal conflict, mental illness, and financial-need domains among stress-relevant subreddit categories.The listed communities include abuse, social, anxiety, and PTSD discussions, with financial domains introduced in the source passage.
- Dataset characteristics: The average post contains 420 tokens, providing substantially more author context than typical microblog data.The authors connect longer posts to future analysis of stress causes and effects.
- Dataset characteristics: The dataset mainly contains first-person narratives and requests for assistance or advice, with variation in topics, language, and annotator agreement.Table 2 presents representative labeled segments from the different domains.
- Data annotation: Annotators label contiguous five-sentence segments because stress expressions may require context while full posts are difficult to annotate.Segment-level annotation also supports future localization of stress expressions within longer posts.
- Data annotation: Workers classify each segment as Stress, Not Stress, or Can’t Tell, focusing on whether the author expresses stress with a negative attitude.The task distinguishes the author’s expressed response from whether the situation itself appears stressful.
- Data annotation: The final annotation set contains 3,553 labeled segments from 2,929 posts after quality filtering, with majority-vote labels.At least five workers annotated submitted segments, and segments with excessive Can’t Tell responses were removed.
4 Data Analysis
The analysis finds domain- and label-level differences in vocabulary, lexical diversity, annotator agreement, and syntactic complexity. These patterns show that stress expression varies across domains and labels, while agreement is inversely related to lexical variety.
- By domain: Domain and topic affect expression variety, with particularly low negative-emotion variety in the financial domain and topic shifts across domain categories.The financial domain contains 1.54% negative-emotion words and uses only 31% of that word list; interpersonal domains contain roughly 1.5 times as many social words as others.
- By domain: Mental-illness domains have exceptionally low overall lexical diversity, but this pattern persists for negative-emotion words only in anxiety.The paper attributes the overall pattern to structured clinical language about symptoms, medical care, and diagnoses.
- By label: Stressful data uses more first-person pronouns, whereas non-stressful data uses more social words.The authors associate these patterns with increased self-focus and a potentially better social support network, respectively.
- By label: Stressful data is generally longer and more complex but receives simpler readability-index ratings, with differences that are non-significant yet persistent.The analysis considers conjunctions, segment length, clauses per sentence, Flesch-Kincaid Grade Level, and Automated Readability Index.
- By agreement: Lexical variety is inversely related to annotator agreement, and the authors argue that the trend reflects data type rather than length because Yule’s I controls for length.Figure 3 compares Yule’s I across domain size and agreement levels.
5 Methods
The authors train supervised stress-detection models on post-grouped train/test splits using textual, lexical, syntactic, and social-media features. They compare traditional and neural approaches with multiple input representations, including n-grams, embeddings, and BERT.
- Data split and features: The post-grouped split assigns 2,838 labeled segments to training and 715 to testing, preventing shared post content across sets.The test set contains approximately 10% of posts, with 51.6% stressful training labels and 52.4% stressful test labels.
- Data split and features: Models use lexical, syntactic, and social-media features alongside bag-of-n-grams and distributed word embeddings.Feature groups include LIWC, affect and sentiment scores, part-of-speech and readability measures, timestamps, voting ratios, karma, and comment counts.
- Traditional models: Traditional models include SVMs, logistic regression, Naïve Bayes, Perceptron, and decision trees tuned with grid search and 10-fold cross-validation.The experiments compare different combinations of inputs and features.
- Input representations: Input representations include 1–3-gram features, 300-dimensional Google News and in-domain Word2Vec embeddings, and 768-dimensional in-domain BERT embeddings.The study also separates feature subsets by lexical, syntactic, and social categories.
- Neural models: Neural experiments use a two-layer bidirectional GRNN, a CNN with parallel filters of sizes 2 and 3, and fine-tuned pretrained BERT-base.BERT-base is fine-tuned on the classification task for three epochs.
6 Results and Discussion
The strongest results come from domain-specific representations and carefully selected lexical features, with a logistic-regression model reaching BERT-comparable performance. Errors remain concentrated in less explicit, retrospective, and other-person-focused expressions of stress.
- 79.8 F1-score is achieved by logistic regression using domain-specific Word2Vec embeddings, high-correlation features, and high-agreement data.This significantly outperforms the majority, n-gram, and pretrained-embedding baselines at p < 0.01.
- Domain-specific Word2Vec and BERT embeddings significantly outperform n-grams and pretrained embeddings, highlighting the value of domain knowledge.
- The best logistic-regression classifier performs comparably to BERT-base while offering greater interpretability and less intensive training.The comparison is not statistically significant, with p > 0.5.
- Lower-agreement data performs worse, but larger 80% agreement data consistently outperforms smaller perfect-agreement data.This indicates a tradeoff between annotation quality and dataset size.
- LIWC and high-correlation features outperform broader or syntactic feature sets, suggesting stress detection is highly lexical.The strongest features include LIWC clout, tone, and first-person pronoun features.
- Both leading models overclassify stress and struggle with less explicit, positive or retrospective experiences and stories focused on someone other than the poster.Misclassified examples often have low annotator agreement, averaging 0.55 for BERT and 0.61 for logistic regression.
7 Conclusion and Future Work
Dreaddit establishes a dataset and baseline for social-media stress classification, while analyses characterize stress detection as lexical and domain-dependent. The authors identify writer framing and intentions as important areas for improvement.
- Dreaddit provides a dataset for social-media stress classification with an 80% F-score baseline on binary stress classification.
- Stress detection is highly lexical and benefits from domain knowledge.
- The authors identify incorporating the writer’s framing and intentions as a remaining challenge.
- Future work will use Dreaddit to contextualize stress, explain predictions through text features, and develop distant-labeling schemes.
A Data Samples
The appendix presents anonymized Reddit posts reproduced as obtained, using typical-length examples selected for readability and space constraints.
- The full posts have identifying information removed and whitespace collapsed, but are otherwise reproduced exactly as obtained.
- The selected examples are reasonably typical in length but were chosen partly for readability and space concerns.
- The average post length is 420 tokens, with interpersonal-subreddit posts longer and other-subreddit posts shorter.
B Full Annotation Guidelines
The annotation guidelines describe a Mechanical Turk task in which workers classify text segments as stressful, non-stressful, or uncertain under monitored conditions.
- Workers selected “Stress”, “Not Stress”, or “Can’t Tell” for each of five text segments.
- Each HIT included one of 50 check questions to assess whether workers followed the instructions.
- Workers had one hour to complete each HIT and received $0.12 for correctly answering its check question.
- Worker participation was limited to 30 total submissions per worker.
C Parameter Settings
The parameter settings combine reproducible grid-search tuning for traditional models with early-stopped, regularized neural training configured for the small dataset.
- Traditional models: Traditional models use grid search, with the best model using unbalanced class weights, L2 penalty, and C=10.
- Traditional models: All traditional-model cross-validation runs use the same random seed for comparability and reproducibility.
- Neural models: Neural models train with Adam for up to ten epochs and use validation-based early stopping.
- Neural models: Neural training applies dropout of 0.5, hidden sizes and batch size of 128, and a learning rate of 5 · 10−4.
D Error Analysis Examples
This section supplements the error analysis with additional test examples that BERT-base and/or the best logistic regressor misclassified. The supplementary material includes full posts from several subreddits and the annotation instructions.
- Additional test examples were misclassified by BERT-base, the best logistic regressor, or both models.The logistic regressor used embeddings trained on the unlabeled corpus and high-correlation discrete features.
- The examples include full posts from r/anxiety, r/ptsd, r/domesticviolence, and r/assistance.The r/assistance example marks hyperlinks with brackets.
- The supplementary material also provides the full annotation instructions and identifies BERT and LogReg as the two compared models.Table 10 labels the material as additional error-analysis examples.