Source-linked AI summary
Utilizing Neural Networks and Linguistic Metadata for Early Detection of Depression Indications in Text Sequences
Marcel Trotzek, Sven Koitka, Christoph M. Friedrich
TL;DR
Depression is widespread, consequential, and often untreated, motivating earlier detection from online text. This paper evaluates neural and linguistic-metadata classifiers, examines the ERDE metric, and reports state-of-the-art results while proposing a modified metric.
Problem
Depression is widespread and often untreated, while affected individuals use online platforms and depression influences language, motivating earlier detection from written text.
Method
The paper evaluates a convolutional neural network with different word embeddings alongside user-level linguistic metadata classification on sequential social-platform messages.
Results
The combined neural and metadata approaches achieve state-of-the-art results in the eRisk 2017 early detection task.
Takeaways & Limitations
The work supports combining text-based neural models with user-level metadata and proposes a more interpretable modification of ERDE for chunk-based shared tasks.
Takeaways & Limitations
Because the models were optimized on eRisk 2017 for comparison, performance on unseen data requires further evaluation.
Abstract
from arXiv · showhide
Depression is ranked as the largest contributor to global disability and is also a major reason for suicide. Still, many individuals suffering from forms of depression are not treated for various reasons. Previous studies have shown that depression also has an effect on language usage and that many depressed individuals use social media platforms or the internet in general to get information or discuss their problems. This paper addresses the early detection of depression using machine learning models based on messages on a social platform. In particular, a convolutional neural network based on different word embeddings is evaluated and compared to a classification based on user-level linguistic metadata. An ensemble of both approaches is shown to achieve state-of-the-art results in a current early detection task. Furthermore, the currently popular ERDE score as metric for early detection systems is examined in detail and its drawbacks in the context of shared tasks are illustrated. A slightly modified metric is proposed and compared to the original score. Finally, a new word embedding was trained on a large corpus of the same domain as the described task and is evaluated as well.
1 INTRODUCTION
Depression is widespread, can have severe consequences including suicide, and often remains untreated. Because affected individuals use online platforms to seek information and support, the paper investigates early classification of depression indications in written text.
- More than 300 million people worldwide were suffering from depression, about 4.4% of the global population.
- 788,000 people died by suicide in 2015, making suicide the second most common cause of death among people aged 15–29 worldwide.
- Treatment remains limited, with only about half of Europeans affected by mental disorders receiving care and treatment-seeking rates varying substantially across regions.Reported first-year treatment-seeking ranged from 29–52% in Europe and 35% in the USA to 6% in Nigeria or China.
- People affected by depression increasingly use social media to connect, share experiences, and support one another.Online peer communities may challenge stigma, increase professional-help seeking, and offer direct help.
- The paper focuses on classifying indications of depression in written texts as early as possible using machine learning methods.
2 RELATED WORK
Related work connects depression with language use, social-media health research, text classification, and sequential early detection. It also emphasizes ethical risks because language data can reveal sensitive information about individuals and groups.
- Depression and language: Previous studies linked depression with increased first-person singular and negative-emotion word use.
- Linguistic analysis tools: LIWC operationalizes linguistic analysis by categorizing written texts through word counts from a comprehensive dictionary.
- Ethical perspective: Ethical concerns arise because language acts as a proxy for human behavior and carries latent information about authors and situations.The literature also discusses social impact, demographic bias, and the need for ethical guidelines in NLP.
- Text classification: Text classification evolved from statistical clue-word categorization and rule-based systems toward machine-learning and deep-learning methods.Related work reports strong results from transfer-learning approaches such as ULMFit and BERT.
- Social-media health research: NLP research using social-media messages has been applied to population-level tracking and detection of depression and other health conditions.
- Early detection: Early detection uses sequential reading so predictions can be made from partial document information, including in depression detection on Reddit.The eRisk 2017 task combined depression detection with early detection using chronological Reddit posts and comments.
3 DATASET OVERVIEW
The study uses chronological Reddit post sequences from depressed and control users to evaluate early depression detection and its metrics. It identifies dataset artifacts affecting evaluation and proposes a chunk-aware modification of ERDE.
- 3.1 Dataset Overview: The dataset contains chronological Reddit posts and comments from 135 depressed users and 752 randomly selected control users.Depressed users were identified through posts explicitly mentioning a diagnosis.
- 3.1 Dataset Overview: 91 completely empty messages are discarded because they appear attributable to faults in Reddit, its API, or preprocessing.Reddit messages may contain titles, text, both, images, URLs, or comments.
- 3.1 Dataset Overview: Timestamps are retained only for within-user ordering and time-pattern analysis, because users’ actual time zones are unknown.Cross-user timestamp comparisons would likely be misleading.
- 3.1 Dataset Overview: A latest-post timestamp alone achieved an F1 score of 0.78, revealing a hidden control-group collection feature that all models therefore discard.The artifact arose because controls were selected from recently active users rather than a matched temporal distribution.
- 3.2 Task and Evaluation Criteria: The eRisk task releases each user’s chronologically ordered messages in ten 10% chunks, allowing classification, rejection, or delayed prediction each week.The paper processes the same training, test, and chronological chunks as the pilot task for direct comparability.
- 3.2 Task and Evaluation Criteria: Because baseline systems could predict after each message but eRisk required whole chunks, their results are not directly comparable to the task results.Chunk sizes range from one to 200 messages per user.
- 3.2 Task and Evaluation Criteria: 10.61% ERDE5 and 8.48% ERDE50 can accompany F1 = 0.51 even for early perfect identification of users with fewer than ten messages per chunk.The example achieves 1.0 precision and 0.35 recall, showing why F1 remains important alongside early-detection scores.
- 3.2 Task and Evaluation Criteria: The proposed ERDE% modification retains a parameterized sigmoid cost while scaling latency by each user’s proportion of documents read.It uses nd for total documents per user, k for documents read, and o ∈[0, 100] for the percentage-centered growth point.
4 LINGUISTIC METADATA
The paper combines user-level linguistic metadata, including word usage, readability, sentiment, and LIWC features, to classify depression indications. These metadata features alone can lead to very good results on the eRisk 2017 dataset.
- Feature construction: User-level metadata is extracted from each message’s text and title, then averaged or summed across a user’s documents.The feature calculations use concatenated text and title fields except for field-specific measures such as average lengths.
- Word and grammar features: Counts of personal-pronoun usage, especially “I,” and past-tense verbs encode previously reported language differences associated with depression.Occurrences of “I” are counted separately in each message’s text and title.
- Explicit indicators: Explicit phrases, antidepressant names, and diagnosis statements are counted to identify obvious positive cases, with these counts summed across each user’s documents.Examples include “my depression,” “my anxiety,” “my therapist,” and variations of “I was diagnosed with depression.”
- Readability: Readability features distinguish discussion-rich messages from controls’ short headlines, sentences, or single-word posts.Four measures with the highest training-set correlation were selected, including Gunning Fog Index, Flesch Reading Ease, and New Dale-Chall Readability.
- Emotions and sentiment: Emotion and sentiment lexicon features were averaged across users, but showed no relevant correlation with the class label in this dataset.The evaluated resources included NRC, VADER, and Opinion Lexicon features; the positive class contained slightly more emotions and sentiments of all kinds.
- Feature summary: The final feature set combines 17 metadata features with ten LIWC features selected for highest training-data correlation, despite slight redundancy.The LIWC features include function-word counts and variations of “I.”
5 NEURAL NETWORK MODELS
The paper evaluates convolutional neural networks using multiple pretrained and reddit-trained word embeddings for document classification. The neural architecture produces document-level outputs that are aggregated into user-level predictions, while direct metadata integration did not improve results.
- CNN architecture: The neural models vectorize documents with word embeddings and use a simple convolutional neural network for text classification.The architecture follows a one-layer CNN design for sentence classification.
- Pretrained embeddings: The experiments compare 50- and 300-dimensional GloVe vectors with three 300-dimensional pretrained fastText models.The pretrained vectors were trained on Wikipedia, Gigaword 5 news, or Common Crawl.
- Reddit-trained embedding: A domain-specific fastText model was trained on about 1.7 billion reddit comments while preserving emoticons, punctuation, and special-character tokens.The resulting corpus contains 1.37 billion messages and 49.9 billion tokens after preprocessing.
- Embedding analysis: The self-trained reddit embedding identifies similar emoticons and subreddits that the evaluated pretrained fastText and GloVe models cannot identify.Nearest-neighbor word clouds provide the qualitative analysis for selected tokens.
- Embedding evaluation: The word analogy evaluation reports semantic, syntactic, and total accuracies for the 300-dimensional embeddings.The benchmark contains 8,869 semantic and 10,675 syntactic analogy questions.
- Prediction aggregation: CReLU and 1-max pooling produce a 200-dimensional document vector, whose 98th-percentile output becomes the user-level prediction.The percentile aggregation gives more weight to documents with higher predicted probability than the mean would.
- Metadata integration: Feeding metadata as a second neural-network input did not improve results over using the text input alone.This also held for an approach combining metadata with an LSTM-derived representation and for adding a pre-softmax vector.
6 EXPERIMENTS
The experiments reproduce the eRisk 2017 chunked evaluation, comparing CNN, metadata-based logistic regression, and late-fusion models using early-detection and classification metrics. The ensemble achieves the best ERDE5/ERDE50 results under the proposed metric, while metric optimization can diverge from overall classification quality.
- 6.1 Experiment Setup: Test documents were processed sequentially in ten chunks, each containing 10% of writings from every user.Training used the full training dataset, while earlier test chunks were reused during later processing.
- 6.1 Experiment Setup: The metadata models used standardized 27-feature vectors with logistic regression to predict probabilities for each test chunk.Boolean flags were exempted from standardization because they already used -1 or 1 values.
- 6.2 Evaluation: Late-fusion ensembles produced the best achieved ERDEo scores and recall, whereas Meta LR Wait achieved the best overall F1 score by waiting for the final chunk.The ensemble averaged metadata probabilities and neural-network outputs without calibration.
- 6.2 Evaluation: ERDEo and F1 are difficult to maximize simultaneously, but the described models achieved a balanced result across the reported scores.The comparison included ERDE5, ERDE50, F1, ERDE%20, ERDE%50, and Flatency.
- 6.2 Evaluation: The same ensemble achieved the best overall ERDE% scores, while higher thresholds that improved ERDE5 produced the worst overall ERDE% scores.The affected models were fastText reddit with p > 0.8 and Meta LR with p > 0.55.
- 6.2 Evaluation: The 50-dimensional GloVe model achieved the best Flatency score, exceeding the original paper’s best reported score of 0.389.The results indicate that optimizing ERDE5 does not necessarily improve classification results.
7 CONCLUSION
The paper combines metadata and neural text classification for early depression detection, reports state-of-the-art results on eRisk 2017, and scrutinizes the ERDE metric. It also proposes a more interpretable metric and identifies directions for evaluation and model improvement.
- 7 CONCLUSION: The ERDE5 metric is not meaningful for this shared task because minimizing false positives can produce the best score despite limited positive-sample correctness.The analysis found that only a few correctly predicted positive samples affect ERDE5 substantially.
- 7 CONCLUSION: A modified ERDE metric was proposed to improve interpretability when systems receive information in chunks.The paper compares exemplary scores from the modified metric with ERDE scores for its experiments.
- 7 CONCLUSION: State-of-the-art results were reported on the eRisk 2017 dataset using user-level metadata and a convolutional neural network.The CNN uses text representations, while the metadata approach examines additional user-level features.
- 7 CONCLUSION: Future evaluation should test the models on unseen data rather than only on results optimized for comparison within eRisk 2017.The paper notes that later eRisk 2018 experiments used 820 new test subjects.
- 7 CONCLUSION: The paper argues that early-detection research should agree on a new metric for tasks such as eRisk.This recommendation follows the detailed examination of the current ERDE metric.
- 7 CONCLUSION: Future work includes integrating metadata directly into the neural network, calibrating larger ensembles, combining embeddings, and evaluating language-modeling methods such as BERT.The paper also identifies publication of the Reddit-trained fastText model as an issue for further examination.