Source-linked AI summary
Multi-Task Learning for Mental Health using Social Media Text
Adrian Benton, Margaret Mitchell, Dirk Hovy
TL;DR
Suicide-risk and mental-health assessment remains challenging, and prior work often modeled conditions in isolation. This paper applies multi-task learning to social-media text, jointly predicting mental-health conditions and suicide risk with gender as an auxiliary task. The authors report gains over parameter-matched single-task models, including strong results for neuroatypicality and suicide and improvements on data-limited conditions.
Problem
Suicide-risk assessment and monitoring related mental-health conditions are challenging, while prior social-media studies typically modeled conditions in isolation.
Source: §1 · p. 1Method
The authors use multi-task learning to jointly predict suicide risk and mental-health conditions from social-media text, with gender as an additional task.
Source: §1 · p. 2Results
All-task MTL significantly outperforms other models, reaching 0.846 TPR for neuroatypicality where FPR=0.1 and AUC of 0.848, TPR of 0.559 for suicide.
Source: §9 · p. 9Takeaways & Limitations
The authors conclude that auxiliary-task selection and gender prediction can improve performance, with especially large MTL gains for conditions with the least data.
Source: §9 · p. 9The matched, anonymized dataset is not population-representative, so the findings are framed as an MTL applicability test in a well-defined clinical setting rather than general social-media assessment.
Source: §2 · p. 2
Abstract
from arXiv · showhide
We introduce initial groundwork for estimating suicide risk and mental health in a deep learning framework. By modeling multiple conditions, the system learns to make predictions about suicide risk and mental health at a low false positive rate. Conditions are modeled as tasks in a multi-task learning (MTL) framework, with gender prediction as an additional auxiliary task. We demonstrate the effectiveness of multi-task learning by comparison to a well-tuned single-task baseline with the same number of parameters. Our best MTL model predicts potential suicide attempt, as well as the presence of atypical mental health, with AUC > 0.8. We also find additional large improvements using multi-task learning on mental health tasks with limited training data.
1 Introduction
The paper targets difficult suicide-risk assessment and proposes jointly modeling suicide risk with mental-health conditions and gender from social-media text. Its findings emphasize related auxiliary tasks, gender prediction, and low-false-alarm performance as promising directions for mental-health assessment.
1 Introduction
- Suicide-risk detection and monitoring related mental-health conditions are challenging, and traditional assessment may leave the timing of risk unclear.The introduction notes that suicide is a leading cause of death worldwide and that over 90% of people who die by suicide experience mental-health conditions.Source: §1 · p. 1
- Automated monitoring of patients’ language could complement traditional assessment with objective measurements that motivate further care and support.The authors also mention possible uses in verifying need for additional care with insurance coverage and assisting clinicians and patients.Source: §1 · p. 1
- Prior social-media studies typically modeled mental-health conditions in isolation, leaving coinciding influence factors unmodeled.The paper argues that jointly predicting related conditions could allow a shared representation to capture conditions that frequently occur together, or comorbidity.Source: §1 · pp. 1–2
- The proposed tasks include neuroatypicality, suicide attempt, seven mental-health conditions, and gender as an additional modeled factor.The mental-health conditions are anxiety, depression, eating disorder, panic attacks, schizophrenia, bipolar disorder, and PTSD; the authors also explore task selection.Source: §1 · p. 2
- The authors report that related or prerequisite tasks are critical to strong models, while gender prediction improves accuracy across conditions, including suicide risk.They describe the best-performing model as a promising direction for automated mental-health and suicide-risk assessment, with possible clinical application.Source: §1 · p. 2
- MTL raises anxiety’s True Positive Rate at 10% false alarms by up to 9.7% absolute, compared with a single-task model.The authors compare MTL with a single-task model having the same number of parameters.Source: §1 · p. 2
2 Ethical Considerations
The authors limit their claims to a well-defined clinical setting and flag risks from misuse of author-attribute detection.
2 Ethical Considerations
- The results are not intended as general assessments of mental-health conditions on social media because the matched, anonymized data are not population-representative.The stated scope is a test of MTL applicability in a well-defined clinical setting.Source: §2 · p. 2
- The authors identify the risk that author-attribute detection could be misused to single out people.They say they sought to minimize this risk by withholding selected features and representative examples.Source: §2 · p. 2
- The experiments’ clinical framing constrains what their findings support beyond the studied setting.The authors caution against interpreting the results as a means to assess mental-health conditions in social media generally.Source: §2 · p. 2
3 Model Architecture
The architecture shares parameters across tasks while retaining task-specific layers, and the evaluation compares MTL against neural single-task models with equal parameter counts.
3 Model Architecture
- The comparison includes logistic regression and parameter-matched neural STL and MTL models to isolate MTL’s contribution from model expressivity and parameter count.The authors identify unequal expressivity and parameter counts as potential confounds in prior comparisons.Source: §3 · p. 3
- The MTL network shares its first hidden layer across tasks and adds a task-specific hidden layer, while STL networks train independently for each task.The MTL output layer uses a logistic nonlinearity for binary predictions, and hidden layers use rectified linear units.Source: §3 · p. 3, Figure 1 · p. 3
- User-sampled mini-batches update all task weights and permit asynchronous learning when users have annotations for only some conditions.This setup supports joint learning from disjoint datasets with limited annotated instances.Source: §3 · p. 3
- The experiments tune L2 regularization and hidden-layer width using development loss, and report test-set performance in Figure 4.The search uses a common hidden-layer width across network layers.Source: §3 · p. 3
- After joint training for 5,000 iterations, the authors separately tune task-specific weights for another 1,000 iterations.They select task-specific weights by development loss and use mini-batch Adagrad for updates.Source: §3 · p. 4
4 Data
The dataset combines Twitter users with mental-health-condition and suicide-attempt labels, alongside neurotypical gender-matched controls; the models use character n-gram frequencies from users’ Twitter histories.
4 Data
- The combined dataset contains 9,611 users drawn from condition-specific Twitter datasets, suicide-attempt users, and neurotypical gender-matched controls.Conditions were identified from users’ explicit statements of diagnosis, verified by a human annotator; a subset of 1,101 users also received manual gender annotations.Source: §4 · p. 4
- Each user’s full Twitter history is represented by relative frequencies of the 5,000 most frequent character 1-to-5-gram features.This input representation is common to all evaluated models, enabling a fair comparison.Source: §4 · p. 4
5 Experiments
The experiments compare logistic regression, single-task neural networks, and multi-task neural networks, and use task ablations to examine which auxiliary predictions help. Performance is evaluated across held-out folds using AUC and true-positive rate at a low false-positive rate.
5 Experiments
- The study compares logistic regression, independently trained single-task perceptrons, and multi-task models trained to predict several conditions simultaneously.The experiment also tests whether multi-task learning helps compared with independent logistic regression models.Source: §5 · p. 4
- Data are split into five equal folds, with three for training, one for tuning, and one for testing.The reported performance is on the test fold.Source: §5 · p. 4
- The evaluation reports AUC and TPR at FPR=0.1, alongside ROC curves that show performance across false-positive-rate levels.The evaluation focuses on bounding false positives, which can cause undue stress in clinical settings.Source: §5 · p. 4
- Because users exhibiting each mental condition are the minority, a majority-baseline classifier would achieve zero recall; Table 1 presents frequency and comorbidity across mental-health conditions.Source: §5 · p. 5, Table 1 · p. 5
6 Results
Multi-task learning performs strongly across the main mental-health and suicide-risk tasks, including at a low false-positive rate, while the benefit depends on task selection and the task being predicted. Gender can help condition prediction, and multi-task models also improve results for some conditions with limited data.
6 Results
- STL performs worse than MTL or LR on AUC and low-FPR TPR, although STL and MTL are often comparable on F1-score.F1 weights false positives and false negatives equally.Source: §6 · p. 5
- 0.848 AUC for suicide is achieved by MTL, which improves over STL and LR on seven conditions; schizophrenia alone is best with STL.Figure 2 compares AUC by main task; STL uses a similar number of parameters to MTL.Source: Figure 2 · p. 5, §6 · p. 5
- MTL leads all nine conditions on TPR at FPR=0.1; adding gender further improves neuroatypicality, suicide, anxiety, schizophrenia, bipolar disorder, and PTSD.This operating point measures correctly predicted cases while holding false alarms low.Source: §6 · p. 5, Figure 3 · p. 6
- PTSD AUC is 0.786 with MTL, 0.770 with LR, and 0.667 with STL; bipolar and panic also improve when modeled with related, higher-data conditions.The authors interpret auxiliary tasks as a form of regularization that may help with data paucity and less-trustworthy labels.Source: §6 · pp. 6–7
- Adding gender as an auxiliary task tends to improve condition prediction, though usually not significantly; MTL predicts gender itself worse, and sparse gender annotation is considered an unlikely explanation.The data use inferred gender to match controls, which may introduce noise.Source: §6 · p. 6
- Using all mental-health tasks as auxiliaries gives each main task its best performance; smaller task subsets usually hurt, except bipolar benefits from depression and suicide-attempt prediction.The task-subset experiments focus on suicide attempt, anxiety, depression, and bipolar disorder.Source: §6 · pp. 6–7
7 Discussion: Multi-task Learning
The discussion reports gains from jointly predicting related mental conditions and demographic attributes, while identifying optimization and feature choices for further work.
7 Discussion: Multi-task Learning
- Multi-task learning yields significant gains over single-task models for suicide risk and several mental health conditions when related conditions and demographic attributes are predicted together.The discussion describes this as a promising direction for automated assessment.Source: §7 · p. 7
- Adagrad converged fastest to a good optimum in the optimizer experiments, while adaptive learning-rate optimizers generally converged quickly.Default SGD did not converge as quickly because it cannot scale learning rates separately by parameter.Source: §7 · p. 7
- A well-tuned SGD optimizer remains future work, since the authors found it took longer to converge than Adagrad.They note that further experimentation would be required.Source: §7 · p. 8
- Relative feature frequencies made models easier to train than raw feature counts, and character 1-to-5-grams were chosen for expected cross-domain generalization.The authors note that longer character n-grams remain possible, subject to time and regularization for overfitting.Source: §7 · p. 8
- Pretrained Google News word2vec unigram embeddings did not improve F-score in an earlier experiment, possibly because of other factors such as feature sparsity.That experiment used an MTL model without task-specific hidden layers.Source: §7 · p. 8
- Table 3 shows parameter sweeps for an MTL model trained on social-media data using character trigrams as input features, with hidden-layer width 256; its sweet spots may be good starting points for future training.Source: §7 · p. 8
8 Related Work
The related-work discussion situates multi-task learning in medical risk prediction and distinguishes this paper’s use of social-media text for mental health prediction.
8 Related Work
- Multi-task learning was introduced by Caruana as an approach motivated by the idea that similarity between related tasks helps learning.The paper frames this as learning tasks jointly rather than in isolation.Source: §8 · p. 8
- Table 2 reports test AUC for a main task after training with subsets of auxiliary tasks, marking significance versus logistic-regression and single-task baselines.The table caption defines * for p = 0.05 improvement over LR and † for improvement over STL.Source: Table 2 · p. 9
- Table 3 reports average development loss over epochs 990–1000 for joint training across learning-parameter settings, using Adagrad and hidden-layer width 256.The caption identifies the table as a parameter sweep.Source: Table 3 · p. 9
- Earlier medical-risk work predicted conditions from patient records, whereas this paper examines mental health conditions from social text without requiring full diagnoses.The prior work by Lipton et al. used an LSTM and found small, probably nonsignificant improvements over a similar single-task model.Source: §8 · pp. 8–9
- The authors use a simpler feed-forward n-gram model than the cited CNN framework, focusing comparison on whether multi-task training helps in this domain.The paper states that further model comparisons would distract from that question.Source: §8 · p. 9
9 Conclusion and Future Work
The conclusion reports that multi-task learning improves several prediction tasks, including suicide and mental health conditions, but is not ready for standalone clinical use.
9 Conclusion and Future Work
- The authors develop neural MTL models for 10 prediction tasks and compare them with single-task models trained independently for each task.The task set includes suicide, seven mental health conditions, neurotypicality, and gender.Source: §9 · p. 9
- The all-task MTL model significantly outperformed other models, reaching 0.846 TPR for neuroatypicality at FPR=0.1 and 0.848 AUC with 0.559 TPR for suicide.The authors also report anxiety, PTSD, and bipolar disorder gains; anxiety error fell by up to 11.9% versus a single-task model.Source: §9 · p. 9
- The study finds that model depth, auxiliary-task selection, and data availability shape MTL performance, while gender prediction improves performance as an auxiliary task.The largest MTL improvements were reported for conditions with the least data; gender itself did not follow the same pattern.Source: §9 · p. 9
- The MTL approach is not yet accurate enough for use in isolation in clinical settings.The authors describe it as a promising direction for aiding clinicians’ evaluations.Source: §9 · p. 9