PaperlayerView exact revision on arXiv

Source-linked AI summary

Mental-LLM: Leveraging Large Language Models for Mental Health Prediction via Online Text Data

Xuhai Xu, Bingsheng Yao, Yuanzhe Dong, Saadia Gabriel, Hong Yu, James Hendler, Marzyeh Ghassemi, Anind K. Dey, Dakuo Wang

arXiv:2307.14385v4cs.CL

TL;DR

Mental-LLM examines how to improve general-purpose LLMs for mental-health prediction from online text, where existing evidence is limited. It evaluates prompting and instruction finetuning across multiple models and datasets, finding that instruction finetuning improves performance across tasks and yields models competitive with strong baselines. The authors provide practical guidelines while emphasizing dataset, model, bias, and deployment limitations.

  • Problem

    Research has limited evidence on how to understand and improve general-purpose LLMs for mental-health tasks.

  • Method

    The paper evaluates multiple LLMs on mental-health prediction tasks using online social-media datasets, comparing zero-shot prompting, few-shot prompting, and instruction finetuning.

  • Results

    Instruction finetuning significantly improves performance across multiple mental-health tasks and datasets, with Mental-Alpaca and Mental-FLAN-T5 outperforming GPT-3.5 by 10.9% and GPT-4 by 4.8% in balanced accuracy.

  • Takeaways & Limitations

    Merging multiple datasets and tasks for instruction finetuning is presented as an effective way to enhance generalizability across mental-health prediction tasks.

  • Takeaways & Limitations

    The findings may not generalize beyond the limited datasets and model types studied, and Reddit-based finetuning can introduce bias.

Abstract

from arXiv · show

Advances in large language models (LLMs) have empowered a variety of applications. However, there is still a significant gap in research when it comes to understanding and enhancing the capabilities of LLMs in the field of mental health. In this work, we present a comprehensive evaluation of multiple LLMs on various mental health prediction tasks via online text data, including Alpaca, Alpaca-LoRA, FLAN-T5, GPT-3.5, and GPT-4. We conduct a broad range of experiments, covering zero-shot prompting, few-shot prompting, and instruction fine-tuning. The results indicate a promising yet limited performance of LLMs with zero-shot and few-shot prompt designs for mental health tasks. More importantly, our experiments show that instruction finetuning can significantly boost the performance of LLMs for all tasks simultaneously. Our best-finetuned models, Mental-Alpaca and Mental-FLAN-T5, outperform the best prompt design of GPT-3.5 (25 and 15 times bigger) by 10.9% on balanced accuracy and the best of GPT-4 (250 and 150 times bigger) by 4.8%. They further perform on par with the state-of-the-art task-specific language model. We also conduct an exploratory case study on LLMs' capability on mental health reasoning tasks, illustrating the promising capability of certain models such as GPT-4. We summarize our findings into a set of action guidelines for potential methods to enhance LLMs' capability for mental health tasks. Meanwhile, we also emphasize the important limitations before achieving deployability in real-world mental health settings, such as known racial and gender bias. We highlight the important ethical risks accompanying this line of research.

1 INTRODUCTION

Mental-LLM addresses limited evidence on improving general-purpose LLMs for mental-health tasks through a comprehensive evaluation of prompting and instruction finetuning. Instruction finetuning substantially improves performance across multiple tasks and datasets, while the authors caution that technical results do not imply real-world deployability.

  • Motivation: LLMs show promising but limited capability for mental-health prediction from natural language, leaving how to improve this capability unresolved.Prior work largely evaluated zero-shot systems, which remained below state-of-the-art domain-specific NLP models.
  • Findings: Few-shot prompting improves performance to some extent, but its advantage remains limited.The reported improvement is Δ = 4.1%.
  • Findings: Mental-Alpaca and Mental-FLAN-T5 outperform GPT-3.5 by 10.9% in balanced accuracy and GPT-4 by 4.8%.The finetuned models also perform on par with the task-specific state-of-the-art Mental-RoBERTa.
  • Approach: The study evaluates zero-shot prompting, few-shot prompting, and instruction finetuning across multiple LLMs using online social-media data.The evaluated models include open-source and closed-source systems, including Alpaca, FLAN-T5, LLaMA2, GPT-3.5, and GPT-4.
  • Findings: Instruction finetuning significantly improves mental-health prediction across multiple tasks and datasets simultaneously.This supports the paper’s vision of using one general-purpose LLM across varied mental-health-related tasks.
  • Contributions: The authors release Mental-Alpaca and Mental-FLAN-T5 and provide guidelines for adapting LLMs to specific domains.They also highlight ethical concerns associated with health-related applications.

2 BACKGROUND

Prior mental-health research uses online text to detect risks, while LLM research has explored health applications but comparatively less mental-health evaluation. Existing mental-health LLM studies mainly use zero-shot prompting, motivating broader evaluation and improvement methods.

  • Online Text for Mental Health: Online platforms provide text and interaction data for studying psychological states and predicting risks such as anxiety, depression, and suicide ideation.Their real-time and archival properties can help mitigate retrospective bias.
  • Online Text for Mental Health: Earlier mental-health prediction research developed linguistic features and machine-learning models for task-specific applications.Examples include correlation analysis and SVM-based depression prediction.
  • LLMs and Health Applications: Instruction finetuning trains models with varied prompts across multiple datasets and tasks within a unified framework.This approach is presented as a way to support diverse task-solving behavior.
  • LLMs and Health Applications: LLMs have been explored in medical and public-health domains, including medical question answering and clinical-note applications.The cited work includes finetuned PaLM-2 and LLaMA models.
  • Mental-Health LLMs: Mental-health LLM research remains relatively limited, with some studies examining sentiment analysis and emotion reasoning.Closer studies tested ChatGPT on stress, depression, and suicide classification tasks.

3 METHODS

The paper evaluates model-agnostic zero-shot, few-shot, and instruction-finetuning strategies for mental health prediction from online text. Its prompt template structures input data, task specifications, questions, and output constraints, while instruction finetuning extends models across multiple datasets and tasks.

  • 3 METHODS: The experiments compare zero-shot prompting, few-shot prompting, and instruction finetuning in model-agnostic mental health prediction setups.Zero-shot and few-shot prompting leave model parameters unchanged, whereas instruction finetuning updates parameters and generates new models.
  • 3.1 Zero-shot Prompting: Prompt_ZS combines online text data, a prediction-target specification, a task question, and an output constraint.The output constraint can require formats such as only returning yes or no for binary classification.
  • 3.1 Zero-shot Prompting: PromptPart1-S varies from a blank specification to context enhancement, mental health enhancement, or their combination.The combined strategy asks the model to act as a mental health expert under social media context.
  • 3.1 Zero-shot Prompting: PromptPart2-Q adapts questions to critical mental states or high-stakes risk actions and supports binary and multiclass classification.The paper evaluates task-specific modifications for targets such as stress, depression, and suicide.
  • 3.2 Few-shot Prompting: Few-shot prompting adds randomly sampled prompt-label examples to Prompt_ZS for in-context domain-specific information without changing model parameters.The number of prompt-label pairs is limited by the model’s input length.
  • 3.3 Instruction Finetuning: Instruction finetuning extends beyond single-dataset training by training LLMs across multiple datasets and tasks simultaneously.The method uses the same finetuning and testing structure for single- and multi-dataset settings.

4 IMPLEMENTATION

The implementation benchmarks multiple LLM families on human-annotated online-text datasets covering six mental health prediction tasks. The study uses held-out external datasets, user-exclusive train-test splits, and instruction-finetuned Mental-Alpaca and Mental-FLAN-T5 models.

  • 4.1 Datasets and Tasks: The experiment uses four established mental health datasets and defines six diverse prediction tasks from them.The datasets were selected for high-quality availability and human expert annotations or supervision rather than weak linguistic labels.
  • 4.1 Datasets and Tasks: The study covers stress, depression, and suicide-related prediction at post, message, and user levels.Examples include binary and four-level depression prediction, binary suicide ideation prediction, and five-level suicide-risk prediction.
  • 4.1 Datasets and Tasks: Each dataset uses an 80%/20% train-test split with each user assigned exclusively to one split.This split design is intended to avoid data leakage between training and testing.
  • 4.2 Models: The benchmark includes Alpaca, Alpaca-LoRA, FLAN-T5, LLaMA2, GPT-3.5, and GPT-4 with different sizes, pretraining targets, and availability.FLAN-T5 emphasizes task solving, while the other listed models are finetuned with natural dialogue as an optimization goal.
  • 4.1 Datasets and Tasks: Seven datasets are organized into four training/testing datasets and three external-evaluation datasets.All datasets contain human annotations as labels.

5 RESULTS

The results show promising but limited performance for zero-shot and few-shot prompting, while multi-dataset instruction finetuning substantially improves performance across mental health tasks. An exploratory case study also reports strong reasoning capability for certain models, especially GPT-4, without establishing deployability.

  • 5 RESULTS: Zero-shot and few-shot prompting show promising but still limited performance on mental health tasks.Few-shot prompting improves performance over zero-shot prompting, but the models remain below task-specific models.
  • 5 RESULTS: Instruction finetuning on multiple datasets significantly boosts performance across all tasks simultaneously.The resulting Mental-Alpaca and Mental-FLAN-T5 models are the paper’s principal finetuned systems.
  • 5 RESULTS: The exploratory reasoning case study reports strong reasoning capability for certain LLMs, especially GPT-4.This case study supplements the paper’s primary focus on prediction tasks.
  • 5 RESULTS: The reported findings do not establish deployability in real-world mental health settings.The paper highlights ethical concerns and research gaps that remain before deployment.

5.1 Zero-shot Prompting Shows Promising yet Limited Performance

Zero-shot LLMs can approach task-specific mental-health models, but performance varies substantially by model and prompt strategy. Context enhancement is generally useful for dialogue-focused models, while task-solving-focused FLAN-T5 and the largest GPT-4 are less helped or can decline.

  • 5.1 Zero-shot Prompting Shows Promising yet Limited Performance: ΔAlpaca = 5.5% and ΔAlpaca-LoRA = 5.6% over the majority baseline, while both remain 20%-25% below task-specific baselines.These balanced-accuracy comparisons summarize zero-shot performance for the two Alpaca models.
  • 5.1 Zero-shot Prompting Shows Promising yet Limited Performance: ΔGPT-3.5 = 12.4% over baseline in zero-shot prompting, consistent with more promising performance from the larger model.GPT-3.5 is 25× larger than Alpaca and Alpaca-LoRA in the reported comparison.
  • 5.1 Zero-shot Prompting Shows Promising yet Limited Performance: ΔFLAN-T5_vs_GPT-3.5 = 4.2%, with FLAN-T5 also exceeding Alpaca, Alpaca-LoRA, and LLaMA2 in overall zero-shot results.FLAN-T5 exceeds Mental-RoBERTa by 4.5% on Task #6 despite its smaller model size.
  • 5.1 Zero-shot Prompting Shows Promising yet Limited Performance: GPT-4’s average zero-shot performance exceeds FLAN-T5 by 6.4%, LLaMA2 by 7.5%, and GPT-3.5 by 10.6%.GPT-4 remains 7.9% below Mental-RoBERTa overall while exceeding it by 4.5% on Task #6.
  • 5.1.1 The Effectiveness of Enhancement Strategies.: Contextual information generally improves zero-shot performance, but prompt enhancements reduce Alpaca-LoRA and FLAN-T5 performance by 2.7% and 1.6%.Across the four models with positive effects, context enhancement is the most stable strategy.
  • 5.1.1 The Effectiveness of Enhancement Strategies.: ΔGPT-3.5−Task#4 = 14.8% and ΔGPT-3.5−Task#6 = 6.8% identify especially strong enhancement effects on suicide-related tasks.The paper attributes these differences as potentially reflecting the differing nature of the datasets.
  • 5.1.1 The Effectiveness of Enhancement Strategies.: Table 4 reports balanced accuracy across zero-shot, few-shot, and instruction-finetuning conditions, with standard deviations for prompt designs and dashes for infeasible few-shot datasets.Best and second-best results are marked within each column.
  • 5.1.1 The Effectiveness of Enhancement Strategies.: Prompt enhancements are generally effective for dialogue-focused models and critical action prediction, but less effective for task-solving-focused models and GPT-4.The paper suggests GPT-4 may already contain similar basic information in its knowledge space.

5.2 Few-shot Prompting Improves Performance to Some Extent

Few-shot prompting improves mental-health prediction performance over zero-shot prompting, but the gains remain limited and models still underperform task-specific systems.

  • 4.1% balanced accuracy improvement was observed with few-shot prompting over zero-shot prompting across tasks.
  • Few-shot prompting improved Alpaca and FLAN-T5 more than GPT-3.5 and GPT-4.Alpaca improved by 8.1% across all three tasks, while FLAN-T5 improved by 5.9% across all three tasks.
  • Few-shot prompting improved performance on most tasks, although models still underperformed task-specific models.
  • Few-shot prompting can improve mental-health prediction performance to some extent, especially for smaller models.

5.3 Instruction Finetuning Boost Performance for Multiple Tasks Simultaneously

Instruction finetuning substantially improves LLM performance across multiple mental-health tasks and datasets, with diverse training data supporting generalization and favorable performance from relatively small samples.

  • 5.3 Instruction Finetuning Boost Performance for Multiple Tasks Simultaneously: Instruction finetuning on six tasks across four datasets produced Mental-Alpaca and Mental-FLAN-T5.The models were trained simultaneously on all six tasks using multiple mental-health datasets.
  • 5.3 Instruction Finetuning Boost Performance for Multiple Tasks Simultaneously: 23.4% and 14.7% balanced accuracy gains over zero-shot versions were achieved by Mental-Alpaca and Mental-FLAN-T5, respectively.
  • 5.3 Instruction Finetuning Boost Performance for Multiple Tasks Simultaneously: 10.1% and 11.6% balanced accuracy gains over GPT-3.5’s best prompting versions were achieved by Mental-Alpaca and Mental-FLAN-T5 across all six tasks.
  • 5.3 Instruction Finetuning Boost Performance for Multiple Tasks Simultaneously: Mental-Alpaca and Mental-FLAN-T5 performed on par with Mental-RoBERTa while solving all tasks through a single finetuning round.Mental-RoBERTa is specialized for one task, whereas the two finetuned models operate across all tasks.
  • 5.3.2 Does Finetuning Generalize across Datasets?: Finetuning on a single dataset improved over zero-shot performance by 4.2%, but cross-dataset changes varied across tasks.
  • 5.3.2 Does Finetuning Generalize across Datasets?: Finetuning on diverse tasks improved external-dataset performance, with Mental-Alpaca and Mental-FLAN-T5 ranking first or second on 2/3 external tasks.The reported gains were 16.3% for Alpaca and 5.1% for FLAN-T5.
  • 5.3.3 How Much Data Is Needed?: With equal training-data sizes, broader data variation produced larger gains than single-dataset finetuning, reaching 12.4% at 20% data.The reported differences were 3.8% at 5%, 8.1% at 10%, and 12.4% at 20%.

5.4 Case Study of LLMs’ Capability on Mental Health Reasoning

The case study finds substantial variation in LLM mental-health reasoning: GPT-4 often provides more insightful analyses, while other models can generate shallow, incorrect, or persuasive but flawed explanations.

  • The study uses a small case study with Chain-of-Thought prompting to explore mental-health reasoning across four LLMs.The authors explicitly do not claim that particular LLMs are definitively better or worse at reasoning.
  • In the stress-prediction example, all models classified correctly, but their reasoning differed substantially in depth and quality.FLAN-T5’s explanation was shortest and superficial, whereas GPT-3.5 and GPT-4 offered more developed reasoning.
  • GPT-4 produced more organic, insightful, and concise reasoning than GPT-3.5 in the stress-prediction example.
  • In the depression-prediction example, GPT-4 correctly linked depressive symptoms to the user’s situation and provided insightful analysis.FLAN-T5 repeated the user’s facts, Alpaca predicted incorrectly, and GPT-3.5 gave an ambiguous prediction containing the correct answer.
  • Incorrect reasoning included hallucinated rationales, misunderstanding of “relief,” false positives, and explanations unrelated to the poster.
  • Reasoning errors can produce negative consequences and support the need for further ethical and safety research before real-world deployment.
  • False positives arose when models treated a stressful interview episode as evidence that the person was chronically stressed.The reasoning could read smoothly and logically despite misunderstanding the post’s main theme.
  • The case study suggests GPT-4 had the strongest reasoning capability, followed by GPT-3.5 and Alpaca, while FLAN-T5 was weaker at reasoning.

6 DISCUSSION

The discussion presents practical guidelines for improving LLMs on mental-health tasks while emphasizing limitations, ethical risks, and the gap to real-world deployment.

  • Guidelines: When computing resources are limited, carefully combine prompt design with few-shot prompting.Inference or API-based use may be more feasible than finetuning large models.
  • Guidelines: With sufficient resources, instruction finetuning across multiple mental-health datasets and tasks can improve generalizability.The authors identify single-round finetuning over merged datasets as especially effective.
  • Guidelines: Hundreds of varied examples may suffice for efficient finetuning when data resources are limited.The discussion favors data variation across sources over collecting a larger dataset from one source.
  • Guidelines: Models finetuned only on classification lose reasoning capability, motivating curated reasoning and causality datasets.Such datasets could help models learn relationships between mental-health outcomes and causal factors.
  • Limitations: LLMs make more mistakes in complex contexts, particularly when literal wording conflicts with underlying real-life scenarios.Models may be swayed by superficial sentiment and overlook cues associated with chronic conditions.
  • Limitations: Promising experimental performance does not establish deployability in real-world mental-health settings.The authors identify unresolved ethical, safety, robustness, bias, privacy, and regulatory concerns.
  • Limitations: The experiments mainly evaluate classification using online text, leaving regression, broader reasoning, sensor data, and other models or datasets insufficiently covered.The authors also note possible Reddit-related bias, uncertain training-data leakage, and the lack of fairness evaluation.

7 CONCLUSION

The conclusion reports a comprehensive evaluation of multiple LLMs on mental-health prediction from online text across prompting and instruction-finetuning settings. It highlights broad gains from instruction finetuning while retaining an exploratory view of reasoning capability and deployment limitations.

  • Conclusion: The paper evaluates multiple LLMs on binary and multi-class mental-health prediction using online text data.The evaluation includes zero-shot prompting, few-shot prompting, and instruction finetuning.
  • Conclusion: Instruction finetuning across multiple datasets significantly boosts performance across varied mental-health prediction tasks and external platforms.Context enhancement and few-shot prompting also improve performance, including with one example per class.
  • Conclusion: Mental-Alpaca and Mental-FLAN-T5 outperform larger LLaMA2, GPT-3.5, and GPT-4 models while performing on par with Mental-RoBERTa.The conclusion also presents reasoning results as promising but limited.
  • Conclusion: The paper distills its findings into guidelines for researchers, developers, and practitioners seeking to improve LLMs for mental-health tasks.These guidelines accompany the reported performance and reasoning findings.

APPENDIX: DETAILED RESULTS TABLES

The appendix provides a balanced-accuracy summary table covering zero-shot, few-shot, and instruction-finetuning results under several prompt-design strategies.

  • Table 10: Table 10 summarizes balanced-accuracy performance for zero-shot, few-shot, and instruction-finetuning settings.It compares context enhancement, mental-health enhancement, and their combination.
  • Table 10: The table identifies context, mental-health, and combined prompt strategies, with small numbers denoting standard deviations across prompt designs.Task-specific baselines are static and therefore have no standard deviation.
Loading 2307.14385v4…