PaperlayerView exact revision on arXiv

Source-linked AI summary

Evaluation of ChatGPT for NLP-based Mental Health Applications

Bishal Lamichhane

arXiv:2303.15727v1cs.CLcs.AI

TL;DR

The paper examines whether ChatGPT can classify mental-health states from social-media posts, an application considered relevant because such posts may reflect mental state. It evaluates zero-shot classification across stress, depression, and suicidality, reporting performance above dominant-class baselines across the tasks.

  • Problem

    The paper examines whether language models can support NLP-based mental-health detection from social-media posts, which may reflect a person’s mental state.

    Source: §I · p. 1

  • Method

    The authors evaluated ChatGPT’s zero-shot classification of labeled social-media posts across stress, depression, and suicidality tasks.

    Source: §I · p. 1, B. ChatGPT Evaluation · p. 2

  • Results

    ChatGPT performed better than dominant-class baselines across the three tasks; depression F1 was 0.86 versus 0.60 for the baseline.

    Source: §IV · p. 2, §III · p. 2

  • Takeaways & Limitations

    The authors conclude that ChatGPT can infer a user’s mental-health state to an extent from social-media posts.

    Source: §IV · p. 2

    The evaluation used GPT-3.5-turbo, a limited prompt setting and first responses, and relatively small datasets.

    Source: §IV · p. 3

Abstract

from arXiv · show

Large language models (LLM) have been successful in several natural language understanding tasks and could be relevant for natural language processing (NLP)-based mental health application research. In this work, we report the performance of LLM-based ChatGPT (with gpt-3.5-turbo backend) in three text-based mental health classification tasks: stress detection (2-class classification), depression detection (2-class classification), and suicidality detection (5-class classification). We obtained annotated social media posts for the three classification tasks from public datasets. Then ChatGPT API classified the social media posts with an input prompt for classification. We obtained F1 scores of 0.73, 0.86, and 0.37 for stress detection, depression detection, and suicidality detection, respectively. A baseline model that always predicted the dominant class resulted in F1 scores of 0.35, 0.60, and 0.19. The zero-shot classification accuracy obtained with ChatGPT indicates a potential use of language models for mental health classification tasks.

I. INTRODUCTION

The introduction motivates measuring mental health from social-media language and considers LLMs as a possible NLP approach. It evaluates ChatGPT’s zero-shot classification of three mental-health states using public labeled datasets.

I. INTRODUCTION

  • More than one billion people worldwide are estimated to live with mental illnesses, and timely diagnosis, intervention, and monitoring require measuring mental health.The introduction notes that social-media posts may reflect mental state through sentiment or linguistic content.Source: §I · p. 1
  • LLMs are presented as language-understanding models relevant to NLP-based mental-health detection, while prior examples include GPT-4 applications in programming, mathematics, and medical question answering.The passage describes transformer-based models trained on large text corpora and subsequently fine-tuned.Source: §I · p. 1
  • The authors identify possible mental-health uses ranging from interpreting user interactions and adapting responses to serving as a backend for front-end applications.They also mention suggesting that users seek medical care as a possible intervention.Source: §I · p. 1
  • The study evaluates ChatGPT’s zero-shot classification of stress, depression, and suicidality from users’ social-media posts using publicly available labeled datasets.Source: §I · p. 1

1) Stress Detection Dataset:

The stress-detection evaluation uses labeled Reddit posts and tests ChatGPT on the dataset’s held-out test posts.

1) Stress Detection Dataset:

2) Depression Detection Dataset:

The depression dataset combines depression-related and non-depression Reddit posts for binary classification.

2) Depression Detection Dataset:

3) Suicidality Detection Dataset:

The suicidality task uses expert-labeled social-media posts from a 500-user subset and distinguishes five categories.

3) Suicidality Detection Dataset:

B. ChatGPT Evaluation

ChatGPT classified user posts from the three mental-health detection tasks using a prompt that requested a class label. The evaluation used single-call predictions from the gpt-3.5-turbo backend and compared them with dataset annotations.

  • ChatGPT API received each user's post alongside a classification prompt.Source: B. ChatGPT Evaluation · p. 2
  • The prompt restricted replies to task class names, with stressed and non-stressed as the stress-task example.Source: B. ChatGPT Evaluation · p. 2
  • Using gpt-3.5-turbo, the authors compared single-call predictions with annotations, computed F1 scores including weighted F1 for multiclass, and evaluated confusion matrices.Source: B. ChatGPT Evaluation · p. 2

III. RESULTS

ChatGPT achieved higher F1 scores than dominant-class baselines on the three detection tasks. Confusion matrices summarize predictions, although some posts could not be evaluated because of reading or API invocation errors.

  • F1 scores were 0.72 for stress detection, 0.86 for depression detection, and 0.37 for five-class suicidality, versus dominant-class baselines of 0.35, 0.60, and 0.19, respectively.Source: §III · p. 2, Table I · p. 2
  • Confusion matrices are shown for all three tasks, but some posts could not be evaluated because of dataset-reading or API-invocation errors.Source: §III · p. 2, Figure 1 · p. 2

IV. DISCUSSION

The discussion presents zero-shot ChatGPT as promising for stress and depression, but weaker for five-class suicidality. It also identifies adaptation and evaluation limitations as important boundaries.

IV. DISCUSSION

  • ChatGPT showed good zero-shot classification against the baseline across three tasks; the authors suggest adapting it for specific mental-health applications.The paper describes ChatGPT as a general-purpose model and proposes further fine-tuning or adaptation.Source: §IV · p. 2
  • Figure 3 assigns unclassifiable inputs to None; predictions mostly fell in ideation, with related suicidality classes likely having overlapping boundaries.The authors also report that found ideation to be the most commonly predicted category and reported high confusion among suicidality classes.Source: Figure 3 · p. 3, §IV · p. 3
  • F1 was 0.73 for ChatGPT on stress detection, below the fine-tuned BERT model’s 0.81, though the evaluation was zero-shot.The BERT result used in-domain training data.Source: §IV · p. 3
  • F1 was 0.86 for ChatGPT on depression detection, versus 0.93 for a neural network using bigram, LIWC, and LDA features.The authors suggest fine-tuning could further improve ChatGPT’s performance.Source: §IV · p. 3
  • Evaluation was limited to GPT-3.5-turbo, a narrow prompt setting, first responses, relatively small datasets, and annotations not consistently reviewed by multiple experts.The authors propose evaluating GPT-4, exploring prompts and repeated responses, using larger corpora, and re-annotating datasets with experts.Source: §IV · p. 3
Loading 2303.15727v1…