MentaLLaMA: Interpretable Mental Health Analysis on Social Media with Large Language Models

1 INTRODUCTION

• Social media posts offer a source for mental health analysis and potential early intervention, but manual review cannot scale; PLMs struggle with unseen tasks and multi-task robustness, while discriminative predictions lack interpretability.ChatGPT can produce approaching-human explanations for correct classifications [45].Source: §1 · pp. 1–2
• ChatGPT’s zero-shot and few-shot [44] classification performance trails supervised state-of-the-art methods, and low precision can yield inaccurate reasoning [45]; fine-tuning is hindered by scarce reliable explanations and the absence of open-source models.Prompting or fine-tuning closed-source LLMs is also described as expensive, time-consuming, and carbon-intensive.Source: §1 · p. 2
• The authors collect data from 10 sources across 8 tasks, prompt ChatGPT with expert-written few-shot examples and annotations for explanations, and evaluate explanation correctness, consistency, and quality automatically and by human review.Posts, annotations, and explanations are converted into instruction-based query-answer pairs for training and evaluation.Source: §1 · p. 2
• On 7 of 10 test sets, MentaLLaMA-chat-13B surpasses or approaches SOTA discriminative methods in prediction correctness, while MentaLLaMA explanations match ChatGPT and outperform generative PLMs.MentaLLaMA models also show stronger predictive correctness than ChatGPT on unseen tasks and better explanation quality than generative PLMs.Source: §1 · p. 2
• MentaLLaMA is the first open-source instruction-following LLM series for interpretable mental health analysis, fine-tuned from LLaMA2 as 7B, chat-7B, and chat-13B models.Its explanation generation quality benefits from instruction tuning, reinforcement learning from human feedback, and increasing model size.Source: §1 · p. 2
• The IMHI benchmark contains 19K test samples covering 8 tasks and 10 test sets, and is presented as the first holistic benchmark for interpretable mental health analysis.The IMHI dataset contains 105K samples and is the first multi-task, multi-source instruction-tuning dataset for this analysis on social media.Source: §1 · p. 2

2 TASK FORMALIZATION

• The task is framed as generation, with a pretrained autoregressive language model adapted to solve multiple mental health analysis tasks and explain its decisions.Tasks include mental health detection and cause detection.Source: §2 · pp. 2–3
• Each task uses context-target pairs: the context contains a post and query, while the target contains an answer and natural-language rationale.The task-specific subsets are merged into one training dataset.Source: §2 · p. 3
• Training maximizes a conditional language-modeling objective to improve prediction correctness and rationale quality.Source: §2 · p. 3
• Prompts are constructed by concatenating three components.Source: Figure 2 · p. 3

3.1 Raw Data Collection

• The raw data comprises 10 annotated mental-health datasets drawn from social-media sources including Reddit, Twitter, and SMS.The datasets provide classification annotations used for explanation generation and evaluation.Source: §3.1 · p. 3

3.2 Explanation Generation with ChatGPT

• Experts wrote one task-specific instruction and 35 explanation examples for each of 10 datasets, creating a 350-sample gold set used in prompts with few-shot examples and target labels.The generated response format pairs a classification label with a reasoning explanation.Source: §3.2 · pp. 3–4

3.3.1 Automatic Evaluation.

• All classifiers exceeded 93.5% weighted F1 on test responses; on gold explanations, 9/10 exceeded 94% (4 reached 100%), while SAD scored 86.6%.The results support high consistency between generated explanations and labels, with lower performance on SAD where some cause labels have similar semantics.Source: Figure 3 · p. 4, §3.3.1 · pp. 4–5
• Seven of 10 datasets had annotation–ChatGPT agreement above 90%; T-SID was below 70%, and loneliness and IRF were below 80%.The passage attributes lower agreement to weakly supervised T-SID labels and subjectivity in loneliness and interpersonal-risk tasks.Source: Figure 3 · p. 4, §3.3.1 · p. 4
• Few-shot outputs improved over zero-shot outputs on all raw datasets, and with-label prompts also outperformed zero-shot explanations.BART-score comparisons used the 350 expert-written explanations as the gold standard.Source: Figure 3 · p. 4, §3.3.1 · p. 5

3.3.2 Human Evaluation.

• On a 0–3 scale, human evaluation found most ChatGPT-generated explanations above 2.5 in consistency and 2.0 in reliability, while professionality and overall evaluations also indicated good performance.Each sample was rated on all aspects by three domain experts (Quantitative Psychology Ph.D. students).Source: §3.3.2 · p. 5

3.4 Instruction Construction

• The instruction-format training split contains 72,095 mixed and randomized question–response samples, and its validation split contains 14,346 samples.Evaluated ChatGPT explanations serve as responses; an additional completion-form version, IMHI-completion, was created for baseline models with weak instruction following.Source: §3.4 · p. 5

4 MENTALLAMA TRAINING

• MentaLLaMA-7B is fine-tuned from LLaMA2-7B on IMHI for 10 epochs, with the best checkpoint selected using validation results.Training uses an actual batch size of 256, AdamW [24], a maximum learning rate of 1e-5, a 3% warm-up ratio, a 2048-token input limit, and Flash-Attention [7].Source: §4 · p. 5
• MentaLLaMA-chat-7B and -13B are trained on the same IMHI dataset and settings, starting from LLaMA2-chat models optimized with instruction tuning [30] and RLHF.The passage describes the LLaMA2-chat models as the first open-source LLMs tuned with RLHF.Source: §4 · p. 5
• For fair comparison with completion-based fine-tuned baselines, the authors also train LLaMA2-7B on the IMHI-completion dataset; all models use four 80GB Nvidia Tesla A100 GPUs.Source: §4 · p. 6

5 IMHI EVALUATION BENCHMARK

• The benchmark divides collected-dataset test data into 10 subsets because evaluation metric settings differ across data sources.Its statistics are presented in Table 1.Source: §5 · p. 6
• Correctness is evaluated as classification using weighted F1, with MentalBERT-based classifiers assigning labels to irregular model responses.This avoids relying on rules that can fail when instruction-tuned models do not use a unified response template.Source: §5 · p. 6
• Explanation quality is evaluated with BART-score [47], following the methods used for evaluating the IMHI dataset.Source: §5 · p. 6

6.1 Baseline Models

• The comparison includes discriminative PLMs, zero- and few-shot LLaMA2, ChatGPT, and GPT-4, plus completion-tuned BART-large, T5-large, and LLaMA-7B.Because discriminative models cannot generate text, they are compared only for correctness; BART and T5 are fine-tuned on the IMHI-completion dataset, as is LLaMA-7B for a fair comparison.Source: §6.1 · p. 6

6.2 IMHI Test Results

• MentaLLaMA-chat-13B surpasses or has a <5% gap to MentalRoBERTa on 7/10 test sets, while MentaLLaMA-7B beats completion-tuned LLaMA2-7B on 8/10.MentalBERT and MentalRoBERTa achieve state-of-the-art correctness on 8/10 test sets; ChatGPT zero-shot outperforms both LLaMA2 zero-shot models on all 10, and few-shot ChatGPT and GPT-4 outperform ChatGPT zero-shot on all test sets. T5 and BART outperform LLaMA2-7B on most test sets despite having 15% of its model size; MentaLLaMA-chat-7B and -13B outperform MentaLLaMA-7B on 9/10. The authors attribute gains to domain-specific instruction tuning and the chat models’ instruction tuning, and note limited generalization and interpretability for discriminative methods.Source: §6.2.1 · p. 6, Table 2 · p. 7

6.2.2 Quality.

• LLaMA2-7B beats its zero-shot version on all 10 test sets in BART-score, but beats BART-large on 9/10 with >0.2 improvement on only 2; the authors recommend BART-large as a capable, cost-efficient completion-based model.T5 and BART have similar scores, and the limited gains over BART-large support the authors’ conclusion that completion-based fine-tuning for LLaMA2 is inefficient.Source: §6.2.2 · p. 6, Figure 5 · p. 7
• MentaLLaMA beats LLaMA2-7B zero-shot on all 10 test sets; MentaLLaMA-chat-7B gains >0.2 on 6, and chat-13B gains >0.2 over chat-7B on 8.The authors attribute the chat models’ explanation-quality gains to instruction tuning and RLHF, and report that MentaLLaMA models perform comparably to few-shot ChatGPT and GPT-4 on most expert-written gold-set test cases despite their smaller sizes. GPT-4 shows no significant generation-quality improvement over ChatGPT, which has comparable correctness and quality at lower inference cost.Source: Figure 5 · p. 7, §6.2.2 · pp. 7–8

6.3 Generalizability

• On four tasks excluded from training, MentaLLaMA beats LLaMA2-13B zero-shot on all datasets and ChatGPT zero-shot on 3; MentaLLaMA-chat beats T5 and BART on Dreaddit and CAMS.The held-out tasks are Dreaddit, T-SID, CAMS, and IRF. MentaLLaMA-chat also scores better on T-SID despite exclusion of all Twitter training data, performs strongly on IRF, and chat-13B further improves explanation quality over chat-7B.Source: Figure 5 · p. 7, Table 3 · p. 8, §6.3 · p. 8

6.4 Human Evaluation

• MentaLLaMA explanations score over 2.2 on all four human-evaluation aspects and match ChatGPT in consistency and reliability, but significantly underperform it in professionality.The evaluation used 200 randomly sampled MentaLLaMA-chat-13B outputs; the authors note that BART-score has only moderate correlation with human evaluation and suggest continual pre-training on high-quality mental-health data to address the professionality gap.Source: Figure 6 · p. 8, §6.4 · p. 8

7 CONCLUSION AND FUTURE WORK

• MentaLLaMA approaches SOTA discriminative methods in correctness, generates human-level explanations, and generalizes to unseen tasks.Evaluations were conducted on the IMHI benchmark.Source: §7 · p. 8
• The paper proposes interpretable mental health analysis, a first multi-task and multi-source IMHI dataset with 105K samples, and the first open-source instruction-following LLM series for the task.ChatGPT was used to build training data, whose reliability was assessed through strict automatic and human evaluations.Source: §7 · p. 8
• Future work will use continual pre-training on high-quality mental health data to address limited domain knowledge and develop more reliable evaluation metrics than BART-score.BART-score showed only moderate correlations with human evaluation results for interpretable mental health analysis.Source: §7 · p. 8

8 ETHICAL CONSIDERATIONS

• The authors anonymize mental health texts under privacy protocols and ethical principles, and paraphrase and obfuscate paper examples using a moderate disguising scheme.Source: §8 · p. 9
• MentaLLaMA’s predictions and explanations should be used only for non-clinical research; helpseekers should consult psychiatrists or clinical practitioners.The authors cite potential gender bias, incorrect predictions, inappropriate explanations, and overgeneralization as risks that leave real-scenario monitoring challenging.Source: §8 · p. 9

A.1 Mental Health Analysis on Social Media

• Traditional mental-health classifiers commonly fine-tune PLMs such as BERT [8] and RoBERTa [23] on small target sets, usually for one condition.Some works instead pre-train on large-scale mental-health social-media data to improve post representations over general PLMs.Source: §A.1 · p. 10
• MentalBERT [19] and MentalXLNet [20] exemplify pre-training language models from scratch on large-scale mental-health social-media data.This approach can produce better post representations than general PLMs.Source: §A.1 · p. 10
• Earlier interpretability approaches added metaphor mappings [14], PHQ-9 information [27], or commonsense knowledge graphs to support model decisions.Source: §A.1 · p. 10
• Recent studies evaluated general foundation LLMs on mental-health tasks [44] [45], including explanation generation by Xu et al. [44] and human evaluation of ChatGPT by Yang et al..Source: §A.1 · p. 10

A.2 Open-source Large Language Models

• Closed-source ChatGPT and GPT-4 [29] motivated efforts to democratize LLMs, including Meta AI’s LLaMA series [35] and instruction-tuned Alpaca and Vicuna.These general instruction-following models sought to replicate ChatGPT-like abilities using large-scale instruction-tuning data [30].Source: §A.2 · p. 10
• Domain-specific instruction tuning has produced models such as biomedical MedAlpaca [15] and finance-focused Pixiu [43].Source: §A.2 · p. 10
• LLaMA-chat [35] were the first open-source LLMs enhanced with RLHF, which significantly aligns responses with human preferences.Source: §A.2 · p. 10

B HUMAN ANNOTATION SCHEME

• Annotators compare ChatGPT-generated explanations with expert-written explanations as the correct references.They score and annotate the generated explanations across the scheme’s evaluation aspects.Source: B HUMAN ANNOTATION SCHEME · p. 10
• Consistency is scored from 0 to 3 based on support for the classification and the explanation’s coherence, readability, and fluency.A score of 0 means inconsistent with the classification; 3 means completely fluent, coherent, and error-free.Source: B HUMAN ANNOTATION SCHEME · pp. 10–11
• Reliability is scored from 0 to 3 by assessing factual support for the classification and misinformation or faulty reasoning against the post.The scale ranges from factual hallucination at 0 to completely reliable information at 3.Source: B HUMAN ANNOTATION SCHEME · p. 11
• Professionality is judged against depression symptoms informed by PHQ-9 and expert criticality rankings, with suicide, self-harm, depressed mood, and self-guilt marked high-criticality.The rubric scores whether explanations include supportive evidence from the post, penalizing missing symptoms according to their criticality.Source: B HUMAN ANNOTATION SCHEME · p. 11
• Overall Score measures a generated explanation’s general effectiveness by combining consistency, reliability, and professionality; excellent performance (3) requires all three dimensions to be excellent (=3).Score 0 applies when consistency or reliability is <=1. Score 1 requires both to be >=2 and professionality to be <=1; score 2 requires both to be =3 and professionality to be <=2.Source: B HUMAN ANNOTATION SCHEME · p. 11

C EXAMPLES

• Table 5 presents examples of ChatGPT disagreements with annotated labels.Source: Table 5 · p. 12

C.1 Templates for IMHI Dataset

• Table 4 presents prompt-construction templates for the IMHI and IMHI-completion dataset.Source: §C.1 · p. 12

• The paper provides examples of ChatGPT disagreeing with annotated labels in Table 5.Source: §C.2 · p. 12