Source-linked AI summary
MentalChat16K: A Benchmark Dataset for Conversational Mental Health Assistance
Jia Xu, Tianyi Wei, Bojian Hou, Patryk Orzechowski, Shu Yang, Ruochen Jin, Rachael Paulbeck, Joost Wagenaar, George Demiris, Li Shen
TL;DR
Conversational mental-health counseling lacks sufficient domain-specific, privacy-conscious training data despite growing interest in LLM-based support. MentalChat16K combines synthetic counseling conversations with anonymized intervention transcripts and evaluates fine-tuned models through automated and human judgments. Fine-tuning generally improves performance over base models, while the study’s model coverage is limited because newer systems such as DeepSeek were unavailable.
Problem
Conversational mental-health counseling remains underdeveloped, constrained by language limitations, scarce domain-specific training data, and privacy concerns.
Method
MentalChat16K combines synthetic counseling conversations and anonymized intervention transcripts, then fine-tunes seven lightweight LLMs with QLoRA and evaluates responses using automated and human judges.
Results
Fine-tuning on synthetic, interview, or combined data generally improves performance across metrics compared with base models, with human results consistent with GPT-4 and Gemini Pro evaluations.
Takeaways & Limitations
MentalChat16K provides a privacy-conscious benchmark and pipeline for developing and evaluating empathetic, personalized conversational mental-health assistance.
Takeaways & Limitations
The experiments did not include newer state-of-the-art models such as DeepSeek because they were unavailable at the time of the study.
Abstract
from arXiv · showhide
We introduce MentalChat16K, an English benchmark dataset combining a synthetic mental health counseling dataset and a dataset of anonymized transcripts from interventions between Behavioral Health Coaches and Caregivers of patients in palliative or hospice care. Covering a diverse range of conditions like depression, anxiety, and grief, this curated dataset is designed to facilitate the development and evaluation of large language models for conversational mental health assistance. By providing a high-quality resource tailored to this critical domain, MentalChat16K aims to advance research on empathetic, personalized AI solutions to improve access to mental health support services. The dataset prioritizes patient privacy, ethical considerations, and responsible data usage. MentalChat16K presents a valuable opportunity for the research community to innovate AI technologies that can positively impact mental well-being. The dataset is available at https://huggingface.co/datasets/ShenLab/MentalChat16K and the code and documentation are hosted on GitHub at https://github.com/ChiaPatricia/MentalChat16K.
1 INTRODUCTION
MentalChat16K addresses limited domain-specific data for conversational mental health counseling by combining synthetic counseling conversations with anonymized intervention transcripts. The paper presents an efficient fine-tuning and evaluation pipeline, with experiments showing improved performance over base models.
- Existing mental-health LLM research emphasizes detection, diagnosis, and prediction, while conversational counseling remains comparatively underdeveloped because of language, data-scarcity, and privacy constraints.
- 16K question-answer pairs combine synthetic mental-health counseling conversations with anonymized interviews involving Behavioral Health Coaches and caregivers of palliative or hospice patients.The dataset covers conditions including depression, anxiety, and grief.
- MentalChat16K prioritizes patient privacy, ethical considerations, and responsible data usage while supporting empathetic and personalized AI assistance.
- Seven state-of-the-art LLMs were fine-tuned with QLoRA, focusing on lightweight 7B models to demonstrate a resource-conscious pipeline.The authors describe the approach as suitable for limited resources such as a single A40 or A100 GPU.
- Fine-tuned models generally outperform their base models across evaluation metrics, with GPT-4, Gemini Pro, and human evaluations showing consistent patterns.The evaluation used 200 counseling questions and seven metrics, with human results consistent with GPT-4 and Gemini Pro judgments.
2 RELATED WORK
Related work applies LLMs to mental-health tasks, but conversational datasets and counseling-oriented resources remain important for advancing NLP applications in this domain. Figure 1 summarizes the paper’s data, fine-tuning, inference, and evaluation pipeline.
- Mental-health research has applied LLMs to tasks including suicide-risk detection, psychotherapy homework assignment, emotion recognition, counselor support, and depression diagnosis.
- The pipeline collects synthetic and paraphrased interview data, fine-tunes lightweight 7B models with QLoRA, elicits responses to 200 questions, and evaluates them with seven metrics.
- Existing mental-health conversation datasets include face-to-face or video-conference interactions with verbal and non-verbal content.
3 APPROACH
MentalChat16K combines anonymized intervention interviews with synthetic counseling conversations, then uses resource-efficient fine-tuning and mixed automated-human evaluation to assess mental-health counseling models.
- 3.1 Data Collection and Processing: MentalChat16K comprises real anonymized interview transcripts and synthetic mental health counseling conversations.The interview data come from behavioral intervention sessions with caregivers of palliative or hospice patients, while the synthetic data were generated with GPT-3.5 Turbo.
- 3.1.1 Interview Data: 378 interview transcripts were paraphrased into 6,338 question-answer pairs after filtering short exchanges and manually removing sensitive information.A local Mistral-7B-Instructv0.2 model processed transcript pages into single caregiver-coach conversations; pairs with fewer than 40 words were excluded.
- 3.1.2 Synthetic Data: 9,775 synthetic question-answer pairs span 33 mental health topics and were randomly reviewed for authentic counseling style and suitable guidance.GPT-3.5 Turbo generated patient queries and responses using a customized Airoboros self-generation framework.
- 3.3 Evaluation: Evaluation used 200 counseling questions, seven metrics, GPT-4 Turbo and Gemini Pro judges, and human evaluators.Judge models rated responses from 1 to 10, while human evaluation ranked responses from 1 to 7 across sampled questions.
- 3.3 Evaluation: Human evaluators achieved a Cohen’s Kappa agreement of 0.441, exceeding the stated acceptable threshold of 0.4.The agreement was calculated from evaluator prediction lists for the human-ranking task.
4 EXPERIMENTS
The experiments fine-tune seven local open-source LLMs on MentalChat16K and evaluate them with specialized mental-health metrics, automated judges, and human experts. Fine-tuning generally improves performance, while GPT-4 favors synthetic data and Gemini places greater value on interview data for some dimensions.
- Metrics: The evaluation uses seven mental-health-specific metrics because conventional measures do not adequately assess empathy, emotional sensitivity, and ethical adherence.The metrics are intended to provide a comprehensive framework tailored to counseling applications.
- Training configurations: Models are fine-tuned separately on synthetic data, interview data, or the combined MentalChat16K dataset.These configurations test scenario-based learning, natural conversational dynamics, and the effect of diverse training input.
- Main results: Fine-tuning generally improves performance across metrics relative to base models under GPT-4, Gemini Pro, and human evaluation.The reported pattern holds for models trained on synthetic data, interview data, or both.
- Evaluator-specific patterns: GPT-4 favors synthetic-data fine-tuning, whereas Gemini more often values interview-data fine-tuning, especially for Safety & Trustworthiness and Boundaries & Ethical.Synthetic-data models win Active Listening comparisons across all seven base models under GPT-4, while interview-data models win 7, 7, 7, 6, 4, 5, and 6 of seven metric comparisons under Gemini.
- Statistical analysis: 18 of 21 Gemini-evaluated and 17 of 21 GPT-4-evaluated fine-tuned models differ significantly from their base model in at least 6 of 7 metrics.The comparison uses five inference rounds on 50 randomly selected questions and two-sample t-tests with a 0.95 confidence interval.
5 ETHICAL CONSIDERATIONS
The study protects participants through informed consent, private sessions, confidentiality procedures, and secure institutional data storage.
- Privacy and confidentiality: Participants signed informed-consent documents, sessions were conducted privately, and recordings and study data were stored on secure institutionally backed systems.Audio files and study data used unique identifiers rather than identifiable labels.
6 LIMITATIONS
The paper identifies limitations involving dataset composition, conversational structure, population and language coverage, evaluation, and model availability. These constraints affect how broadly the findings and dataset can be applied.
- Dataset composition: Combining synthetic and interview data did not consistently improve performance and sometimes degraded it, so the datasets should be handled separately with caution.The synthetic data may also lack the depth of real human interactions.
- Scope and generalizability: The dataset is primarily English, and interview data from caregivers of palliative or hospice patients may not generalize to other linguistic, cultural, or population settings.The interview population differs from the broader profile of the synthetic data.
- Evaluation: Automated and human evaluations may not fully capture real-world counseling efficacy because evaluator models introduce bias and human judgments remain subjective.The reported moderate inter-rater agreement indicates room for improvement in evaluation methodology.
- Model coverage: The experiments did not include newer models such as DeepSeek because they were unavailable when the study was conducted.Future work can evaluate these models and their distilled variants across multiple sizes.
- Conversational structure: Breaking interviews into isolated question-answer pairs removes multi-turn conversational flow, although preserving original order may support partial context reconstruction.Follow-up questions and clarifications are not preserved, and a multi-turn subset is planned for future work.
7 CONCLUSION
MentalChat16K combines synthetic counseling conversations with anonymized real-life intervention interviews to address the need for domain-specific mental health training data. The authors report that fine-tuning on the dataset supports empathetic and personalized mental health assistance.
- Dataset contribution: MentalChat16K combines synthetic counseling conversations and anonymized real-life intervention interview data in a benchmark dataset.The dataset is presented as a resource for developing large language models for mental health support.
- Research contribution: The dataset addresses the need for domain-specific training data for large language models in mental health support.The conclusion connects this resource with empathetic and personalized interactions.
- Reported outcome: The evaluation framework reports superior performance from models fine-tuned on MentalChat16K in delivering nuanced and compassionate mental health assistance.The conclusion describes the evaluation as using state-of-the-art models and advanced metrics.
A.1 Details of Fine-tuning Technique
The fine-tuning approach uses LoRA adapters and QLoRA quantization to reduce trainable parameters and memory requirements while keeping the pretrained model weights frozen. This enables fine-tuning large language models with limited computational resources.
- Low-rank adaptation: QLoRA adapts LoRA for efficient fine-tuning by compressing weight updates into two low-rank matrices while retaining the pretrained weights.The trainable matrices are updated through backpropagation, whereas the original weight matrix remains frozen.
- Resource efficiency: Together, QLoRA techniques make it possible to fine-tune models with billions of parameters on a single GPU.The method is intended to reduce computational resource and time requirements.
- Low-rank adaptation: LoRA reduces trainable parameters and accelerates computation by using a low-rank update instead of directly updating the full weight matrix.The update rank is much smaller than the input and output dimensions.
- Quantization: QLoRA quantizes pretrained weights with 4-bit NF4 and stores trainable adapter weights in BF16 for computation.Double quantization further reduces memory by quantizing the quantization constants.
- Quantization: Double dequantization converts quantization constants and pretrained weights into BF16 during computation.These operations reduce the model memory footprint for fine-tuning.
A.2 Related Work
Prior mental-health NLP research has addressed detection, prediction, therapy support, and counseling datasets, but resources and systems for English conversational counseling remain limited. Existing datasets include text-based and live counseling conversations, with differences in language, access, and data scale.
- Mental health context: Mental health disorders such as depression and anxiety create substantial personal and socioeconomic challenges.The paper cites an estimated annual global productivity loss of approximately $1 trillion due to these disorders.
- AI applications: AI systems have been applied to mental health tasks including suicide-risk detection, psychotherapy homework assignment, and emotion recognition during therapy.These applications predate the newer generation of large language models.
- LLMs in mental health care: Large language models have been evaluated for mental health prediction tasks, while emotional-support chatbots provide on-demand conversational support as a supplement to traditional therapy.Instruction fine-tuning has been reported to improve performance across multiple prediction tasks.
- Benchmark datasets: Mental health benchmark datasets draw on text-based and live counseling conversations, including SNAP, Chinese PsyQA, and GPT-generated counseling question-answer resources.These resources differ in access requirements, language, and counseling orientation.
A.3 Dataset Metadata
The caregiver survey included 421 respondents, predominantly female and White Caucasian.
- 421 caregivers completed the demographic information survey.
- Female caregivers formed the majority, while male caregivers totaled 6.
- Among female caregivers, White Caucasians comprised approximately 88%, with less than 1% identifying as Hispanic.
A.3.1 Demographic Statistics of Caregivers in the Anonymous Study.
The caregiver data is skewed toward White female hospice caregivers, consistent with demographic patterns reported in prior hospice-care research. The synthetic dataset spans 33 mental health topics, including anxiety and depression.
- A.3.1 Demographic Statistics of Caregivers in the Anonymous Study.: The caregiver dataset is skewed toward White female participants, consistent with demographic patterns in hospice populations and research participation.Prior reviews found predominantly female and above-average White/Caucasian representation in many hospice-care studies.
- A.3.1 Demographic Statistics of Caregivers in the Anonymous Study.: The anonymous caregiver study is summarized in Table 4 as demographic statistics for caregivers.
- A.3.2 Topic Distribution For Synthetic Data.: The synthetic dataset covers 33 mental health topics, including relationships, anxiety, depression, intimacy, and family conflict.Topic proportions were specified from CounselChat patterns to reflect topics arising in counseling sessions.
A.4 Prompts
The appendix documents prompts, evaluation procedures, visualized results, score distributions, implementation settings, costs, and representative dataset examples. Across these materials, MentalChat16K combines synthetic and interview data and evaluates models using multiple LLM and human-centered criteria.
- A.4 Prompts: The appendix includes prompts for paraphrasing interview data, generating synthetic queries, evaluating responses, and applying LLM-judge rubrics.These materials are organized in Tables 5–8 and the prompts section.
- A.5 Statistical Analysis of Results: Table 9 compares GPT-4 Turbo and Gemini Pro scores across seven mental health metrics, with P-values and human rankings.Highlighted scores identify the best result from each LLM judge, and significant P-values are marked when below 0.05.
- A.6 Human and LLM-Based Evaluation Correlation: Holistic Approach showed the strongest human–LLM evaluation correlation for both GPT-4 and Gemini at 0.489.Empathy & Validation and Active Listening followed with GPT-4 correlations of 0.433 and 0.411, respectively.
- A.7 Visualization of Results: The visualizations compare base, synthetic-data fine-tuned, interview-data fine-tuned, combined-data fine-tuned, and baseline models for each metric.Figures 4–10 cover Active Listening, Empathy & Validation, Safety & Trustworthiness, Open-mindedness & Non-judgment, Clarity & Encouragement, Boundaries & Ethical, and Holistic Approach.
- A.8 Score Distribution: Across 200 evaluation examples, GPT-4 Turbo and Gemini-Pro scores generally clustered between 7 and 9 while varying across the full 1–10 range.GPT-4 Turbo showed more lower-score variability, and both judges differentiated among responses within the high-quality range.
- A.9 Hyperparameters: The appendix specifies QLoRA-based training settings and reports API usage, token counts, request counts, and total spending.These implementation details appear in the hyperparameter and API-cost sections.
- A.11 Data Examples: Representative interview-data examples show responses that acknowledge distress, validate feelings, encourage professional help, and avoid harmful advice.The examples compare Mistral-Instruct-V0.2 with a version fine-tuned on MentalChat16K.