ChatCounselor: A Large Language Models for Mental Health Support

1 INTRODUCTION

• Professional counselors face high costs, complicated appointments, and limited resources that can delay immediate assistance for mental health concerns.Source: §1 · p. 1
• Although LLMs may offer initial assessments and early interventions, general-domain systems often lack counseling-specific training and expertise.Source: §1 · p. 1
• Psych8k comprises conversations with 260 individuals and is intended to support research on personalized psychological counseling assistants.Source: §1 · p. 1
• The Psych8k-tuned model outperforms existing open-source models on the counseling benchmark.The authors attribute the improvement to a small amount of high-quality domain-specific data.Source: §1 · p. 2
• The proposed benchmark uses GPT-4 for automated evaluation of psychological counseling assistants.The authors describe it as a rapid assessment method for development.Source: §1 · p. 2

2.1 LLMs for psychological service

• Research on psychological-service LLMs includes counseling and peer support, emotional-support dialogue systems, multi-turn conversation datasets, and safety benchmarking.The paper positions its focus as condensing information from real counseling conversations to make feedback more professional.Source: §2.1 · p. 2

2.2 Personalized large language models

• Personalized LLMs tailor responses to user preferences, context, and characteristics; prior evaluation spans generation and recommendation, while this paper adds a question-answering dataset covering 260 individuals.Source: §2.2 · p. 2

3 METHODOLOGY

• ChatCounselor integrates authentic counseling data and psychological knowledge through instruction tuning and autoregressive training, targeting informative, professional responses with counseling skills and a professional tone.Source: §3 · p. 2

3.1 Data Collection

• 260 accredited, experienced counselors provided real-life recordings to address concerns about online answers lacking professional accreditation and offering advice before learning clients’ backgrounds.The recordings were transcribed and processed to make conversations concise.Source: §3.1 · p. 2
• The conversations cover five major mental-health topics, including anxiety, depression, stress, relationships, and concerns affecting minority groups.The topic distribution is shown in Figure 2.Source: §3.1 · p. 2, Figure 2 · p. 3

3.2 Data Processing

• Transcripts were cleaned to remove personally identifiable and other sensitive information that could reveal participants’ identities or compromise data integrity.Source: §3.2.1 · p. 2

3.2.1 Data Cleaning & Information Extraction.

• Because transcripts contained intonation-only or low-information turns, GPT-4 segmented and summarized them in sections of 10 rounds to support informative query-answer generation.Figure 3 shows an example extraction and reformatting prompt.Source: §3.2.1 · pp. 2–3, Figure 3 · p. 3

3.2.2 Generate instruction data for Instruction Tuning.

• GPT-4 distilled 8,187 query-answer pairs from ten-round clips, with prompts encouraging interactive elements such as questions and guidance for clients.The resulting Psych8k dataset contains 2 million tokens; Figure 4 shows an example distillation prompt.Source: §3.2.2 · p. 3, Figure 4 · p. 3

3.3 Counseling-specific Instruction Tuning

• Instruction tuning minimizes autoregressive loss on concatenated instruction and input fields, with the output serving as the response; at inference, the user’s question is the instruction.Source: §3.3 · p. 3

• ChatCounselor starts from Vicuna-v1.3-7B, already fine-tuned on general-purpose instruction data, and is fine-tuned on the 8K counseling set for 5 epochs.Source: §4 · p. 3
• Training uses a 2048-token context, batch size 256, AdamW with learning rate 2e-5, and a 0.03 warm-up ratio.Source: §4 · p. 3
• The model is trained with FSDP, bf16, and gradient checkpointing on 8 A100-40GB GPUs within 1 hour.Source: §4 · p. 3

5.1 Counseling Bench

• The evaluation framework assesses counseling from seven perspectives, including information, guidance, emotional support, reflection, interpretation, self-disclosure, and information gathering.The seven metrics are intended to assess counseling strategies and emotional support.Source: §5.1 · p. 4
• The benchmark comprises 229 challenging, single-turn open-ended questions covering counseling scenarios such as addiction, anxiety, depression, and relationships.The questions are designed to assess chatbot performance across a range of counseling scenarios.Source: §5.1 · p. 4

5.2 Evaluation with GPT-4

• GPT-4 compares responses from different models across the assessed aspects and provides brief rationales, which the authors found made its choices more robust.The prompts also ask GPT-4 to consider counseling-strategy use, suitability of suggestions, counselor-like tone, and response length bias.Source: §5.2 · p. 4

5.3 Results

• ChatCounselor outperforms LLaMA-7B [37], Alpaca-7B [35], ChatGLM-v2-7B [7], and Robins-v2-7B [6]; Vicuna-v1.3-7B approaches it on suggestions and self-disclosure but lags on information gathering and reflection.The comparison is presented as GPT-4 evaluation in Figure 7.Source: §5.3 · p. 4, Figure 7 · p. 5
• ChatCounselor still trails GPT-3.5-turbo (ChatGPT); the authors note that GPT-4 judging favors longer, suggestion-listing answers, while some surveyed users preferred more interactive replies.The paper also reports that some users are less inclined toward ChatGPT-style extensive suggestion listing.Source: §5.3 · p. 5
• The authors report that fine-tuning on dialogue-format data lets the model learn counseling techniques and produce more natural dialogue flow.They state that carefully curated domain-specific data can yield competitive results.Source: §5.3 · p. 5

6 CONCLUSION

• ChatCounselor is a mental-health-support LLM incorporating real counseling conversations and specialized psychology knowledge.The conclusion frames this as the basis for personalized mental-health support.Source: §6 · p. 5
• ChatCounselor outperforms existing open-source models on the Counseling Bench and approaches ChatGPT’s performance.Source: §6 · p. 5
• The conclusion says domain-specific fine-tuning enables ChatCounselor to generate interactive and meaningful responses.Source: §6 · p. 5

A STRATEGIES DEFINITION

• Response quality is measured using seven metrics.Source: A STRATEGIES DEFINITION · p. 6
• The appendix provides a definition for each metric.Source: A STRATEGIES DEFINITION · p. 6
• Each metric is accompanied by its corresponding strategies and examples.Source: A STRATEGIES DEFINITION · p. 6

B MORE EXAMPLES

• The authors say ChatGPT tends to offer suggestions without proactively encouraging clients to express more about themselves or their thoughts.Source: B MORE EXAMPLES · p. 6
• The authors say this suggestion-focused style can lead clients to perceive ChatGPT as AI rather than as a counselor who listens.Source: B MORE EXAMPLES · p. 6
• The examples compare ChatCounselor and ChatGPT in stress and relationship dialogues, with ChatCounselor shown on the left and ChatGPT on the right.Source: B MORE EXAMPLES · p. 6, Figure 8 · p. 7, Figure 9 · p. 8