CBT-LLM: A Chinese Large Language Model for Cognitive Behavioral Therapy-based Mental Health Question Answering

• The authors do not recommend using the model as a substitute for psychological treatment without professional supervision.Source: §1 · p. 1
• Pre-trained language models have potential for mental-health support, and preliminary systems have explored their use for psychological assistance.The cited systems include Liu et al. (2021), Cheng et al. (2023), and Lai et al. (2023).Source: §1 · p. 1
• Existing mental-health Q&A approaches still struggle to provide precise, effective support, and large datasets alone do not ensure constructive, empathetic responses.The introduction identifies a scarcity of data grounded in bona fide psychological counseling techniques and a gap between current systems and professional standards.Source: §1 · pp. 1–2
• The study creates a CBT-based counseling prompt and structured Chinese mental-health QA dataset, then instruction-fine-tunes CBT-LLM on that dataset.The authors present CBT’s structured intervention strategies as the theoretical foundation for the dialogue model.Source: §1 · p. 2
• Automatic metrics and human evaluation show CBT-LLM significantly outperforms existing models in mental health support.The introduction also reports that its responses adhere to CBT structure and are professional, structured, and relevant to users’ needs.Source: §1 · p. 2

2.1. Counseling Techniques

• CBT targets cognition–emotion–behavior links and distorted thoughts, while ACT, humanistic therapy, and DBT emphasize values-based flexibility, self-growth, and acceptance/change with mindfulness, respectively.The passage also describes counseling as supporting people in identifying, resolving, and coping with psychological issues.Source: §2.1 · p. 2

2.2. LLMs for Mental Health Support

• Mental-health LLM work includes ChatGPT analysis and interpretability, data-enrichment efforts such as ExTES and SMILE, and Psy-LLM feedback for professionals (Zheng et al., 2023; Qiu et al., 2023; Lai et al., 2023).ExTES provides emotion-support dialogue data, while SMILE extends single-turn dialogues to multi-turn interactions.Source: §2.2 · pp. 2–3
• LLMs are also being explored for counseling interventions, especially motivational interviewing and cognitive behavioral therapy (Welivita and Pu, 2023; Maddela et al., 2023).Source: §2.2 · p. 3

3.1. Problem Definition

• For each PsyQA question and description, ChatGPT applies the CBT prompts to generate an answer; the resulting question-description-answer triples make up the CBT QA dataset for instruction fine-tuning.Source: §3.1 · p. 3

3.2. Generation of CBT Responses

• The CBT Prompt generated answers with gpt-3.5-turbo-16k from PsyQA questions and descriptions, addressing the dataset’s limited CBT coverage and the impracticality of writing responses individually.The authors also tested a sample with GPT-4 and incorporated its prototype answer into the prompt for subsequent generation.Source: §3.2 · pp. 3–4
• The single-turn CBT response structure has five components: empathy, identifying beliefs, reflection, practical strategies, and encouragement.The components aim to guide CBT-consistent responses from validating feelings through encouraging strategy use and noting that further support may be needed.Source: §3.2 · p. 4

3.3.1. General Statistical Analysis

• 22,327 entries form the CBT QA dataset, with 522.8-character average responses and cognitive dissonance identified in 12,136 instances (54.4%).Questions average 21.6 characters and descriptions 168.9 characters.Source: §3.3.1 · p. 4, Table 1 · p. 4

3.3.2. Cognitive Distortion Statistical Analysis

• 7,115 samples (approximately 59%) show All-or-Nothing Thinking, while 7,782 (about 64%) show Overgeneralization among the 12,136 distortion instances.Distortions can co-occur; Emotional Reasoning, Catastrophizing, and Mind Reading account for 29% of cases.Source: §3.3.2 · p. 4, Table 2 · p. 5

3.3.3. Recognition of Quality Analysis

• 0.69 accuracy, 0.93 recall, and 0.65 F1 score measure the CBT Prompt’s cognitive-distortion recognition against professional annotations.The evaluation used 500 randomly selected samples; the authors interpret the accuracy–F1 gap as indicating false positives.Source: §3.3.3 · pp. 4–5, Table 3 · p. 5

3.4. CBT-LLM

• Instruction tuning and LoRA adapt the Transformer-decoder language model to CBT Q&A, with cross-entropy loss used for training.LoRA augments layers using low-rank matrices while leaving the original weights unchanged.Source: §3.4 · p. 5

4.1. Data Preparation

• The CBT QA dataset was split 90% for training and 10% for testing, with question and description as input and the CBT response as output.Each fine-tuning example used an instruction, input, and output, including a therapist-role instruction.Source: §4.1 · p. 6

4.2. Baselines

• The baselines were LLaMA-Chinese-7B (Cui et al., 2023), Alpaca-Chinese-7B (Cui et al., 2023), Qwen-7B (Bai et al., 2023), and Baichuan-7B.Their descriptions cite Chinese processing, instruction following, multilingual processing, and Chinese-language performance as relevant characteristics.Source: §4.2 · p. 6

4.3. Experimental Setups

• Training used an NVIDIA V100 32G GPU for three epochs, with gradient accumulation of four batches, a 5 × 10^-5 learning rate, cosine scheduling, and 16-bit precision.Fine-tuning was implemented with LLaMA Factory (hiyouga, 2023).Source: §4.3 · p. 6

4.4. Automatic Evaluation

• Automatic evaluation used BLEU, METEOR, CHRF, BLEURT, and BERTSCORE to assess lexical, character-level, and representation-based similarity against reference answers.The July toolkit (Cavusoglu et al., 2023) computed the metrics; results are presented in Table 4.Source: §4.4 · pp. 6–7

4.5. Main Results

• CBT-LLM outperformed LLaMA-Chinese-7B, Alpaca-Chinese-7B, and Qwen-7B on CBT structured-intervention tasks.The paper attributes the gap to CBT-LLM’s task-specific tailoring, while the baselines were optimized for Chinese processing, CBT adherence, or multilingual processing.Source: §4.5 · p. 7, Table 4 · p. 7

4.6. Human Evaluation

• The human evaluation found Baichuan-7B marginally outperformed Alpaca-Chinese-7B across relevance, CBT structure, and helpfulness.Four senior psychology students and one experienced psychotherapist rated 100 randomly selected test samples on three 0–2 scales.Source: §4.6 · p. 7, Table 5 · p. 7

4.7. Case Study

• The case study shows CBT-LLM validating emotions, identifying cognitive biases, offering practical strategies, and encouraging users facing social isolation and loneliness.The response emphasized social engagement and professional counseling, then offered positive encouragement.Source: §4.7 · pp. 7–8, Figure 4 · p. 8

Conclusions

• CBT-LLM combines a CBT-specific prompt and tailored Chinese mental-health QA dataset, with evaluations reporting structured, professional, and highly relevant support responses.The model was created by fine-tuning a large language model on the dataset.Source: §5 · p. 8

Future Work

• Integrating ACT and DBT alongside CBT could broaden the model to address diverse therapeutic needs.The paper identifies this as a future direction.Source: §5 · p. 8
• Moving from single-turn Q&A to multi-turn dialogue could better reflect counseling sessions and deepen model-patient interactions.The paper identifies this as a second future direction.Source: §5 · p. 8

• Without guided annotations for cognitive distortions, the model may misidentify them, reducing the precision and effectiveness of its CBT advice.Source: §6 · p. 8
• Delivering the comprehensive CBT process in one response may pressure or overwhelm users, particularly when it presents consecutive questions.Source: §6 · pp. 8–9

Ethical Statement

• The CBT QA dataset will be publicly released for research purposes only, and online forum questions have been anonymized to protect participant privacy.The release follows PsyQA data copyright protocols.Source: Ethical Statement · p. 9
• Dataset questions originate from online mental health forums, while responses are generated by ChatGPT rather than professionals; therefore, the work offers no therapeutic recommendations or diagnostic statements.Source: Ethical Statement · p. 9

A. Human Evaluation Guidelines

• Each evaluated entry consists of a question, its description, and the model-generated answer.Source: A. Human Evaluation Guidelines · p. 11
• Relevancy measures whether an answer addresses the question’s theme or core points rather than going off topic.Source: A. Human Evaluation Guidelines · p. 11
• Relevancy is scored 0 for irrelevant, 1 for partially relevant, and 2 for fully relevant answers.Source: A. Human Evaluation Guidelines · p. 11
• CBT structure assesses whether answers identify and challenge unhelpful thoughts, offer more beneficial alternatives, and possibly provide behavioral advice.Source: A. Human Evaluation Guidelines · p. 11
• CBT structure is scored 0 for nonadherence, 1 for partial adherence, and 2 for full adherence.Source: A. Human Evaluation Guidelines · p. 11
• The beneficial metric assesses counseling applicability and whether an answer provides genuine help, support, or guidance.Source: A. Human Evaluation Guidelines · p. 11
• Beneficial answers are assessed for avoiding potentially harmful, misleading, or confusing information and scored 0 for low, 1 for some, or 2 for high applicability.Source: A. Human Evaluation Guidelines · p. 11