PaperlayerView exact revision on arXiv

Source-linked AI summary

CPsyCoun: A Report-based Multi-turn Dialogue Reconstruction and Evaluation Framework for Chinese Psychological Counseling

Chenhao Zhang, Renhao Li, Minghuan Tan, Min Yang, Jingwei Zhu, Di Yang, Jiahao Zhao, Guancheng Ye, Chengming Li, Xiping Hu

arXiv:2405.16433v3cs.CLcs.AIcs.CY

TL;DR

Authentic counseling-session datasets are scarce, and multi-turn counseling dialogue evaluation remains an understudied area. CPsyCoun reconstructs dialogues from counseling reports and proposes a dedicated benchmark; experiments report that its fine-tuned model outperforms other models in the benchmark.

  • Problem

    Authentic counseling-session datasets that include client symptoms and counselor treatment methods are scarce, and automatic evaluation of multi-turn counseling dialogues is understudied.

    Source: §1 · pp. 1–2

  • Method

    CPsyCoun processes anonymized counseling reports with privacy shadowing and a two-phase reconstruction method, then proposes a benchmark for automatic multi-turn dialogue evaluation.

    Source: §1 · p. 2

  • Results

    CPsyCounX, fine-tuned on CPsyCounD, outperforms other models in the benchmark, and the authors report that experiments validate the framework's effectiveness.

    Source: §1 · p. 2, §5 · p. 9

  • Takeaways & Limitations

    The framework provides report-derived counseling dialogues and a multi-turn evaluation benchmark, with datasets and model weights publicly available.

    Source: §1 · p. 2, §5 · p. 9

    The framework still needs to balance dialogue authenticity with professional knowledge and account for counselors' timing of empathy and counseling techniques.

    Source: Limitations · p. 9

Abstract

from arXiv · show

Using large language models (LLMs) to assist psychological counseling is a significant but challenging task at present. Attempts have been made on improving empathetic conversations or acting as effective assistants in the treatment with LLMs. However, the existing datasets lack consulting knowledge, resulting in LLMs lacking professional consulting competence. Moreover, how to automatically evaluate multi-turn dialogues within the counseling process remains an understudied area. To bridge the gap, we propose CPsyCoun, a report-based multi-turn dialogue reconstruction and evaluation framework for Chinese psychological counseling. To fully exploit psychological counseling reports, a two-phase approach is devised to construct high-quality dialogues while a comprehensive evaluation benchmark is developed for the effective automatic evaluation of multi-turn psychological consultations. Competitive experimental results demonstrate the effectiveness of our proposed framework in psychological counseling. We open-source the datasets and model for future research at https://github.com/CAS-SIAT-XinHai/CPsyCoun

1 Introduction

CPsyCoun addresses the shortage of authentic counseling-session dialogue data and the lack of automatic multi-turn counseling evaluation by reconstructing dialogues from reports and proposing a benchmark.

  • Authentic counseling-session datasets are scarce, while real-world counseling information in structured reports is unsuitable for model training and raises privacy concerns.Source: §1 · p. 2
  • The framework combines report-based dialogue reconstruction with a benchmark for automatic multi-turn consultation evaluation.The authors collect anonymized reports, apply privacy shadowing to create CPsyCounR, and use Memo2Demo to construct CPsyCounD.Source: §1 · p. 2
  • The proposed benchmark for automatic multi-turn counseling evaluation includes comprehensive evaluation metrics, datasets, and methods.Source: §1 · p. 2
  • CPsyCounD contains 3,134 high-quality multi-turn consultation dialogues, and CPsyCounX outperforms other models in the benchmark.The authors present this as evidence for the framework's effectiveness in psychological counseling.Source: §1 · p. 2

2 Related Work

Prior work explores LLM-based dialogue reconstruction and increasingly semantic evaluation, while this paper proposes a two-phase reconstruction method and a counseling-specific benchmark.

2.1 Dialogue Generation and Reconstruction using LLMs

  • SAFARI uses LLM planning and understanding for persona-consistent, knowledge-enhanced responses, while DISC-MedLLM reconstructs medical consultation records from forums (Wang et al., 2023a; Bao et al., 2023).Source: §2.1 · pp. 2–3
  • The authors propose a two-phase method for efficient dialogue reconstruction.Source: §2.1 · p. 3

2.2 Evaluation of Generated Dialogues using LLMs

  • Transformer-based and LLM-based evaluation metrics include BERTScore, BARTScore, GPTScore, and G-EVAL, which uses chain-of-thoughts and form filling (Zhang et al., 2020; Yuan et al., 2021; Fu et al., 2023; Liu et al., 2023b).Source: §2.2 · p. 3
  • The authors design a psychological counseling benchmark for automatic evaluation.Source: §2.2 · p. 3

3 CPsyCoun

CPsyCoun combines report-based dialogue reconstruction with an evaluation framework for multi-turn counseling. Its methods organize reports, generate dialogues through two phases, and assess counseling quality across multiple turns.

3.1 Data Collection

  • The authors collected 4,700 counseling reports from online sources; these reports are not publicly released unless a Privacy Data Protection Agreement is signed upon reasonable request.The collection drew on Chinese psychological communities including Yidianling and Psy525.Source: §3.1 · p. 3

3.2 Data Processing

  • CPsyCounR comprises 3,134 selected single-case reports, standardized into six fields and grouped into nine representative topics and seven counseling schools.The six fields are Title, Type, Method, Case Brief, Consultation Process, and Experience Thoughts; the consultation process is written in third person and contains no specific dialogue.Source: §3.2 · pp. 3–4

3.3 Dialogue Generation Method for Psychological Counseling

  • Memo2Demo addresses baseline gaps in completeness, professionalism, and authenticity by converting a report into a supervisor’s note before counselor dialogue generation.The supervisor’s note condenses core client information and adds a consultation plan; the counselor then uses the report and note with a four-stage consultation framework.Source: §3.3 · pp. 4–5

3.4 Automatic Evaluation of LLM-based Psychological Counseling

  • Four perspectives—Comprehensiveness, Professionalism, Authenticity, and Safety—support automatic turn-based evaluation, with GPT-4 scoring each response and averaging scores across turns.The method evaluates each counselor reply with its query and dialogue history before producing a dialogue-level score.Source: §3.4 · p. 6
  • CPsyCounE contains 45 manually selected dialogues: five representative SMILECHAT cases for each of nine counseling topics.SMILECHAT comprises 56k multi-turn dialogues averaging 6.36 rounds, but its open-source data lacks topic classification.Source: §3.4 · p. 6

4 Experiments

Experiments show that Memo2Demo improves reconstructed-dialogue quality over direct role-play, while CPsyCounX performs strongly against other models on professional and authentic counseling. The evaluations also expose a trade-off between comprehensiveness and authenticity.

4.2 Intrinsic Evaluation of CPsyCoun

  • 53% in Comprehensiveness, 53% in Professionalism, and 30% in Authenticity were Memo2Demo’s improvements over direct role-play on 140 cases.The evaluation sampled 20 cases from each of seven counseling schools and used GPT-4 for comparative evaluation.Source: Table 2 · p. 7, §4.2 · p. 7

4.1 CPsyCounD

  • CPsyCounD contains 3,134 multi-turn consultation dialogues covering nine topics and seven classic counseling schools.The dialogues were generated with Memo2Demo from CPsyCounR; direct role-play prompting was used as a comparison method.Source: Table 1 · p. 6, §4.1 · p. 7

4.3 Extrinsic Evaluation of CPsyCoun

  • The extrinsic evaluation uses turn-based dialogue assessment and compares CPsyCounX with InternLM2-7B-Chat, SoulChat, ChatGPT, and GLM-4.The evaluation adjusts the authenticity criterion for multi-turn dialogues.Source: §4.3 · p. 7
  • CPsyCounX surpasses other models in Professionalism and Authenticity, while GLM-4 is slightly higher in Comprehensiveness; CPsyCounX leads all metrics on Mental Disease.Figure 4 compares detailed scores across nine counseling topics.Source: Table 3 · p. 8, §4.3 · p. 8, Figure 4 · p. 8
  • GLM-4’s higher Comprehensiveness is attributed to information-rich single-turn dialogues and summarizing prior turns, while excessive content can compromise Authenticity.The authors prioritize natural, authentic dialogues with key information and report that CPsyCounX learns counseling techniques and style through fine-tuning on CPsyCounD.Source: §4.3 · pp. 7–8

5 Conclusion

CPsyCoun combines report-based multi-turn dialogue reconstruction with an evaluation benchmark for Chinese psychological counseling. The experiments support the framework’s effectiveness, while future work must better balance authenticity and professional knowledge.

5 Conclusion

  • CPsyCoun combines a two-phase approach to construct consultation dialogues from counseling reports with a benchmark for multi-turn dialogue evaluation.The benchmark includes evaluation metrics, datasets, and methods.Source: §5 · pp. 8–9
  • Experimental results support the framework’s effectiveness in building a comprehensive, professional, and authentic psychological counseling assistant.Source: §5 · p. 9
  • The authors identify a more refined balance between authenticity and professional knowledge as a future-work need, and report that the datasets and model weights are publicly available.Source: §5 · p. 9

Limitations

The paper identifies balancing authenticity with professional knowledge as an outstanding challenge for dialogue generation and evaluation in complex counseling scenarios.

Limitations

  • Counseling requires empathy and techniques used at appropriate times; mistimed techniques may harm problem-solving.Source: Limitations · p. 9
  • Dialogue generation and evaluation still need to balance authenticity with professional knowledge.Source: Limitations · p. 9

Ethics Statement

The ethics statement describes data sanitization for privacy, warns that model responses may harm users, and explains the report-formatting approach.

Data Privacy

  • Sanitization combines rule-based cleaning, manual rewriting, and proofreading to remove sensitive information; the evaluation dataset is released for research purposes only.Source: Data Privacy · p. 9

Potential Risks of the Model

  • Without human feedback during fine-tuning, some responses may harm users; the model is not a replacement for therapy and should be presented as AI-generated reference material.The authors recommend seeking professional help if interaction with the model does not noticeably improve the user’s situation.Source: Potential Risks of the Model · p. 9

A Unified Format of Psychological Counseling Reports

B Prompts of Dialogue Reconstruction Methods

The section contrasts direct role-play prompting with Memo2Demo, a two-phase method for dialogue reconstruction. It identifies the prompts used for the baseline and each Memo2Demo phase.

B Prompts of Dialogue Reconstruction Methods

C Evaluation Metrics and Score Criterion

The section proposes four automatic evaluation metrics for multi-turn psychological counseling dialogues and points to Table 4 for their descriptions and scoring criteria.

C Evaluation Metrics and Score Criterion

D Prompts of Automatic Evaluation

GPT-4 is used as a judge for intrinsic and extrinsic evaluation. The section describes comparing reconstructed consultations for intrinsic evaluation and comparing model responses after fine-tuning CPsyCounX for extrinsic evaluation.

D Prompts of Automatic Evaluation

E Full Results of extrinsic evaluation

The section reports full extrinsic-evaluation results for CPsyCounX and baseline models. Table 5 organizes scores by counseling topic and marks the best and second-best scores.

E Full Results of extrinsic evaluation

Loading 2405.16433v3…