PaperlayerView exact revision on arXiv

Source-linked AI summary

PsychEval: A Multi-Session and Multi-Therapy Benchmark for High-Realism AI Psychological Counselor

Qianjun Pan, Junyi Wang, Jie Zhou, Yutao Yang, Junsong Li, Kaiyin Xu, Yougen Zhou, Yihan Li, Jingyuan Zhao, Qin Chen, Ningning Zhou, Kai Chen, Liang He

arXiv:2601.01802v3cs.AI

TL;DR

Existing benchmarks and metrics do not capture counseling’s longitudinal process or its therapeutic quality, while counseling requires continuity and adaptation across sessions. PsychEval constructs a multi-session, multi-therapy benchmark and evaluation framework; the paper reports higher benchmark performance and presents it as a reinforcement-learning environment for counselor training.

  • Problem

    Existing benchmarks are often static, single-turn, or limited to specific modalities, while counseling requires longitudinal client tracking and progression across sessions.

    Source: §1 · p. 1, §2.2 · p. 3

  • Method

    PsychEval builds a multi-session, multi-therapy benchmark from structured clinical cases and evaluates client simulation and counselor proficiency with therapy-specific and shared metrics.

    Source: §1 · p. 2, §4.2 · p. 5

  • Results

    PsychEval significantly outperforms existing benchmarks, with a CTRS score of 9.19 and a WAI score of 7.26.

    Source: §7 · p. 11

  • Takeaways & Limitations

    PsychEval serves beyond static evaluation as a high-fidelity reinforcement-learning environment for training clinically responsible and adaptive AI counselors.

    Source: §1 · p. 2

    The benchmark uses text interactions only, excluding non-verbal cues such as vocal prosody, facial expressions, and body language.

    Source: §8 · p. 11

Abstract

from arXiv · show

To develop a reliable AI for psychological assessment, we introduce \texttt{PsychEval}, a multi-session, multi-therapy, and highly realistic benchmark designed to address three key challenges: \textbf{1) Can we train a highly realistic AI counselor?} Realistic counseling is a longitudinal task requiring sustained memory and dynamic goal tracking. We propose a multi-session benchmark (spanning 6-10 sessions across three distinct stages) that demands critical capabilities such as memory continuity, adaptive reasoning, and longitudinal planning. The dataset is annotated with extensive professional skills, comprising over 677 meta-skills and 4577 atomic skills. \textbf{2) How to train a multi-therapy AI counselor?} While existing models often focus on a single therapy, complex cases frequently require flexible strategies among various therapies. We construct a diverse dataset covering five therapeutic modalities (Psychodynamic, Behaviorism, CBT, Humanistic Existentialist, and Postmodernist) alongside an integrative therapy with a unified three-stage clinical framework across six core psychological topics. \textbf{3) How to systematically evaluate an AI counselor?} We establish a holistic evaluation framework with 18 therapy-specific and therapy-shared metrics across Client-Level and Counselor-Level dimensions. To support this, we also construct over 2,000 diverse client profiles. Extensive experimental analysis fully validates the superior quality and clinical fidelity of our dataset. Crucially, \texttt{PsychEval} transcends static benchmarking to serve as a high-fidelity reinforcement learning environment that enables the self-evolutionary training of clinically responsible and adaptive AI counselors.

1 Introduction

PsychEval addresses the gap between static chatbot benchmarks and the longitudinal, theory-sensitive work of counseling. It combines multi-session, multi-therapy training with a two-level evaluation framework and is presented as both a benchmark and a reinforcement-learning environment.

1 Introduction

  • Real counseling requires memory continuity, evolving-goal tracking, and progression across stages, while generic lexical metrics miss therapeutic quality and clinical appropriateness.Existing benchmarks often use single-turn interactions that miss changes in clients over time and theoretical versatility.Source: §1 · p. 2
  • PsychEval simulates counseling’s three-stage trajectory across 6–10 sessions, five therapeutic modalities plus an integrative approach, and over 2,000 client profiles.Its stages are Case Conceptualization, Core Intervention, and Consolidation.Source: §1 · p. 2
  • PsychEval grounds training in empirically sourced case reports and uses a hierarchical skill taxonomy to support coherence and goal tracking across sessions.The taxonomy maps strategic meta-skills to executable atomic actions.Source: §1 · p. 2
  • Its evaluation framework combines Client-Level simulation fidelity and Counselor-Level clinical proficiency, using therapy-specific and shared metrics.The hybrid metrics are intended to assess versatility and specialization.Source: §1 · p. 2
  • PsychEval is reported to achieve the highest quality, session depth, and real-world counseling alignment among compared benchmarks, while also serving as a high-fidelity reinforcement-learning environment.The environment simulates diverse, evolving client states with reward functions for agent training.Source: §1 · p. 2

2 Related Work

Prior work spans empathy-focused counseling agents, single-modality models, and benchmarks for classification, advice, competency, dialogue, and safety. PsychEval is positioned as a longitudinal multi-therapy benchmark with evaluation spanning client simulation and counselor adherence.

2.1 LLMs for Psychological Counseling

  • Prior counseling agents address empathy, therapeutic reasoning, or continuity, but many models remain confined to one therapeutic approach or cannot flexibly switch strategies.Examples include SoulChat [9], PsyLLM [10], HealMe [11], SMILE [12], SouLLMate [17], and Chain-of-interaction [18].Source: §2.1 · pp. 2–3

2.2 Psychological Benchmarks

  • Existing benchmarks cover classification, single-turn advice, counselor knowledge, multi-turn dialogue, or safety, but lack a unified continuous multi-session framework across modalities [1].PsychEval is presented as simulating the full longitudinal counseling process across diverse therapeutic schools.Source: §2.2 · p. 3

2.3 Psychological Evaluation Frameworks

  • PsychEval’s evaluation system combines Client-Level simulation feedback with Counselor-Level clinical-adherence metrics for holistic assessment.Prior evaluation approaches include safety and ethics metrics, human ratings, and LLM-based patient simulation [30] [31].Source: §2.3 · p. 3

3 Background

The benchmark covers five therapeutic approaches alongside integrative therapy, and organizes counseling into a shared three-stage workflow. The stages establish a case formulation, adapt interventions through monitoring, and consolidate gains while preparing for termination.

3.1 Different Psychological Therapies

  • The benchmark includes Psychodynamic, Behavioral, Cognitive-Behavioral, Humanistic-Existential, and Postmodernist therapies, alongside Integrative therapy.The approaches differ in theory, techniques, and focal points.Source: §3.1 · pp. 3–4

3.2 Unified Counseling Workflow

  • The unified workflow moves from Case Conceptualization through Core Intervention to Consolidation and Prevention.Conceptualization typically spans 1–3 sessions, intervention 3–5, and consolidation 1–3.Source: §3.2 · p. 4

4 Benchmark Construction

PsychEval is constructed by converting clinical case reports into structured cases and using a skill-informed pipeline to generate multi-stage dialogues. Its process combines empirically grounded source material, structured planning and dialogue features, and preparation, execution, and consolidation across sessions.

4.1 Data Collection

  • The dataset contains 369 peer-reviewed clinical case reports, including 148 Cognitive Behavioral Therapy reports, 50 each for Psychoanalysis, Humanistic, and Postmodernist approaches, 43 Behaviorism reports, and 28 Integrative Therapy reports.The reports are described as providing verified frameworks, theoretical orientations, and longitudinal trajectories.Source: §4.1 · p. 5

4.2 Structured Case Extraction

  • An intermediate extraction step converts raw reports into four structured modules: Client Profile, Therapeutic Plan, Dialogue Features, and Therapeutic Skills.The authors use this schema to preserve clinical-context granularity before dialogue reconstruction.Source: §4.2 · p. 5
  • The Therapeutic Plan provides a three-phase session roadmap with chronological and completeness constraints, while Dialogue Features supply verbatim 3–5-turn exemplars and stylistic guidance.The plan fields include session theme, persona links, case material, rationale, and psychoeducation skills.Source: §4.2 · p. 6

4.3 Multi-stage Dialogue Construction

  • The hierarchical skill taxonomy links strategic Meta Skills to executable Atomic Skills, and retrieval narrows skills by workflow stage and session goals.Before each utterance, reasoning covers client assessment, alliance tracking, skill selection, and strategy formulation; post-session consolidation updates the profile and next-session plan.Source: §4.2 · p. 6, §4.3 · p. 7
  • The skill-informed pipeline organizes dialogue construction into Pre-Session Preparation, In-Session Execution, and Post-Session Consolidation to ground interventions and maintain consistency across the counseling arc.Preparation retrieves skills, execution generates counselor turns, and consolidation updates memory and planning.Source: §4.3 · p. 7

5 Holistic Evaluation Framework

PsychEval’s holistic evaluation framework uses external supervision and psychometric instruments to assess clinical and technical efficacy beyond lexical metrics. It distinguishes universal from therapy-specific competencies and tracks therapeutic progress across sessions.

5 Holistic Evaluation Framework

  • The framework combines external supervisory assessment with longitudinal trajectory analysis to evaluate clinical and technical efficacy beyond lexical metrics such as BLEU.It uses surrogate supervision to distinguish universal from therapy-specific competencies.Source: §5 · p. 7

5.1 Counselor-Level Evaluation

  • Counselor-level evaluation assesses longitudinal coherence and protocol adherence, using relational instruments and four domains: clinical perception, strategy, depth, and ethical safety.Shared measures include WAI, HTAIS, and RRO.Source: §5.1 · p. 7
  • Therapy-specific counselor scales assess CBT and psychodynamic fidelity, empathy and affective processing in Humanistic/Behavioral approaches, and change-discourse evocation in Postmodern therapies.The framework uses CTRS, PSC, TES, EFT-TFS, and MITI.Source: §5.1 · p. 7

5.2 Client-Level Evaluation

  • Client-level evaluation uses an LLM supervisor to turn dialogue into standardized clinical scores and assess simulation fidelity, clinical consistency, and psychological shifts.Source: §5.2 · p. 8
  • Client-level shared measures track symptoms and affect, counselor authenticity, and the working alliance, including potential relationship ruptures.The instruments are SCL-90, PANAS, RRO, and SRS.Source: §5.2 · p. 8
  • Client-level therapy-specific measures track depression in CBT, personality integration in Psychodynamic therapy, anxiety in Behavioral approaches, self-acceptance in Humanistic approaches, and solution-oriented discourse in Postmodern interventions.The measures include BDI-II, IPO, STAI, CCT, and SFBT Indicators.Source: §5.2 · pp. 8–9

6 Analysis of Benchmark

PsychEval reports strong counselor- and client-level quality measures and coherent client-state changes across sessions. Its benchmark combines longitudinal interactions, hierarchical skills, and broad therapy coverage.

6.1 Statistic Information

  • PsychEval comprises 369 cases across five therapeutic genres, averaging 7.6 sessions per case (up to 10), 24.1 turns per session, and 94.7 words per turn.Its hierarchical taxonomy includes 677 Meta and 4,577 Atomic skills.Source: §6.1 · p. 9

6.2 Comparison with Existing Benchmark

  • PsychEval averages 7.6 sessions and 92.9 words per turn, compared with ESConv's 16.3 words per turn, and shifts benchmark focus toward longitudinal, skill-aware assessment.The authors describe it as the only open-source dataset combining turn-level reasoning, multi-stage workflows, and hierarchical skill annotations, alongside five therapy genres and an integrative approach.Source: §6.2 · pp. 9–10

6.3 Data Quality Assessment

  • CTRS 9.19 and PSC 7.27 indicate strong therapy-specific adherence, alongside WAI 7.26 and HTAIS 6.40.The authors report that CTRS and PSC scores are nearly double those of prior models such as PsyDial and SimPsyDial.Source: §6.3 · p. 10, Table 5 · p. 10
  • SRS 8.29 marks the highest client satisfaction and alliance, while SCL-90 1.48 is the largest reduction in symptom distress.The client-level measures assess post-session psychological states and longitudinal changes.Source: §6.3 · pp. 10–11, Table 6 · p. 10
  • Adjacent changes show BDI-II -0.45 and IPO -0.09, while CCT +0.14 and SFBT +0.16; Psy-Insight trends reverse.The paper treats session-to-session differences as proxies for progress.Source: Table 7 · p. 10, §6.3 · p. 11

7 Conclusions and Future Work

PsychEval combines authentic clinical cases, multi-session structure, and a hierarchical skill taxonomy in a benchmark and training ecosystem. The authors report strong clinical-rating scores and position it as a reinforcement-learning environment for counselor training.

7 Conclusions and Future Work

  • PsychEval combines 369 authentic clinical case reports, an average of 7.6 sessions per case, a three-stage framework, and over 4,500 atomic skills.The taxonomy is intended to support coarse-to-fine clinical reasoning.Source: §7 · p. 11
  • PsychEval significantly outperforms existing benchmarks, achieving a CTRS score of 9.19 and a WAI score of 7.26.Source: §7 · p. 11
  • PsychEval is presented as a high-fidelity reinforcement-learning environment for training clinically responsible, logically coherent, and ethically aligned AI counselors.The authors identify developing a self-evolving counselor for autonomous, adaptive mental health care as future work.Source: §7 · p. 11

8 Limitations

PsychEval’s current scope is bounded by text-only interactions, its 369 published case reports, and a Chinese linguistic and socio-cultural context. The authors identify multimodal, high-risk, and cross-cultural extensions as future directions.

8 Limitations

  • Text-only evaluation excludes non-verbal counseling cues such as vocal prosody, facial expressions, and body language.These modalities are described as challenging to incorporate at scale and beyond current text-based evaluation protocols.Source: §8 · p. 11
  • The 369 professionally published case reports necessarily underrepresent acute suicidality and highly resistant clients.The passage attributes this boundary to the emphasis on clinically representative, ethically shareable trajectories and broader data-availability challenges.Source: §8 · p. 11
  • PsychEval is currently grounded in the Chinese linguistic and socio-cultural context, while mental-health expressions, help-seeking behaviors, and therapeutic norms vary across cultures.Source: §8 · p. 11

9 Ethical Statement

PsychEval is built from ethically reviewed, consented published case reports and is released for academic research with restrictions on clinical use. Its coverage spans five named therapeutic schools and an integrative approach.

9 Ethical Statement

  • The dataset uses 369 publicly available, peer-reviewed clinical case reports whose original publications underwent ethical review and obtained informed consent.It excludes social media, private records, and unregulated platforms; release is restricted to academic research, and trained systems are designated auxiliary tools or simulations, not substitutes for clinicians.Source: §9 · p. 11

A Different Psychological Therapies

  • The benchmark includes Psychodynamic, Behavioral, Cognitive-Behavioral, Humanistic-Existential, and Postmodernist therapies alongside an Integrative therapy.The therapies are presented as five major orientations plus an integrative category.Source: A Different Psychological Therapies · p. 15
  • The orientations differ in focus: unconscious dynamics, learned behavior, cognitive appraisal, growth and meaning, or socially constructed narratives.Integrative therapy combines techniques or theories across traditions, including empathetic listening with cognitive restructuring.Source: A Different Psychological Therapies · p. 15

B Evaluations

The Holistic Evaluation Framework uses clinical instruments and external supervision to assess both client simulation fidelity and counselor proficiency. Its metrics cover shared competencies, modality-specific skills, longitudinal progress, and customized counselor domains.

B Evaluations

  • The framework operationalizes validated psychometric instruments, including SCL-90 and WAI, with LLMs serving as surrogate expert supervisors for observer-rated assessments.It is designed to assess clinical efficacy and technical performance beyond lexical-overlap measures such as BLEU or ROUGE.Source: B Evaluations · p. 16
  • Shared metrics assess universal competencies such as rapport and empathy, while therapy-specific metrics assess modality-dependent technical precision.Examples include CBT cognitive restructuring and Psychodynamic analysis of defense mechanisms.Source: B Evaluations · p. 16
  • Session-to-session differential scores (∆Score) track client improvement and the counselor’s strategic, long-term treatment planning over time.Source: B Evaluations · p. 16

B.1 Counselor-Level Evaluation

  • Counselor-Level evaluation assesses clinical proficiency, longitudinal coherence, and adherence to modality-specific technical requirements through external supervision.Its shared measures include alliance, facilitative behavior, and genuineness; modality-specific scales include CTRS for CBT and PSC for Psychodynamic therapy.Source: §B.1 · pp. 16–17
  • Customized counselor assessment covers clinical perception, intervention strategy, conversational therapeutic depth, and clinical ethics and safety.These domains include emotional recognition, longitudinal memory and adaptation, dialogue coherence, crisis-marker identification, professional boundaries, and cultural sensitivity.Source: §B.1 · p. 16

B.2 Client-Level Evaluation

  • Client-Level evaluation measures simulation fidelity and authentic psychological shifts using an LLM-based supervisor to score session transcripts.Shared measures include SCL-90 symptom dimensions, PANAS affect, RRO bond authenticity, and SRS alliance; modality-specific measures include BDI-II, IPO, CCT, SFBT indicators, and STAI.Source: §B.2 · p. 17

C Statistic Information

PsychEval annotates counseling with a hierarchical skill taxonomy and reports stage-level metrics aligned with a three-stage clinical framework.

C Statistic Information

  • The taxonomy comprises 677 Meta Skills and 4,577 Atomic Skills, including 210 meta and 1,144 atomic skills in Psychodynamic therapy.The hierarchy spans strategic-level skills and tactical-level skills for turn-level interventions.Source: C Statistic Information · p. 18
  • Stage 1 has 7.47 average P-Links; Stage 2 averages 4.13 sessions and 5.92 C-Materials, while Stage 3 supports structured closure.These distributions are reported as reflecting personalization at intake, sustained intervention, and closure, respectively.Source: C Statistic Information · p. 18

D Experimental Settings and Dataset Construction Details

The construction process combines clinical source materials, constrained dialogue generation, and longitudinal memory and planning prompts. Its prompts use a shared framework while adapting skills and stage-specific guidance to therapy and session goals.

D.1 Model Configurations

  • Generation used GPT-5 with high reasoning effort, and DeepSeek-V3.1 served as the primary LLM-as-a-Judge; comparative analysis sampled 1,000 instances per existing benchmark.The evaluator choice was based on Chinese linguistic comprehension and logical reasoning, with selection also balancing evaluation effectiveness and computational cost.Source: §D.1 · p. 18

D.2 Data Reconstruction and Quality Control

  • Quality control enforced session timelines, generated session agendas from global plans, and used researcher audits followed by regeneration or manual correction.The audits assessed clinical logic, role adherence, and annotation accuracy.Source: §D.2 · p. 18

D.3 Authoritative Psychology Resources

  • Case reports came partly from two peer-reviewed Chinese psychology journals, while textbooks informed skill extraction and prompt design.The cited resources include works on counseling and therapy, CBT, existential-humanistic therapy, behavioral treatment, psychodynamic formulation, and solution-focused brief therapy.Source: §D.3 · pp. 18–19

E.1 Prompt for Client Profile

  • The client-profile prompt extracts static traits, clinical presentation, growth experience, and therapy-specific parameters from case reports; the therapeutic-plan prompt applies constraints to improve extraction accuracy and mitigate misleading information.The illustrated profile prompt is the CBT version; the plan prompt transforms case reports and profiles into a therapeutic plan.Source: §E.1 · p. 21, §E.2 · p. 21

E.3 Prompt for Dialogue Features

  • Authentic counseling dialogues supply few-shot exemplars, while a counselor-and-client contextual field and stylistic analysis guide naturalistic, clinically faithful interactions.The session-goal prompt combines the longitudinal plan with historical session data and authentic clinical agendas, with conversational boundaries intended to mitigate hallucinations and off-topic deviations.Source: §E.3 · p. 21, §E.4 · pp. 21, 27

E.5 Prompt for Skill Suggestion

  • A library of over 4,500 atomic skills supports session-level selection, with the prompt directing the model to choose the 60 most relevant skills for each session goal.The selected skills are intended to ground dialogue in professional therapeutic methodology.Source: §E.5 · p. 27

E.6 Prompt for Dialogue Generation

  • Dialogue generation combines case extraction, session objectives, prior-session summaries, and relevant skills in stage- and therapy-specific prompts under a unified framework.The detailed example is the Integrative Therapy prompt for Stage 1; the prompts differ in stage concepts and supplied therapeutic skills.Source: §E.6 · p. 27

E.7 Prompt for Client Memory Extraction

  • Memory prompts extract information disclosed during each session, merge new insights into the evolving client profile, and produce clinical summaries that support continuity and future longitudinal planning.The merge prompt addresses redundancy, logical contradictions, and conceptual confusion; the clinical summary integrates dialogue summaries, memory extraction, and planning.Source: §E.7 · p. 35, §E.8 · p. 35, §E.9 · p. 35

Related papers

Loading 2601.01802v3…