Cactus: Towards Psychological Counseling Conversations using Cognitive Behavioral Theory

1 Introduction

• Because actual counseling data is not publicly available amid privacy and identification concerns, realistic synthetic data offers an alternative for training open-source models.The paper notes that conversations often contain highly personal information sent to third-party providers by closed-source models.Source: §1 · p. 2
• Existing synthetic counseling datasets often use single-turn strategies or only 2–3 turns, leaving therapeutic consistency and progression across longer dialogues unaddressed.The paper identifies this as a need for more realistic psychological counseling data.Source: §1 · p. 2
• CACTUS is a publicly available multi-turn counseling dataset designed to capture the depth and flow of realistic interactions.Human evaluations find it more helpful and empathetic than existing counseling datasets.Source: §1 · p. 2
• COUNSELINGEVAL assesses counselor communication skills and clients’ psychological changes across multi-turn dialogues.It uses AI clients and modified real-world evaluation methods including CTRS and PANAS.Source: §1 · p. 2
• CAMEL, trained on CACTUS, outperforms other baseline models in counseling abilities.The paper presents COUNSELINGEVAL results from both LLM-based and psychological expert evaluation.Source: §1 · p. 2

2 Design Considerations for CACTUS

• CACTUS simulates an AI counselor guiding clients through CBT reframing, using PatternReframe personas for the AI clients.The setup assumes an initial intake session has already provided basic client information.Source: §2 · p. 2

2.1 Can LLMs be Competent Counselors?

• GPT-4o selected Evidence-Based Questioning in almost half of cases, while experts selected CBT techniques more evenly.The comparison used 150 randomly selected client thoughts and patterns annotated by psychological experts.Source: Figure 2 · p. 2, §2.1 · p. 3
• AI counselors may give clients direct reframes instead of helping them discover their own solutions, risking client resistance and less independent reframing.The paper describes the counselor’s role as fostering a collaborative relationship rather than providing instant solutions.Source: §2.1 · p. 3

2.2 Can LLMs act like Real Clients?

• AI clients may state provided information using psychological terminology rather than the detailed contextual stories real-life clients use.For example, a simulated client says, “It might be labeling.”Source: §2.2 · p. 3
• AI clients can remain overly positive and compliant, lacking the range of attitudes, including strongly negative ones, seen in real-life clients.The paper gives “I am willing to try it” as an example of a compliant response.Source: §2.2 · p. 3

3.1 Dataset Construction

• The dataset uses PatternReframe personas, thoughts, and patterns as client-simulation contexts because the thoughts mirror actual clients and psychology-background contributors helped create the resource.Script mode was selected over two-agent mode because it produced more natural, well-constructed dialogues; longer sessions were generated to reflect typical 30–60-minute counseling.Source: §3.1 · p. 4
• Counselors receive clients’ negative thoughts, patterns, and reframed thoughts, then plan around the three most suitable CBT techniques to support clients’ independent discovery of solutions.The process aims to guide clients rather than directly suggest reframed thoughts.Source: §3.1 · p. 4
• Client simulation turns negative thoughts into detailed narratives and intake-form personas, with three attitudes and associated behaviors representing varied counseling interactions.The intake form includes help-seeking reasons, name, gender, and age.Source: §3.1 · p. 4

3.2 Analysis of CACTUS

• CACTUS dialogues show significantly greater helpfulness and empathy than dialogues generated without CBT planning.The ablation comparison reports p < 0.05 for both criteria.Source: Figure 7 · p. 5, §3.2 · p. 5

3.1 Dataset Construction

• Filtering removes malformed or too-short dialogues and retains only those averaging at least 5 across six CTRS criteria, leaving 31,577 conversations or 86.3% of the initial set.The six criteria cover general counseling and CBT-specific skills, each scored from 0 to 6.Source: §3.1 · p. 5

3.2 Analysis of CACTUS

• CACTUS contains 31,564 dialogues and approximately 1 million utterances, with varied client backgrounds, attitudes, and behaviors.Personas vary in therapy reason, age, and occupation; client attitudes include positive, neutral, and negative.Source: §3.2 · p. 5
• CACTUS outperforms Psych8k and SmileChat across all evaluated human-rated metrics, while its coherence difference from SmileChat is not significant.The comparisons assess helpfulness, coherence, empathy, and guidance.Source: §3.2 · p. 5, Figure 6 · p. 5

4.3 Results

• CACTUS training improves general counseling and CBT-specific skills over Psych8k, SmileChat, and CBT-LLM, supporting its quality as a realistic counseling dataset.Psych8k outperforms SmileChat despite using a smaller dataset; the authors attribute this to Psych8k’s real counseling conversations.Source: Table 2 · p. 6, §4.3 · p. 7

4.1 COUNSELINGEVAL: A Psychological Counseling Evaluation Framework

• COUNSELINGEVAL assesses counselor skills with CTRS-derived general and CBT-specific criteria and evaluates client emotional change with modified PANAS measures.The framework uses G-Eval to score criteria on a Likert scale.Source: §4.1 · pp. 6–7

4.2 Experiments Setup

• The experiments use GPT-3.5-Turbo for AI clients and GPT-4o for multi-turn G-Eval; CBT-LLM is compared by inference because its training dataset is undisclosed.Source: §4.2 · p. 7

4.3 Results

• Planning with CBT techniques significantly outperforms planning without CBT or not planning, particularly on Strategy.Planning without CBT performs slightly better than not planning; the authors link the Strategy gap to counselors’ use of evidence-based strategies for client change.Source: §4.3 · pp. 7–8
• CAMEL strongly improves positive emotions but has limitations in reducing negative emotions, which the authors attribute to exploring rather than directly changing thought patterns.Source: §4.3 · p. 8
• CAMEL-LLAMA3’s average PANAS scores—28.4 positive and 22.5 negative—are presented as close to human-subject means of 33.3 and 17.4, respectively.The authors interpret these similarities as supporting PANAS use with AI clients.Source: §4.3 · p. 8
• Filtering significantly improves counseling skills by removing erroneous CBT technique selections.The ablation is shown in Figure 8.Source: Figure 8 · p. 7, §4.3 · p. 8

4.4 Expert Evaluation

• GPT-4o correlates more highly with experts than non-expert evaluators; psychologists also find CAMEL more exploratory and empathetic, but note superficial questions and limited personalization.Psychologists judged qualitative comparison more appropriate because quantitative scores were similar despite quality differences.Source: §4.4 · p. 8, Table 5 · p. 8

5.1 Cognitive Behavioral Therapy

• CBT guides clients to recognize unhelpful thoughts and challenge distortions, gradually reconstructing more positive thoughts and beliefs; CACTUS simulates CBT for clients with depression and anxiety.Source: §5.1 · p. 9

5.2 Psychological Counseling

• Prior LLM counseling work is often single-turn or limited to two or three turns, while CACTUS applies CBT across realistic multi-turn dialogues.The paper frames this as addressing the challenge of maintaining therapeutic consistency over multiple turns.Source: §5.2 · p. 9

• CACTUS simulates counselor and client behaviors and varies client personas and counseling strategies to create diverse counseling dialogues.Source: §6 · p. 9
• CAMEL, trained on CACTUS, demonstrates high performance across all domains.Source: §6 · p. 9
• COUNSELINGEVAL simulates counselor–client dialogues and applies CTRS and PANAS criteria to support human-aligned evaluation.Source: §6 · p. 9

Limitations

• Real counseling commonly spans hour-long sessions and about 10 sessions, while this work uses shorter conversations and omits multi-session interactions.The authors identify the missing multi-session interactions as a limitation and recommend longer conversations and multi-session scenarios.Source: Limitations · p. 9
• Counselors may change strategies in response to clients, but the dataset adheres to the initial CBT technique selection and planning.Future work recommends more flexible strategy selection based on client responses.Source: Limitations · p. 9

Ethical Considerations

• The model is intended for mild depression and anxiety and should be used under professional supervision, not as a standalone tool or for more severe conditions.The authors note that AI could negatively affect people with mental health challenges and that unintended harm is possible.Source: Ethical Considerations · p. 10
• Scenarios avoid actual client data, using publicly available, expert-curated datasets based on crowdsourced information instead.The expert-provided information is generalized and does not represent a specific individual's psychological profile.Source: Ethical Considerations · p. 10
• Demographic bias may persist because ChatGPT-generated scenarios can reflect societal biases, and trained models could over- or underrepresent groups.The authors call for cautious deployment, ongoing monitoring, proactive adjustments, and transparent communication about potential biases.Source: Ethical Considerations · p. 10

A.1 The Types of Patterns

• Pattern definitions follow Sharma et al. (2023), while examples come from PatternReframe (Maddela et al., 2023).Source: §A.1 · p. 12

A.2.1 CBT Technique Selection

• The dataset uses 12 of Beck's 20 CBT techniques, selected with psychological experts; exclusions reflect overlap, low real-world use, or poor fit for remote counseling.Guided discovery and Socratic questioning were excluded as broad concepts used across techniques; scaling questions and thought experiments were considered less frequent, while four others were considered less suitable for remote counseling.Source: §A.2.1 · p. 12

A.2.2 Types of CBT Technique

• Table 7 provides explanations of the 12 selected CBT techniques.Source: §A.2.2 · p. 12

B.1 Limitations in selecting CBT Technique

• Three psychological experts selected up to three suitable techniques per thought-pattern pair, while LLMs were asked to select two.The experts' selections formed the gold labels when at least two experts agreed.Source: §B.1 · p. 12
• GPT-4o showed less bias than GPT-3.5 in CBT technique selection, but still demonstrated significant bias.The comparison used 150 client thought-pattern pairs; experts' gold labels included techniques selected by at least two of three experts, averaging 1.6 techniques per pair.Source: §B.1 · pp. 12–13

B.2 Limitations of LLM as an AI Client and an AI Counselor

• The authors also empirically investigate LLM behavior as simulated counselors and clients using prompts proposed by Na (2024).The prompts used for this investigation are shown in Figure 10.Source: §B.2 · p. 13

C.1 Implementations of Human Evaluation

• Three annotators assessed 100 samples per evaluation using win/lose comparisons and detailed rubrics, with $0.30 compensation per sample.The evaluations were conducted through Amazon Mechanical Turk.Source: §C.1 · p. 13

C.2 Human Evaluation Criteria

• The criteria cover helpfulness, coherence, empathy, and guidance: suitability of suggestions, session flow, responsiveness to client feelings, and practicality of advice.Source: §C.2 · p. 13
• Table 7 presents explanations of CBT techniques.Source: Table 7 · p. 14
• ChatGPT struggles to select appropriate CBT techniques compared with psychological experts.Figure 9 compares the distribution of techniques selected by ChatGPT and experts.Source: Figure 9 · p. 14

D.1 Method

• COUNSELINGEVAL uses 150 client profiles with three attitudes each, producing 450 interactive counseling instances evaluated with CTRS and PANAS.For CBT-planning models, technique selection and planning precede the first counselor utterance.Source: §D.1 · pp. 14–15

D.3 Positive and Negative Affect Schedule (PANAS)

• PANAS measures clients’ positive and negative emotions before and after counseling using ratings across 20 emotion items.The 10 positive and 10 negative items are rated on a 1-to-5 scale.Source: §D.3 · p. 15

E.1 Client Simulation

• Client simulation uses intake information and assigns positive, neutral, or negative attitudes to make client behavior more concrete.The intake includes presenting problems and past history.Source: §E.1 · pp. 15–16

E.2 Counselor Simulation

• Counselor simulation selects three CBT techniques for each thought-reframing case and plans the session around the client’s intake and selected techniques.The process is designed to reflect CBT’s goal-oriented, structured approach.Source: §E.2 · p. 16

E.3 Dialogue Generation

• Human evaluation found script-mode dialogues more natural and better constructed than two-agent dialogues, so the authors used script mode for generation.The comparison also assessed naturalness, alongside the criteria used for dataset quality.Source: §E.3 · p. 16

D.2 Cognitive Therapy Rating Scale (CTRS)

• The adapted CTRS averages six selected general and CBT-specific counseling criteria, omitting criteria deemed unsuitable for text-based sessions or GPT-based evaluation.Dialogues are scored from 0 to 6 with GPT-3.5 and retained when the six-criterion average is at least 5.0.Source: §D.2 · p. 15, §E.4 · p. 17

E.4 Filtering

• Filtering focuses CBT dialogue generation on anxiety or depression, excludes cases such as delusions or auditory hallucinations, and retains 31,577 dialogues after length, format, and CTRS-score filters.Basic filtering keeps 35,252 dialogues; subsequent CTRS filtering retains dialogues averaging at least 5.0 across six criteria.Source: §E.4 · pp. 16–17

G.2 Main Results

• CAMEL-LLAMA3 outperforms SmileChat-LLAMA3 on all COUNSELINGEVAL axes in a comparison using Chinese-translated CAMEL results.The authors note that translation may degrade response quality.Source: Figure 15 · p. 18, §G.2 · p. 18

G.3 Expert Evaluation

• CAMEL-LLAMA3 had a 50% win rate against Psych8k-LLAMA3, yet experts judged that neither model fully met professional counseling standards.The authors caution that quantitative evaluation alone is insufficient and underscore qualitative evaluation.Source: §G.3 · p. 18

H.2 Effectiveness of planning with CBT

• The appendix presents a case comparing ChatGPT counseling dialogues with and without CBT planning, alongside prior ablation results.The supplied passage identifies the comparison but does not state its outcome.Source: §H.2 · p. 18, Figure 23 · p. 29

H.3 Psych8k vs. CACTUS

• Psych8k-LLAMA3 is described as overwhelming clients with information, while CAMEL-LLAMA3 gradually draws out their concerns and fosters a more supportive interaction.This contrast is illustrated with counseling dialogue examples.Source: §H.3 · p. 18, Figure 24 · p. 30