Source-linked AI summary
Comparing Human Role-Players and LLM-Simulated Clients in Online Counselling Training: An Analysis of Counselling Patterns
Eric Rudolph, Philipp Steigerwald, Jens Albrecht
TL;DR
This study asks whether LLM-simulated counselling clients reproduce realistic conversational dynamics and elicit the same counsellor communication patterns as human clients. It compares educational Human–Human and Human–LLM role-plays using a counselling communication framework. LLM clients disclose more immediately and structurally, while human clients engage in more elaborative problem clarification, with different counsellor responses across conditions.
Problem
Human-peer role-plays can face scheduling and consistency constraints, while whether LLM clients produce authentic conversational dynamics and elicit the same counsellor communication patterns remains an open question.
Source: §1 · p. 1Method
Results
Human clients used Analysis & Clarification of problems in 59.07% of cases versus 45.47% for LLM-simulated clients, while LLM clients disclosed more immediately and structurally.
Source: §5.1 · p. 3, §7 · p. 6Takeaways & Limitations
LLM-based role-plays may augment counsellor training, while virtual-client simulations need further refinement to narrow differences from human clients.
Source: §1 · p. 1, §7 · p. 6The findings concern simulated role-plays rather than real-world counselling sessions.
Source: §6 · p. 5
Abstract
from arXiv · showhide
This study investigates the capabilities of Large Language Models to simulate counselling clients in educational role-plays in comparison to human role-players. Initially, we recorded role-playing sessions, where novice counsellors interacted with human peers acting as clients, followed by role-plays between humans and clients simulated by Mistrals Mixtral 8x7b using 4-bit quantization. These interactions were analysed with a counselling communication pattern system at sentence level. We investigated two key questions: (1) to what extent LLM-generated responses replicate authentic conversational dynamics and (2) whether counsellors' communication behaviour differs when interacting with human versus LLM-simulated clients. The findings highlight both similarities and differences in the application of counselling patterns across scenarios, showing the potential of LLM-based role-playing exercises to enhance counselling competencies and to identify areas for further refinement in virtual client simulations.
1. INTRODUCTION
Online counselling role-plays build communication skills, but human-peer sessions face logistical and consistency constraints. This study examines whether LLM clients reproduce conversational dynamics and elicit similar counsellor communication patterns.
1. INTRODUCTION
- Role-playing is central to counsellor education because it gives trainees controlled opportunities to explore, reflect on, and refine communication skills.Human-peer sessions can be difficult to schedule and may vary in complexity, availability, engagement, and character consistency.Source: §1 · p. 1
- LLM-driven virtual clients may offer standardized, flexible, round-the-clock practice, but the authenticity of the learning experience remains uncertain.The study asks whether their conversational dynamics are realistic and whether counsellors use the same communication patterns with human and LLM clients.Source: §1 · p. 1
- The study compares human–human and human–LLM text-based role-plays using a counselling communication framework to analyse patterns and their temporal evolution.The authors report similarities and differences in client disclosure and counsellor responses and session structure.Source: §1 · p. 1
2. RELATED WORK
Prior work has explored LLM role-play feasibility and potential, while leaving a quantitative comparison of human–human and human–LLM counselling patterns unaddressed. This study combines a virtual-client approach with GeCCo analysis to examine that gap.
2. RELATED WORK
- A prior educational study found LLM role-plays could be structured and engaging, but differed from human interactions in spontaneity and contextual adaptation.A GPT-4 counselling study found conversational-pattern replication potential alongside consistency and emotional-authenticity challenges [6].Source: §2 · pp. 1–2
- Prior work has explored LLMs for structured role-play, counselling-session generation, virtual-client systems, psychotherapy training, and counselling education, while noting refinement needs and risks.Reported concerns include response consistency, emotional authenticity, bias, ethics, and misinformation; reviews also note the importance of integration with established educational frameworks.Source: §2 · p. 2
- The authors identify a gap: prior work had not quantitatively contrasted human and AI-client interaction patterns in training using a standardized category system such as GeCCo.They position their work as combining a virtual-client platform based on VirCo with GeCCo communication analysis.Source: §2 · p. 2
3. DATASET
The dataset comprises educational Human–Human and Human–LLM text counselling role-plays, with shared case studies and a single LLM selected for comparability. Table 1 is identified as comparing messages across the two conditions.
3. DATASET
- The Human–Human dataset contains 64 chat-based counselling sessions from a social-work degree program, with students acting as clients based on selected case studies.The sessions focused on online counselling fundamentals.Source: §3 · p. 2
- The Human–LLM dataset contains 74 completed conversations from two student groups, using Mixtral 8x7B with 4-bit quantization in a learning platform based on VirCo.The same case studies were incorporated into the model prompt, and one model was used to maintain comparability and minimize sentence-count confounds.Source: §3 · p. 2
- Table 1 is titled as a comparison of Human–Human and Human–LLM counselling messages.The caption identifies the conditions compared but does not specify message-level results.Source: Table 1 · p. 2
4. METHOD
The analysis uses GeCCo to classify counsellor and client communications, with the study operating at layer 3. Pattern frequencies, temporal changes, and embedding-based semantic clusters provide complementary views of the conversations.
4. METHOD
- GeCCo is a hierarchical framework for classifying online counselling communications, with up to five abstraction layers and 40 counsellor and 28 client categories at layer five.It covers broad dimensions and finer communicative nuances.Source: §4 · p. 2
- The study used layer 3 of GeCCo, selected for alignment with German-language conversations, and applied a GBERT Large classifier fine-tuned with GPT-4 synthetic data.GeCCo synthesizes established categorisation frameworks, including Motivational Interviewing [1].Source: §4 · p. 2
- The category scheme distinguishes client communications such as problem clarification, problem-solving, motivation, and resources from counsellor behaviours such as moderation and factual or emotional reflection.It also includes goal-setting and opening and closing formalities for clients and counsellors.Source: §4 · pp. 2–3
- Pattern-distribution analysis counts GeCCo categories across conditions, while temporal analysis divides conversations into deciles to track changes such as factual reflection and problem-solving.These analyses examine both frequency and how patterns vary across session stages.Source: §4 · p. 3
- Semantic analysis embeds messages with the GBERT classifier and visualizes them using t-SNE to reveal category-related clusters.Examples include Analysis & Clarification of Problems and Formalities.Source: §4 · p. 3
5. RESULTS
Across Human-Human and Human-LLM counselling role-plays, client disclosure and counsellor responses differed, and the interaction patterns evolved differently over time. Semantic analyses and educational implications further highlight structured features of LLM interactions and the value of balancing them with human role-play.
5.1 Pattern Distribution Analysis
- Human clients used Analysis & Clarification of Problems more often than LLM clients (59.07% vs. 45.47%), while LLM clients more often used Help, Problem-Solving (17.63% vs. 10.67%).LLM clients tended to disclose prompted problems more readily, enabling a faster transition to problem-solving.Source: §5.1 · p. 3, Figure 1 · p. 4
- Counsellors used factual reflection more in Human-LLM sessions (36.37% vs. 26.07%), while moderation was more frequent in Human-Human sessions (23.20% vs. 12.17%).The authors suggest factual reflection may compensate for direct disclosure, while human-client sessions involve more active discussion-leading.Source: §5.1 · p. 3, Figure 1 · p. 4
5.2 Temporal Analysis
- Factual reflection peaked early in Human-LLM sessions but remained more stable in Human-Human sessions; Help, Problem-Solving and Moderation rose later, especially with human clients.Generating Motivation remained relatively low but increased slightly near the end.Source: §5.2 · pp. 3–4, Figure 2 · p. 4
- Human clients elaborated problems more intensely early, while LLM clients disclosed them more immediately and shifted toward problem-solving and resource activation in the second half.LLM-client conversations also showed higher opening formality that quickly decreased and a sharp rise in closing formality in the last decile.Source: §5.2 · p. 4, Figure 3 · p. 4
5.3 Semantic Analysis
- In the client-sentence projection, problem-clarification statements were dispersed, while General Inquiries appeared more often in Human-Human interactions and agreements formed a distinct cluster.LLM-generated agreements tended to occur within longer sentences; formalities at the beginning and end formed concentrated regions.Source: §5.3 · pp. 4–5, Figure 4 · p. 5
- Counsellor-sentence projections placed factual and emotional reflection categories near each other; moderation statements clustered and appeared more often in Human-Human conversations.The authors note that emojis were unavailable on the learning platform, which may have influenced moderation-related messages.Source: §5.3 · p. 5, Figure 5 · p. 5
5.4 Educational Implications
- LLM role-plays offer structured, consistent practice that can support early training, but may limit exposure to spontaneous, emotionally nuanced interactions.The paper recommends balancing LLM role-plays with traditional human interactions to support both technical and interpersonal competencies.Source: §5.4 · p. 5
6. LIMITATIONS
The study’s findings are bounded by automated classification, simulated role-play settings, coarse GeCCo categorisation, and use of a single LLM. The single-model choice aids comparability but leaves open whether other LLMs would show different patterns.
6. LIMITATIONS
- Automated GBERT Large classifiers may miss subtle nuances in conversational dynamics.Source: §6 · p. 5
- Because the sessions were role-plays rather than real counselling, the findings describe simulated environments.Source: §6 · p. 5
- Analysis focused mainly on GeCCo level 3, while finer sub-categories could reveal further nuances.Source: §6 · p. 5
- Only Mixtral 8x7B with 4-bit quantisation was tested, so alternative LLMs might exhibit different conversational patterns.Using one model offers better comparability across sessions.Source: §6 · p. 6
7. CONCLUSION AND FUTURE WORK
The conclusion reports contrasting client-disclosure and counsellor-response patterns in human–human and human–LLM role-plays, alongside distinct temporal and semantic dynamics. Future work proposes improving simulation authenticity and strengthening evaluation and generalisability.
7. CONCLUSION AND FUTURE WORK
- LLM-simulated clients tend to disclose concerns more immediately and structurally, prompting more factual reflection; human clients elaborate problems, prompting more counsellor moderation.Source: §7 · p. 6
- Temporal and semantic analyses reveal distinct conversational dynamics and clustering patterns across the two interaction types.Source: §7 · p. 6
- Future work could fine-tune LLMs with representative conversational data or use Direct Preference Optimization to improve naturalness and emotional authenticity.Source: §7 · p. 6
- Manual annotation, a counselling-communications gold standard, finer utterance categories, and diverse linguistic and cultural contexts are proposed to strengthen evaluation and generalisability.Source: §7 · p. 6