Source-linked AI summary
CAIA in Practice: Field Evaluation of an AI-Assisted Support System for Text-Based Online Counselling
Philipp Steigerwald, Nico Bienlein, Jennifer Burghardt, Mara Stieler, Robert Lehmann, Jens Albrecht
TL;DR
Rising demand makes accessible mental health support difficult to deliver, while evidence about counsellors’ real-world adoption of AI assistance remains limited. This paper presents CAIA, a co-designed tool suite evaluated in field-based email counselling, finding that interpretive functions were used most extensively and that autonomy and accuracy shaped sustained acceptance.
Problem
Evidence remains limited on how professional counsellors adopt multifunctional AI assistance in asynchronous email counselling, despite growing demand for accessible mental health support.
Method
CAIA was co-designed with practitioners and evaluated through a four-week field study involving 34 professional counsellors using realistic counselling cases enacted by trained students.
Results
Interpretive functionalities were used most extensively, while information extraction showed potential value during case handovers or extended response times.
Takeaways & Limitations
Professional autonomy and accurate generated content are central to sustained acceptance of AI assistance in counselling practice.
Takeaways & Limitations
The study assessed professional acceptance rather than counselling effectiveness, and its four-week evaluation with trained students outside habitual platforms may not capture stable adoption patterns.
Abstract
from arXiv · showhide
Rising global demand for mental health support creates significant service delivery challenges, with asynchronous email counselling serving as a crucial low-threshold channel for accessing care. This paper presents CAIA, a co-designed AI-based tool suite that demonstrates responsible AI integration into counselling practice through seven LLM-driven functions enhanced by retrieval-augmented generation. A field evaluation involved 34 professional counsellors conducting authentic sessions with trained student counsellees (36 threads, 321 messages, 1,257 AI outputs). User behaviour analysis confirms substantial adoption, revealing that professional autonomy and information accuracy are decisive for sustained acceptance, with counsellors particularly valuing interpretive functionalities that provide new perspectives and stimulate professional reflection.
I. INTRODUCTION
CAIA addresses limited evidence about how counsellors adopt LLM-based assistance in asynchronous email counselling. It was co-designed as a supplementary tool suite intended to preserve counsellor autonomy while supporting counselling practice.
- Asynchronous email counselling provides low-threshold psychosocial support for people facing barriers related to stigma, cost, or geography.
- Evidence remains limited on whether and how counsellors adopt functions such as summarisation, information processing, and visualisation in practice.
- CAIA augments counsellors’ skills through supplementary AI-driven analytical functionalities operating alongside traditional email correspondence.
- Co-design with professional counsellors, social science researchers, and software developers aligned CAIA with professional practice requirements.
- CAIA was evaluated in the field with 34 practising counsellors across three cohorts using realistic case vignettes enacted by trained students.
- The study evaluates whether co-designed, non-intrusive AI functionalities preserve counsellor autonomy and which functions achieve professional acceptance.
II. RELATED WORK
Prior systems show promise, but routine adoption of multifunctional AI assistance by professional counsellors remains underexplored. CAIA combines co-designed functionality with tools for extracting information and generating interpretive professional insights.
- Prior work and gap: Earlier systems include empathic rewrites, reinforcement-learning support, motivational-interviewing guidance, and educational training tools.
- Prior work and gap: Field evidence on user adoption remains limited, especially for multifunctional AI assistance in asynchronous email counselling.
- System design: CAIA’s co-design process used usability testing and stakeholder workshops to refine its architecture, AI functionalities, and conventional tools.
- Information extraction: The suite includes generated subject lines, counselling briefs, histories, current-situation overviews, and social-network graphs for extracting and organising case information.
- Interpretive functionalities: Hypotheses and Methodological Suggestions interpret contextual patterns and offer professional insights about case dynamics and intervention strategies.
- System operation: Functionality outputs are regenerated at message-specific intervals, while the Generated Subject Line remains static after the initial counsellee message.
- Conventional tools: Conventional annotations and notes complement the AI functionalities with text review, navigation, observation, planning, and feedback capture.
B. Knowledge Integration via RAG
CAIA uses retrieval-augmented generation to align outputs with domain-specific counselling knowledge. Its two-tier knowledge base separates general principles from functionality-specific theoretical sources.
- The RAG pipeline integrates domain-specific counselling knowledge through a general corpus and functionality-specific subsets.
- During computation, CAIA retrieves passages from both tiers and combines established counselling principles with case-relevant details.
- The knowledge base can be dynamically expanded or modified as counselling approaches and theoretical frameworks evolve.
C. Architecture
CAIA uses a modular, distributed architecture for counselling interfaces, API services, and asynchronous AI processing. Its deployment supports independent component scaling, multiple institutes, and varied LLM environments.
- The system separates the web frontend, REST API, and asynchronous computation tier into three architectural layers.
- Docker Compose, Flask, Redis, Celery, PostgreSQL, ChromaDB, and Nginx support orchestration, APIs, background jobs, storage, retrieval, and traffic routing.
- CAIA operates either as a stand-alone counselling platform or as an API service integrated with external email counselling systems.
- A multi-tenant deployment lets counselling institutes customise functionalities, LLM endpoints, prompts, and knowledge bases for different specialisations.
- Message-level sequential processing preserves ordering within threads while functionality generation runs concurrently across threads.
D. Optional User Interface
CAIA uses a non-intrusive split-screen interface that keeps the counselling conversation primary while offering optional, editable AI assistance. Counsellors can inspect references, revise generated content, and preserve earlier versions as cases evolve.
- The split-screen layout places the email conversation on the left and collapsible AI functionality panels on the right.
- Counsellors choose which functionality to open and when, with only one panel expanded to maintain focus and professional autonomy.
- Functionality histories keep newer generated content visible above earlier versions, supporting transparency as cases develop.
- Integrated toolbars let counsellors format and refine AI-generated content, including contextual word replacement and sentence reformulation.
- Reference symbols expose source passages on hover and can navigate directly to the original counsellee message.
IV. FIELD TEST
CAIA was field-tested in professional online counselling over four weeks, using naturalistic sessions and domain-specific configurations. The evaluation included practising counsellors from multiple counselling cohorts and self-hosted AI generation for data sovereignty.
- CAIA was deployed to examine professional counsellors’ adoption of AI assistance during counselling interactions in practice settings.
- 34 professional counsellors participated in the field evaluation across three groups and online counselling domains.
- The naturalistic study ran for four weeks, with counsellors conducting sessions as usual while exploring AI tools and documenting feedback.
- Each institute used domain-specific prompts and retrieval documents, while AI generation ran on self-hosted German servers for data sovereignty.
C. Data Acquisition
The study collected authentic counselling interactions and detailed behavioural logs while allowing counsellors to use or ignore AI tools according to professional judgment. Usage analysis found strong preferences for analytical functions and minimal engagement with Annotations.
- 36 counselling threads produced 321 messages, comprising 180 counsellee messages and 141 counsellor messages.
- Behavioural tracking recorded panel openings, visible duration, navigation paths, and panels left open during reply composition.
- Counsellors retained autonomy to use or ignore AI tools, with informed consent, anonymisation, and secure EU-compliant data handling.
- 1,257 AI outputs were generated, while 31 counsellors contributed browser-tracked data covering 1,128 panel openings.
- Methodological Suggestions led usage with 190 openings (16.8%), followed by Hypotheses with 178 openings (15.8%).
- Annotations received only 36 panel openings (3.2%), indicating minimal engagement relative to the other functionalities.
- Methodological Suggestions and Hypotheses each averaged more than 20% of thread open-time.
- As counselling progressed, Counselling History increased steadily, while documentation functions declined and Methodological Suggestions increased again in complex cases.
B. Functionality Transition Analysis
Counsellors generally navigated CAIA panels in interface order, but transition analysis also identified significant feedback loops and shortcuts linking analytical functions. These patterns indicate purposeful movement between evidence, interpretation, guidance, and notes.
- 741 recorded panel transitions largely followed the order in which functionalities appeared in the interface.
- The heatmap encodes transition counts by cell and marks significantly over-represented transitions with adjusted-p-value asterisks.
- The transition analysis excluded three central diagonals before testing the remaining 42 cells for purposeful shortcuts.
- Two bidirectional loops connected Current Situation Overview with Hypotheses and Methodological Suggestions with Notes.
- Current Situation Overview ↔ Hypotheses transitions numbered 18*** and 11***, while Methodological Suggestions ↔ Notes numbered 26*** and 16**.
- Counsellors also used shortcuts from Counselling Brief to Current Situation Overview and from Counselling History to Social Network Graph.
C. Consultation During Response Composition
Counsellors actively consulted AI functionalities while composing replies, with interpretive tools used more often than information-extraction tools. Feedback highlighted practical value but also accuracy problems that undermined confidence.
- 52.94% of typing sessions kept at least one functionality panel open while counsellors composed replies.This indicates active consultation during response composition.
- 25.67% of typing sessions involved interpretive functionalities, compared with 17.64% involving information-extraction functionalities.Notes accounted for 9.63% of typing sessions, though this may reflect either active reference or leaving the panel open.
- Counsellors valued compact summaries for rapid case orientation but reported hallucinations and incomplete extractions that undermined professional confidence.Feedback described summaries as “compact and helpful” and “well put together,” while identifying unsupported and missing information.
- The Social Network Graph received 14 negative and 1 positive feedback segments, with role-assignment and duplicate-institution errors overshadowing its visualisation concept.Counsellors nevertheless appreciated the possibility of gaining a quick outside view of the system.
- Interpretive tools elicited mixed reactions: counsellors found hypotheses stimulating but sometimes superficial, while methodological suggestions generally provided useful confirmation and fresh impulses.The feedback analysis covered 58 counsellor segments across six functionalities.
E. Overall Adoption Patterns
Adoption was selective rather than uniform across CAIA’s functionalities. Interpretive tools achieved the strongest integration into counselling work, whereas extraction tools mainly supported orientation and accuracy problems constrained sustained use.
- Information extraction supported rapid case orientation but saw brief engagement, limited response-composition use, and accuracy problems including hallucinations and incomplete information.The findings suggest greatest potential during case handovers or when counsellors return to cases after extended periods.
- The Social Network Graph showed declining usage despite initial openings, with frequent transitions to Notes associated with documenting implementation errors.Counsellors valued its systemic-perspective concept but reported systematic technical problems.
- Hypotheses ranked second in frequency and prompted extended consultation during response composition, despite mixed evaluations of their professional grounding.Bidirectional transitions with Current Situation Overview indicated hypothesis validation and reflection.
- Methodological Suggestions led across opening frequency, session duration, feedback transitions, and response-composition consultation, with predominantly positive feedback.Counsellors valued fresh impulses without imposed solutions, although some suggestions conflicted with professional standards.
- Interpretive functionalities achieved the highest engagement, while information-extraction tools served primarily orientation purposes.The overall pattern reflects selective integration based on professional value.
VI. CONCLUSION
CAIA’s field evaluation demonstrates a non-intrusive approach to AI-assisted email counselling that preserves counsellor autonomy and supports professional adoption. Interpretive functions were used most extensively, but errors quickly undermined trust, and the study did not establish counselling effectiveness.
- CAIA contributes a co-designed, non-intrusive approach that preserves counsellor autonomy and supports professional acceptance of AI functionalities.The field evaluation identifies which AI-generated content may provide meaningful value in counselling practice.
- Interpretive functionalities were adopted and used most extensively, while information-extraction tools may be especially useful for case handovers or long response intervals.These findings concern acceptance and adoption rather than counselling outcomes.
- Errors in generated content quickly undermined trust, making accurate processing and robust language models important for sustained adoption.The conclusion links content errors to weakened trust in AI assistance.
- The study examined professional acceptance rather than counselling effectiveness, using a four-week evaluation outside habitual platforms with trained students rather than genuine counsellees.AI-generated errors may also have distorted observed adoption patterns.
- Future research should test counselling outcomes through controlled studies, long-term deployments in established systems, genuine counsellees, and improved technical implementations.
- The study provides a foundation for assessing whether AI assistance improves mental health service delivery, without itself establishing that effectiveness.