PaperlayerView exact revision on arXiv

Source-linked AI summary

DeepSAGE: Stage-Aware Reinforcement Learning for Structured CBT Counseling Dialogue

Qi Zhang, Heajun An, Prakriti Dumaru, Sang Won Lee, Lifu Huang, Pamela J. Wisniewski, Jin-Hee Cho

arXiv:2608.22615v1cs.AI

TL;DR

Counseling agents can generate fluent responses but often lack the structured, goal-directed progression needed for coherent therapeutic sessions. DeepSAGE combines eleven-stage CBT modeling with DRL-based intention selection and, in simulated evaluations, improves engagement, openness, stage-goal completion, and dialogue efficiency, while remaining unvalidated clinically.

  • Problem

    Existing counseling agents and LLM-CBT systems lack reliable, structured, session-level therapeutic progression despite CBT's structured methodology.

  • Method

    DeepSAGE models the first CBT session as eleven stages with explicit objectives and success criteria, while a DRL policy selects therapeutic intentions guiding LLM response generation.

  • Results

    DeepSAGE achieved the strongest balance of engagement, openness, stage-goal completion, and dialogue efficiency among the evaluated stage-structured systems in simulated-client evaluations.

  • Takeaways & Limitations

    Stage-structured modeling combined with learned strategy selection is a promising approach for structured counseling dialogue.

  • Takeaways & Limitations

    The evaluation primarily uses simulated clients and model-based metrics, and the prototype lacks dedicated escalation or crisis-referral responses, so it is not clinically validated or ready for real-user deployment.

Abstract

from arXiv · show

Large Language Model (LLM)-based counseling agents can generate fluent and supportive responses, but they often lack the structured, goal-directed progression required to conduct a coherent therapeutic session. We present DeepSAGE (Strategic AI Guidance Engine), a hybrid LLM--Deep Reinforcement Learning (DRL) framework for stage-aware counseling dialogue grounded in the first session of Cognitive Behavioral Therapy (CBT). DeepSAGE represents the session as eleven stages with explicit therapeutic objectives, with an external controller determines stage completion and the DRL model selects therapeutic intentions that guide LLM response generation. We evaluate DeepSAGE against six retrieval-, prompting-, stage-, and policy-based alternatives. DeepSAGE elicits higher simulated client engagement and openness and achieves the strongest balance of stage-goal completion and dialogue efficiency among stage-structured systems. Domain expert review further indicates that the generated conversations exhibit broadly plausible emotional trajectories and recognizable CBT processes. Because the evaluation relies primarily on simulated clients and model-based metrics, these findings demonstrate comparative dialogue-control improvements rather than clinical effectiveness. These results suggest that combining stage-structured dialogue with learned strategy selection is a promising approach for AI counseling, though clinical effectiveness, safety, and real-world utility require further human evaluation.

Introduction

AI counseling systems often provide fluent or conversational responses without therapeutically grounded, session-level progression. DeepSAGE addresses this gap with stage-structured CBT guidance and learned therapeutic-intention selection for LLM response generation.

  • Motivation: Access barriers and limited infrastructure motivate scalable, accessible, safe, therapeutically grounded AI support.The cited barriers include cost, confidentiality concerns, and strained care infrastructure.
  • Limitations of Existing Systems: Existing counselor chatbots often respond passively or rely on shallow scripts, limiting their ability to guide users through psychologically grounded processes.Earlier systems used rule-based or shallow NLP pipelines, while newer platforms remained largely client-led.
  • Limitations of Existing Systems: LLM-based CBT systems capture relevant techniques but generally focus on individual turns or isolated modules rather than complete sessions.The paper identifies reliable, structured, session-level therapeutic guidance as an unresolved need.
  • DeepSAGE: DeepSAGE combines stage-structured counseling with a DRL policy that selects therapeutic intentions to guide flexible LLM response generation.The framework is evaluated with LLM-simulated clients in a controlled, reproducible environment, with domain experts verifying simulated-conversation realism.
  • DeepSAGE: The framework models an initial CBT session as eleven stages with explicit therapeutic objectives and automated stage detection for dialogue management.Its stated contributions also include a therapeutically grounded action space and reward function targeting engagement, relevance, progression, and completion.

Proposed Approach: DeepSAGE

DeepSAGE models the first CBT session as eleven stages with explicit success criteria, while a DRL policy selects therapeutic intentions to guide LLM responses. In simulated evaluations, it combined stronger engagement and openness with more efficient stage completion than comparison systems.

  • Stage-structured counseling: DeepSAGE represents the first CBT session as eleven stages, each with a therapeutic role and goal-success criterion governing progression.The system evaluates client replies against stage objectives and uses the resulting score to determine transitions.
  • Stage-structured counseling: Stage transitions occur when the client reply satisfies the current objective or when the stage-specific maximum turn limit is reached.The turn limit depends on stage complexity and provides a forced-transition condition.
  • DRL-guided strategy selection: A modular reward function balances response relevance, client engagement, self-disclosure, and stage progress while penalizing unnecessarily long stages.The coefficient α balances response quality and self-disclosure, while λ penalizes extended stages.
  • Evaluation results: DeepSAGE achieved the highest user utterance length and self-disclosure count in most simulated anxiety-disorder and major-depressive-disorder conditions.Relative to the strongest conventional baselines, AD UL improved approximately 20.0% and AD SDC 33.1%; MDD UL and SDC improved approximately 59.9% and 80.8%.
  • Evaluation results: DeepSAGE produced a more balanced MDD outcome than the Naïve LLM by combining strong distress reduction with the highest engagement.The Naïve LLM achieved higher measured MDD distress reduction but elicited shorter, less self-disclosing responses.
  • Evaluation results: DeepSAGE achieved the strongest balance of stage-goal completion and dialogue efficiency among stage-structured systems, outperforming random intention selection and prompt-based alternatives.For simulated AD clients, SSR improved approximately 10.9% and counselor utterances fell 9.3% versus DeepSAGE_R; corresponding MDD changes were 8.6% and 7.0%.

Supplementary Material for “DeepSAGE: Stage-Aware Reinforcement Learning

The supplementary material positions DeepSAGE as a stage-structured CBT counseling framework that combines therapeutic grounding, adaptive interaction, and explicit dialogue progression. Its method separates strategic intention selection from LLM language generation and adds ethical guardrails, reproducible prompting, and stage-transition definitions.

  • Related Work: DeepSAGE unifies therapeutic grounding, adaptive interaction, and explicit dialogue progression for coherent session-level counseling aligned with an initial CBT workflow.The framework targets the full counseling process rather than isolated conversational behaviors.
  • Stage-Structured Counseling: The framework models the initial CBT session as eleven stages with explicit therapeutic objectives and automated stage detection for dialogue management.Each stage includes a role, an objective formulation where appropriate, and a goal-success criterion for advancing the dialogue.
  • Therapeutic Strategy Selection: DeepSAGE selects a clinically meaningful therapeutic intention before the LLM generates the counselor’s natural-language response.This separates strategic decision making from language realization and supports interpretability while preserving conversational flexibility.
  • Therapeutic Strategy Selection: The DRL action space uses seven CBT-oriented intentions selected from a broader counselor-intention taxonomy to balance therapeutic expressiveness with compact policy representation.The taxonomy is adapted from Hill and O’Grady (2001).
  • Prompted Response Generation: The counselor prompt conditions generation on the current CBT stage, stage goal, selected intention, therapeutic purpose, strategy, constraints, example realization, and recent dialogue.The generator is instructed to produce one concise, empathetic response that advances the current stage without diagnosing, prescribing medication, claiming clinical authority, or revealing the internal action label.
  • Safety and Reproducibility: Common ethical guardrails require empathetic, non-diagnostic, non-prescriptive, clinically safe responses, with a dedicated safety protocol for detected severe self-harm or other high-risk situations.The safety protocol bypasses the learned therapeutic intention when such situations are detected.

F Baseline Details

The baselines span retrieval, unconstrained and CBT prompting, full-protocol prompting, and stage-controlled systems, enabling comparisons of knowledge access, session structure, stage progression, and strategic action selection.

  • Retrieval-Augmented CBT FAQ Bot: RAB retrieves CBT psychoeducational passages from a curated knowledge base and responds without explicit therapeutic planning or stage-aware dialogue management.At each counselor turn, the client’s latest utterance is embedded and compared with document embeddings to retrieve the top three passages for prompting.
  • Naïve LLM-Bot: The Naïve LLM-Bot uses a general-purpose counseling prompt without CBT stages, external knowledge, stage transitions, or strategic policy control.It conditions responses on the eight most recent dialogue turns and limits responses to 2–4 conversational sentences.
  • LLM4CBT: LLM4CBT evaluates utterance-level CBT alignment by prompting the backbone model with only the latest client utterance rather than the full dialogue history.Its responses are constrained to 2–4 sentences and avoid explicit CBT terminology or therapeutic-technique references.
  • Full-Protocol Prompt LLM: The Full-Protocol Prompt LLM receives all eleven ordered stages and recent dialogue turns, but internally decides stage tracking and advancement without external completion scoring.Recorded trajectories are constrained to remain non-decreasing and advance by at most one stage, while sessions end at final feedback or 50 counselor turns.
  • Stage-Prompt LLM: The Stage-Prompt LLM shares DeepSAGE’s eleven-stage representation and goal-based external transitions but removes therapeutic-intention selection and PPO-based strategic control.This comparison isolates whether learned action selection adds value beyond stage-aware prompting and goal-based progression.

H Safety Stress Test Evaluation

The safety stress test probes DeepSAGE with diverse high-risk client inputs while retaining its original stage-conditioned policy-control mechanism. Across scenarios, the policy selects the same available intention, exposing a safety-related action-space limitation rather than establishing clinical safety.

  • Evaluation scope: The stress test examines DeepSAGE’s behavior on high-risk inputs, explicitly distinguishing interpretable edge-case behavior from evidence of clinical safety.The experiments use simulated LLM-based clients, and the stress test is not intended to establish clinical safety.
  • Safety scenarios: The scenarios cover self-harm, abuse disclosure, psychosis-like symptoms, acute medical or panic events, diagnosis or medication requests, and excessive dependency.These utterances are inserted into appropriate CBT stages for evaluation.
  • Evaluation procedure: The trained policy selects an intention from the same seven-action space used in the main experiments, and the LLM generates the response from that intention and dialogue stage.This preserves the original policy-control mechanism during safety testing.
  • Observed policy behavior: Across all evaluated scenarios, the policy consistently selects Identify Maladaptive Cognitions, the closest available action under its existing constraints.The action encourages elaboration of distress-related thoughts but is not an appropriate clinical response to every high-risk scenario.
  • Implications: The authors motivate adding crisis escalation, crisis-resource referral, refusal of diagnosis or medication advice, and dependency-related boundary setting as dedicated safety-oriented behaviors.These additions are presented as a way to improve practical applicability while preserving the structured CBT foundation.

K Sensitivity Analysis

The sensitivity analysis varies the stage-transition threshold and reward weight, evaluating how these settings affect goal success, counselor-turn efficiency, and forced transitions.

  • Parameters: Table 9 varies the stage-transition threshold τ and reward weight α across DeepSAGE configurations.The main experiments use the highlighted settings τ = 0.8 and α = 0.5.
  • Metrics: The analysis reports average goal-success score, average counselor utterance turns per session, and forced stage-transition rate for each setting.Forced transitions are triggered by the maximum turn limit rather than the stage-completion criterion.
  • Interpretation: Higher goal-success scores and lower forced stage-transition rates indicate better stage progression in the sensitivity analysis.The table also treats fewer counselor utterances as greater dialogue efficiency.

L.2 Long-Segment Dialogue Evaluation

The long-segment dialogue evaluation asks evaluators to judge generated counselor–simulated-client conversations under standardized instructions and common response scales, with optional qualitative comments.

  • Evaluation design: Each evaluator reviewed three counselor–simulated-client conversation excerpts under standardized evaluation conditions.The excerpts varied across survey groups, while the evaluation instructions were shared.
  • Instructions: Evaluators were instructed to assess dialogue quality rather than personal counseling-style preference and to distinguish improvable quality from clear harm.Non-harmful weaknesses were to be reflected in quality ratings rather than automatically judged unsafe.
  • Questionnaire: The questionnaire used six statements with a six-point scale ranging from Strongly disagree to Unable to judge.Evaluators could also provide optional open-ended comments about realistic or unrealistic aspects.

L.3 Emotional-Intensity Change Evaluation

The emotional-intensity evaluation asked experts to compare negative emotion across excerpts from the same session using observable language and a shared seven-point scale. It also assessed rating confidence and whether emotional-intensity drop was a suitable metric.

  • Evaluation procedure: Evaluators judged whether clients’ expressed negative emotion increased, decreased, or remained similar between earlier and later session excerpts.Judgments were based on observable indicators including urgency, hopelessness, distress, fear, sadness, anger, shame, and perceived loss of control.
  • Interpretation: Lower emotional intensity was not treated as inherently better therapy because stronger later disclosure may follow greater client trust.
  • Questionnaire design: The questionnaire used Q1–Q2 for emotional-intensity annotations and Q3–Q4 for rating confidence and metric suitability.
  • Questionnaire design: All questions used a seven-point scale ranging from Very low to Very high, and chatbot identity was concealed from evaluators.

M Session Completion and Statistical Significance

DeepSAGE showed its most consistent advantage in structured session success, achieving higher stage-goal completion with fewer turns than the other stage-structured systems. Session length was interpreted separately from engagement quality because shorter sessions coincided with higher SSR.

  • Statistical analysis: The analysis paired sessions by run index and, where applicable, simulated-client seed and configuration, then used two-sided Wilcoxon signed-rank tests with Holm correction.The evaluated measures included UL, SDC, EID, client-utterance count, and SSR where stage-indexed responses were available.
  • Session success: DeepSAGE’s most consistent advantage across session-level comparisons was structured session success.
  • Session success: DeepSAGE achieved significantly higher SSR than DeepSAGE_R, Stage-Prompt, and Full-Protocol for both AD and MDD clients.For AD, the SSR deltas were 0.0880, 0.0782, and 0.3956; for MDD, they were 0.0662, 0.0522, and 0.3663, respectively, with Holm-adjusted significance reported for each comparison.
  • Emotional-intensity results: For AD, DeepSAGE’s EID advantage was selective and not significant against DeepSAGE_R, Stage-Prompt, or Full-Protocol after multiple-comparison correction.
  • Efficiency and engagement: DeepSAGE produced fewer client utterances than DeepSAGE_R and Stage-Prompt for both client conditions, and fewer than Full-Protocol for AD.Because these shorter sessions also had significantly higher SSR, the reductions may indicate more efficient progression through structured CBT stages rather than reduced engagement.

N Stage-Wise Action Distribution

The DRL policy selected multiple therapeutic intentions with condition- and stage-dependent distributions rather than relying on one dominant action. Anxiety and depression profiles differed in their emphasis on Focus, Support, Identify Maladaptive Cognitions, Identify Feelings, and Normalize Experience.

  • Action space and distributions: The policy’s seven therapeutic intentions remained relatively diffuse across stages, with no single dominant intention overall.The action set included Support, Encourage Catharsis, Clarify, Focus, Identify Feelings, Identify Maladaptive Cognitions, and Normalize Experience.
  • Anxiety Disorder: For AD clients, Focus was most frequent in Obtain Update S4 (27.78%), Apply Cognitive Model S8 (18.33%), and Review Homework S10 (21.28%).Identify Maladaptive Cognitions was most frequent in Set Agenda S2 (23.53%), Discuss Diagnosis S5 (22.29%), and Educate About the Cognitive Model S7 (16.76%); Normalize Experience dominated Mood Check S3 (24.00%) and Elicit Feedback S11 (18.18%).
  • Major Depressive Disorder: For MDD clients, Support was most prominent in Greet S1 (22.50%) and Obtain Update S4 (25.93%), while Identify Maladaptive Cognitions led Mood Check S3 (24.44%).Identify Feelings peaked in Identify Problems and Goals S6 (20.31%), and Normalize Experience was most frequent in Educate About the Cognitive Model S7 (15.71%).
  • Condition differences: The anxiety policy emphasized Focus in update, cognitive-model application, and homework-review stages, whereas the MDD policy relied more on Support during rapport-building and information-gathering stages.The MDD policy also selected Identify Feelings more frequently in S6, while the anxiety policy more frequently selected Normalize Experience during mood checking.
  • Policy implementation: At each interaction step, PPO observed dialogue semantics, the current CBT stage, and normalized within-stage utterance count before selecting an intention.The selected intention was passed with stage and dialogue context to a counselor-generation prompt, while the LLM produced the surface response.
  • Stage progression: The dialogue advanced when the stage-completion score reached 0.8 or the maximum within-stage utterance limit was reached.The score combined semantic similarity between the stage goal and client response with an entailment component.

O.4 PPO Training

DeepSAGE was trained with PPO using separate actor and critic networks over a state representation of dialogue, CBT stage, and within-stage time. Actions were sampled during training but selected greedily during evaluation.

  • PPO setup: PPO trained an actor producing a categorical distribution over seven therapeutic intentions and a critic estimating the current state value.
  • Optimization: The PPO clipped objective used a clipping parameter of 0.2, with actor and critic learning rates of 3 × 10^-4 and 1 × 10^-3, respectively.The discount factor was 0.99, each update used four optimization epochs, and updates occurred after every 1,000 environment interactions.
  • Training procedure: Training episodes alternated randomly between Anxiety Disorder and Major Depressive Disorder simulated clients, using global seed 2027 and session-specific derived seeds.
  • Network architecture: The actor and critic received the same state vector but used separate multilayer perceptrons without shared hidden layers or parameters.
  • Action selection: Training sampled actions from the actor distribution, whereas evaluation selected the intention with the highest policy probability.

P Generalization Across LLM Backbones

Across Qwen2.5-7B-Instruct and Llama-3.1-8B-Instruct, DeepSAGE preserves its engagement advantage over comparison methods, including its random-policy counterpart. The appendix examples also illustrate how its stage framework maps dialogue turns to explicit CBT objectives.

  • Backbone generalization: DeepSAGE remains strongest on engagement measures with Qwen2.5-7B-Instruct and achieves the highest AD EID.Compared with DeepSAGE_R, it improves AD UL from 0.5926 to 0.6248 and AD SDC from 0.5584 to 0.6017.
  • Backbone generalization: With Qwen2.5-7B-Instruct, the within-backbone ranking is preserved despite smaller absolute values than with gpt-4o-mini.The results indicate that surface-generation quality affects engagement, while DeepSAGE’s advantage over DeepSAGE_R isolates the learned policy’s contribution more clearly.
  • Backbone generalization: With Llama-3.1-8B-Instruct, DeepSAGE outperforms DeepSAGE_R and prompt-based baselines on UL and SDC under both conditions.The larger DeepSAGE–DeepSAGE_R gap than with Qwen suggests greater value from policy-guided action selection when stage instructions are followed less consistently.
  • Overall pattern: Across gpt-4o-mini, Qwen2.5-7B-Instruct, and Llama-3.1-8B-Instruct, DeepSAGE consistently achieves the strongest UL and SDC results and outperforms its random-policy counterpart.The preserved relative advantage suggests that engagement gains are attributable to the learned therapeutic-intention policy rather than a particular proprietary language model.
  • Stage-grounded dialogue: The eleven-stage examples connect generated exchanges to explicit CBT objectives, from rapport and agenda setting through cognitive-model application.The examples describe stage-specific success criteria, including mood ratings, recent-event updates, problem narrowing, and thought–emotion–behavior chains.

Q.5 Stages 9–11 (S9–S11): Summary, Action Planning, and Feedback

Stages 9–11 consolidate client insights, plan a concrete between-session action, and invite reflection on follow-through. The examples connect these exchanges to formal stage goals and confirmation functions.

  • Stages 9–11: The final stages consolidate observations, identify a manageable between-session action, and invite reflection on what may support follow-through.This sequence covers summary, homework review, and feedback within the eleven-stage framework.
  • Stages 9–11: S9 elicits client-generated insights, while S10 proposes a concrete between-session action that the client agrees to undertake.The summarized insights concern self-isolation worsening mood and fear of judgment inhibiting connection; the passage’s discussion of S11 is truncated.
  • Conversation examples: The appendix includes conversation examples for stages 9–11 covering summary, action planning, and feedback.These examples are presented as Table 25 within the stage-wise conversation-example sequence.
Loading 2608.22615v1…