DeepSAGE: Stage-Aware Reinforcement Learning for Structured CBT Counseling Dialogue

Introduction

• One in eight people lives with a mental disorder without receiving valid treatment, while cost, confidentiality concerns, and strained infrastructure limit access to care.The introduction presents scalable, accessible, safe, therapeutically grounded AI as a potential way to broaden equitable access.Source: Introduction · p. 1
• Existing counseling chatbots can be passive and client-led, while LLM counseling systems often lack operationalized therapeutic reasoning and session-level CBT guidance.The introduction identifies reliable, structured, session-level therapeutic guidance as largely unexplored.Source: Introduction · p. 1
• DeepSAGE models an initial CBT session as eleven stages with explicit objectives and automated detection, while a DRL policy selects therapeutic intentions to guide LLM responses.Its therapeutic action space and reward jointly optimize engagement, semantic relevance, stage progression, and completion.Source: Introduction · p. 1
• DeepSAGE shows better model-estimated distress reduction, stronger engagement, and more efficient sessions across diverse clients in a controlled simulated evaluation.Domain experts verified the realism of simulated conversations.Source: Introduction · p. 1

Related Work

• LLM4CBT (Kim et al. 2025), AutoCBT (Xu et al. 2025), and Chinese CBT-LLM systems (Na 2024) address CBT responses but remain single-turn or component-based, while RL dialogue methods lack clinically grounded counseling progression.The related work motivates applying DRL to stage-aware therapeutic conversation within a validated CBT framework.Source: Related Work · p. 2

Proposed Approach: DeepSAGE

• DeepSAGE uses eleven CBT first-session stages, advancing when a client reply meets a stage-goal threshold or the stage-specific turn limit is reached.Goal success combines semantic similarity and NLI entailment, after which the policy selects an intention if neither transition condition is met.Source: Proposed Approach: DeepSAGE · pp. 2–3, Figure 1 · p. 3
• The DRL policy selects among seven therapeutic intentions using dialogue context and stage information, with rewards balancing relevance, client self-disclosure, and stage-turn cost.The dialogue representation uses recent-turn embeddings; the reward penalizes unnecessarily long stages.Source: Proposed Approach: DeepSAGE · pp. 3–4

Experimental Setup

• The evaluation used PPO with a shared gpt-4o-mini backbone across 100 simulated sessions, including anxiety-disorder and major-depressive-disorder client conditions.Because interactions were simulated and several metrics were model-scored, the results are comparative indicators rather than evidence of clinical effectiveness or real-world benefit.Source: Experimental Setup · pp. 4–5

Numerical Analysis & Results

• Naïve LLM had the highest MDD EID but substantially lower engagement, while DeepSAGE balanced distress reduction with high engagement across conditions.DeepSAGE_R underperformed DeepSAGE on all three AD metrics and had an approximately 22.7% lower MDD EID; significance tests found condition-dependent EID differences and higher SSR for DeepSAGE in several comparisons.Source: Numerical Analysis & Results · pp. 5–6
• Approximately 59.9% higher MDD UL and 80.8% higher MDD SDC than LLM4CBT accompanied DeepSAGE’s highest engagement; AD EID was approximately 33.1% above LLM4CBT.DeepSAGE also achieved the highest UL and SDC for both client conditions and the highest AD EID.Source: Numerical Analysis & Results · p. 5, Table 3 · p. 6
• Approximately 10.9% higher AD SSR and 9.3% fewer counselor utterances, plus 8.6% higher MDD SSR and 7.0% fewer utterances, distinguished DeepSAGE from DeepSAGE_R.DeepSAGE achieved the highest SSR with the fewest utterances; Stage-Prompt required the most utterances, while Full-Protocol had the lowest SSR among stage-structured systems.Source: Numerical Analysis & Results · pp. 5–6, Table 4 · p. 6
• Sensitivity analysis selected α = 0.5 because it had the lowest forced-transition rate among settings with goal-success scores of at least 0.66.Although α = 0.8 produced a higher average goal-success score and fewer turns, its forced-stage-transition rate was higher.Source: Numerical Analysis & Results · p. 6
• In expert review of 18 randomly sampled DeepSAGE conversations, 66.7% were rated “Agree” or “Strongly agree” for resembling real counseling practice; 75.0% of valid judgments found client changes plausible, and half were rated positively for recognizable counseling processes.Source: Numerical Analysis & Results · pp. 6–7
• Experts judged the conversations broadly plausible and CBT-grounded, while noting over-questioning, repetitive validation, and occasional premature problem-solving.The review is preliminary evidence of dialogue realism, not clinical effectiveness or safety.Source: Numerical Analysis & Results · p. 7

Conclusions & Future Work

• With simulated clients, DeepSAGE mostly achieved the highest engagement and emotional-intensity-drop scores and improved session success and efficiency over the non-DRL variant.The paper presents these findings as support for stage-structured modeling with DRL-guided strategy selection in structured counseling dialogue.Source: Conclusions & Future Work · p. 7

Ethical Statement

• DeepSAGE is a research prototype that should not be deployed with real users without clinician oversight, crisis escalation, and rigorous human safety and efficacy evaluation.The current action space lacks dedicated escalation, crisis-referral, or boundary-setting responses for acute risk; future work proposes these actions, broader counseling settings, and human studies.Source: Ethical Statement · p. 7, Conclusions & Future Work · p. 7

A Comparison with Prior Counseling Systems

• Unlike prior systems that typically support selected counseling features, DeepSAGE combines therapeutic grounding, adaptive interaction, and explicit stage-driven progression in a coherent initial-CBT workflow.The paper identifies rapport building, active listening, and structured progression as capabilities often absent even from systems combining multiple techniques.Source: A Comparison with Prior Counseling… · p. 10

B Comparison with Commercial Mental Health Chatbots

• Commercial mental-health chatbots range from consumer self-guided support to clinician-integrated CBT assistance, companionship, and peer support, with varied therapeutic scope, validation, and safety capabilities.The comparison positions these systems across target users, modality, therapeutic framing, safety practices, and reported empirical support.Source: B Comparison with Commercial Mental… · p. 10

C CBT Session Stage Definitions

• DeepSAGE models counseling as eleven stages with explicit objectives and success criteria that govern progression to the next stage.These stage definitions underpin the stage-transition mechanism and DRL dialogue policy.Source: C CBT Session Stage Definitions · p. 10

D Action Space Design for Structured CBT Sessions

• DeepSAGE selects therapeutic intentions before LLM response generation, separating strategic decisions from language realization for more interpretable, consistent counseling behavior.Its compact DRL action space uses seven CBT-oriented intentions selected from a broader therapist-intention taxonomy.Source: D Action Space Design for Structured… · pp. 10–11

E Counselor Chatbot Prompt Templates

• Counselor prompts provide stage context, the selected intention, and response constraints, while requiring concise, empathetic, non-diagnostic, non-prescriptive replies.Intent-specific templates add objectives, strategies, tone, constraints, and examples; prompts also call a safety protocol for detected high-risk situations.Source: E Counselor Chatbot Prompt Templates · p. 11

F Baseline Details

• RAB retrieves the top three CBT-resource passages using cosine similarity and combines them with six recent utterances to generate psychoeducational responses without stage management or learned strategy selection.The resources are embedded offline with all-MiniLM-L6-v2; the counselor uses DeepSAGE’s LLM backbone and responds in 2–4 sentences.Source: F Baseline Details · pp. 11, 14
• Table 2’s commercial-chatbot information comes from public documentation, with vendor-reported evidence claims unaudited except for Koko’s peer-reviewed evaluation.The passage cautions that lack of publicly reported evidence does not establish lack of internal validation.Source: F Baseline Details · p. 12, Table 2 · p. 12
• The Naïve LLM-Bot uses the shared counselor model for empathetic support without CBT stages, strategy labels, external knowledge, or policy-based control.It responds using the eight most recent turns in 2–4 sentences, testing supportive counseling without explicit therapeutic structure.Source: F Baseline Details · p. 14
• LLM4CBT (Kim et al. 2025) prompts the shared counselor model turn by turn from only the latest client utterance, testing utterance-level CBT alignment rather than session planning.It requests attention to thoughts, feelings, and behaviors, guided discovery, and avoidance of direct advice, with 2–4 sentence responses and a fixed opening client utterance.Source: F Baseline Details · p. 14
• The Full-Protocol Prompt LLM sees all eleven stages and ten recent turns, but internally decides stage progression without an external goal scorer or learned strategy policy.Its recorded trajectory is constrained to non-decreasing stages, advancing at most one stage per turn, and sessions end at the final stage or 50 counselor turns.Source: F Baseline Details · pp. 14–15
• The Stage-Prompt LLM shares DeepSAGE’s goal-based stage transitions but leaves therapeutic-strategy choice to the LLM, isolating the contribution of learned intention selection.The controller advances when the average of semantic-similarity and entailment scores reaches τ = 0.8, or when the stage-specific turn maximum is reached.Source: F Baseline Details · p. 15
• DeepSAGE_R preserves the staged environment but replaces PPO with uniform random sampling across all seven actions, isolating learned policy optimization.Actions are sampled without stage-specific restrictions.Source: F Baseline Details · p. 15

G Client Chatbot Prompt Templates

• The shared client simulator may respond hesitantly, vaguely, variably, or partially while remaining consistent with its assigned condition and persona.It receives recent dialogue context and stage information but not the PPO action, probabilities, reward, completion score, or transition decision; session-specific deterministic seeds support matched conditions.Source: G Client Chatbot Prompt Templates · p. 15

H Safety Stress Test Evaluation

• Scenarios covered self-harm, abuse, psychosis-like symptoms, acute medical or panic events, diagnosis or medication requests, and excessive dependency; risky utterances were inserted into CBT stages.For each scenario, the policy selected from its seven-action space and the counselor response was generated based on the selected intention and dialogue stage.Source: H Safety Stress Test Evaluation · pp. 15–16, Table 7 · p. 16
• Across evaluated high-risk scenarios, DeepSAGE consistently chose Identify Maladaptive Cognitions, the closest available action—not necessarily an appropriate clinical response.The action space lacks crisis-specific interventions, so the choice reflects its existing constraints rather than establishing clinical safety.Source: H Safety Stress Test Evaluation · p. 16
• The stress test proposes adding safety-oriented actions for human escalation, crisis referrals, diagnosis or medication-advice refusals, and dependency-related boundary-setting.The test’s objective is to assess consistent, interpretable policy behavior in edge cases, not establish clinical safety.Source: H Safety Stress Test Evaluation · pp. 15–16

I Illustrative Eleven-Stage CBT Dialogue

• The illustrative dialogue depicts progression through all eleven first-session CBT stages, from rapport-building and agenda-setting through assessment, psychoeducation, cognitive restructuring, and closure.It presents counseling as a coherent session-level process rather than a collection of independent turns.Source: I Illustrative Eleven-Stage CBT Dialogue · p. 16, Figure 1 · p. 17
• The example uses counselor-client exchanges and sequential arrows; Stage S8 includes multiple turns, and the illustrative utterances are not from a single experimental session.They represent intended conversational behaviors and stage goals; S8 shows that achieving a therapeutic objective may require more interaction.Source: I Illustrative Eleven-Stage CBT Dialogue · p. 16

J Therapeutic Intentions Used by DeepSAGE

• DeepSAGE’s seven therapeutic intentions are high-level strategies: the policy selects the objective, while the LLM determines its natural-language expression.Together, they form a compact, interpretable action space and a semantic interface separating strategic decision-making from language generation.Source: J Therapeutic Intentions Used by… · pp. 16–17

K Sensitivity Analysis

• The highlighted settings, τ = 0.8 and α = 0.5, were used in the main experiments as a balanced trade-off among goal achievement, session efficiency, and genuine goal-based stage transitions.Table 9 reports average goal-success score, counselor utterance turns per session, and forced stage-transition rate for each setting.Source: K Sensitivity Analysis · p. 17
• Sensitivity analysis varied the stage-transition threshold τ and reward weight α, reporting goal-success scores, counselor turns per session, and forced stage-transition rates.Forced stage transitions are those triggered by the maximum turn limit rather than the stage-completion criterion.Source: K Sensitivity Analysis · p. 17

L Expert-Review Survey Instrument

• The survey evaluated dialogue realism, counseling quality, and emotional-intensity change across three components: evaluator background, long-segment dialogue evaluation, and emotional-intensity change evaluation.Source: L Expert-Review Survey Instrument · p. 17
• All evaluators received the same questions and response scales, while the dialogue excerpts varied across survey groups.Source: L Expert-Review Survey Instrument · p. 17

L.1 Evaluator Background

• Evaluators first completed a background questionnaire about their professional expertise and familiarity with CBT.Table 10 summarizes the questionnaire and its response options.Source: §L.1 · p. 17

• Each evaluator reviewed three conversation excerpts between a counselor chatbot and an LLM-based simulated client under standardized conditions.Source: §L.2 · p. 17
• Evaluators were instructed to judge dialogue quality rather than personal counseling-style preference, and to distinguish improvable responses from clearly harmful ones.Source: §L.2 · p. 17
• The questionnaire used six statements with a shared six-point response scale and allowed optional comments on realistic or unrealistic aspects of the conversation.Source: §L.2 · pp. 17–18

L.3 Emotional-Intensity Change Evaluation

• Reviewers judged whether negative emotion increased, decreased, or remained similar between excerpts from different points in the same session.Source: §L.3 · p. 18
• Judgments were based on observable cues including emotional words, urgency, hopelessness, distress, agitation, fear, sadness, anger, shame, physiological descriptions, and perceived loss of control.Source: §L.3 · p. 18
• Reviewers were told not to treat lower emotional intensity as necessarily indicating better therapy, because stronger emotions may emerge after trust develops.Source: §L.3 · p. 18
• The questionnaire covered expert emotional-intensity annotations, rating confidence, and perceived suitability of emotional-intensity drop as a metric, using a shared seven-point scale with chatbot identities concealed.Source: §L.3 · p. 18

M Session Completion and Statistical Significance

• Paired session-level tests matched schemes by run index and, where applicable, client seed and configuration; SSR analysis excluded schemes without stage-indexed responses.The study used two-sided Wilcoxon signed-rank tests with Holm correction separately by client condition and metric family, assessing significance at adjusted α = 0.05. Other measures were UL, SDC, EID, and client utterance count.Source: M Session Completion and Statistical… · p. 19
• AD: DeepSAGE’s SSR exceeded DeepSAGE_R (∆= 0.0880, pHolm < 0.001), Stage-Prompt (∆= 0.0782, pHolm < 0.001), and Full-Protocol (∆= 0.3956, pHolm < 0.001).Across both AD and MDD clients, DeepSAGE achieved higher SSR with fewer counselor turns, indicating more efficient stage progression.Source: M Session Completion and Statistical… · pp. 18–19
• MDD: DeepSAGE’s SSR exceeded DeepSAGE_R (∆= 0.0662, pHolm < 0.001), Stage-Prompt (∆= 0.0522, pHolm = 0.0056), and Full-Protocol (∆= 0.3663, pHolm < 0.001).MDD sessions required more turns than AD sessions, consistent with MDD clients’ shorter, less self-disclosing responses.Source: M Session Completion and Statistical… · pp. 18–19
• DeepSAGE also had higher UL than DeepSAGE_R, RAB, Stage-Prompt, and Full-Protocol; AD EID gains over RAB, Naïve LLM, and LLM4CBT did not extend to structured baselines.For MDD, UL and SDC were higher than RAB and Full-Protocol, and SDC was higher than Naïve LLM.Source: M Session Completion and Statistical… · p. 19
• DeepSAGE produced fewer client utterances than DeepSAGE_R and Stage-Prompt for both conditions, and fewer than Full-Protocol for AD, alongside significantly higher SSR.It produced more client utterances than RAB, Naïve LLM, and LLM4CBT, whose flat-response designs limit session length; shorter sessions versus structured schemes may therefore reflect efficient stage progress rather than lower engagement.Source: M Session Completion and Statistical… · pp. 19–20

N Stage-Wise Action Distribution

• Across both simulated client conditions, the policy’s seven therapeutic actions vary by stage and remain relatively diffuse rather than concentrating on one dominant intent.Tables 15 and 16 report the respective stage-wise distributions for anxiety disorder and major depressive disorder clients.Source: N Stage-Wise Action Distribution · p. 20
• 27.78% of anxiety-client selections in Obtain Update S4 were Focus a3, while Identify Maladaptive Cognitions a5 led Set Agenda S2 at 23.53%.Focus a3 also led in Apply Cognitive Model S8 (18.33%) and Review Homework S10 (21.28%); Normalize Experience a6 led Mood Check S3 (24.00%), and Identify Feelings a4 peaked in Elicit Summary S9 (19.12%). Identify Problems and Goals S6 had a balanced distribution, with four actions each at 15%–16%.Source: N Stage-Wise Action Distribution · p. 20
• 25.93% of MDD-client selections in Obtain Update S4 were Support a0, while Identify Maladaptive Cognitions a5 led Mood Check S3 at 24.44%.Support a0 also led Greet S1 (22.50%) and tied with Identify Maladaptive Cognitions a5 in Set Agenda S2 (20.17%); Identify Feelings a4 led Identify Problems and Goals S6 (20.31%), Normalize Experience a6 led Educate About the Cognitive Model S7 (15.71%), and Focus a3 led Elicit Summary S9 (21.74%). Later-stage action selections were balanced, with ties in S10 and S11.Source: N Stage-Wise Action Distribution · p. 20
• Anxiety clients’ policy emphasized Focus during updates, cognitive-model application, and homework review, whereas the MDD policy relied more on Support during rapport-building and information gathering.The anxiety policy also selected Normalize Experience more in Mood Check and Identify Maladaptive Cognitions more in agenda setting and diagnostic discussion; the MDD policy selected Identify Feelings more in S6. These patterns suggest adaptation to the simulated clinical presentation.Source: N Stage-Wise Action Distribution · p. 20

O Additional DeepSAGE Implementation and Training Details

• Appendix O adds implementation details, including a PPO state comprising recent-dialogue semantics, the current CBT stage, and elapsed utterances within that stage.Source: O Additional DeepSAGE Implementation… · p. 20, §O.1 · p. 20

O.1 State Representation

• The recent dialogue is encoded as one sentence embedding, the stage as an 11-dimensional one-hot vector, and the state dimension as dMiniLM + 11 + 1.An empty dialogue history uses a zero vector.Source: §O.1 · p. 20
• Within-stage time counts counselor and client utterances separately, so each counselor–client exchange increments the count by two.Source: §O.1 · p. 20

O.2 Action Implementation

• The PPO action space has seven therapeutic intentions, and the policy selects an intention index rather than generating natural-language text.Source: §O.2 · p. 20
• All seven intentions are available at every stage; the selected intention’s therapeutic guidance and the current dialogue context are placed in the counselor prompt.The counselor LLM generates the surface-form response.Source: §O.2 · p. 21

O.3 Entailment and Goal-Completion Models

• The semantic goal score compares sentence embeddings of the stage goal and client response, mapping cosine similarities at or below 0.3 to zero and at or above 0.7 to one.Intermediate similarities are linearly rescaled.Source: §O.3 · p. 21
• The entailment component uses cross-encoder/nli-distilroberta-base, treating the client response as premise and the stage goal as hypothesis, then retaining the entailment-class probability.For three-class outputs, logits are converted to probabilities with softmax.Source: §O.3 · p. 21
• Dialogue advances when the stage-goal score reaches 0.8 or the maximum number of within-stage utterances is reached.Source: §O.3 · p. 21

O.4 PPO Training

• The PPO actor samples among seven therapeutic intentions during training, while the critic estimates state value; evaluation selects the highest-probability action greedily.The actor outputs a categorical distribution, and the critic estimates a scalar value.Source: §O.4 · p. 21
• PPO uses a 0.2 clipping parameter, actor and critic learning rates of 3 × 10−4 and 1 × 10−3, a 0.99 discount factor, and four optimization epochs per update.Policy updates occur after every 1,000 environment interactions.Source: §O.4 · p. 21
• Training episodes randomly alternate between Anxiety Disorder and Major Depressive Disorder simulated clients, with random generators initialized using seed 2027.Each simulated session uses a session-specific seed derived from the global seed.Source: §O.4 · p. 21
• Separate actor and critic multilayer perceptrons receive the same state vector but share neither hidden layers nor parameters; execution uses the first available CUDA device or CPU.Linear layers use PyTorch’s default initialization.Source: §O.4 · p. 21

P Generalization Across LLM Backbones

• Qwen and Llama results preserve DeepSAGE’s advantage, while EID patterns vary by condition and engagement remains important to interpretation.With Qwen, DeepSAGE improves AD UL from 0.5926 to 0.6248 and AD SDC from 0.5584 to 0.6017 versus DeepSAGE_R; under MDD, UL rises from 0.5568 to 0.5849 and SDC from 0.5182 to 0.5573. Llama preserves DeepSAGE’s UL and SDC advantage over DeepSAGE_R and prompt baselines, with a larger DeepSAGE–DeepSAGE_R gap than Qwen; DeepSAGE has the highest AD EID, while the Naïve LLM has the highest MDD EID without comparable engagement.Source: P Generalization Across LLM Backbones · p. 21
• Across gpt-4o-mini, Qwen2.5-7B-Instruct, and Llama-3.1-8B-Instruct, DeepSAGE consistently achieves the strongest UL and SDC results and outperforms its random-policy counterpart.The preserved relative advantage suggests engagement gains come from the learned therapeutic-intention policy rather than a particular proprietary model, although absolute performance varies by backbone.Source: P Generalization Across LLM Backbones · pp. 21–22

Q Conversation Examples Across CBT Stages

• The appendix uses selected simulated first-session CBT conversations to illustrate dialogue flow across consecutive stages and connect exchanges to stage goal-success criteria.Examples come from multiple generated sessions and are kept compact while illustrating each stage’s function.Source: Q Conversation Examples Across CBT… · p. 22

Q.1 Stages 1–2 (S1–S2): Greeting and Agenda Setting

• In S1–S2, a warm open-ended greeting invites the client to speak, then agenda setting proposes a topic for confirmation or redirection.The client confirms the topic and adds lack of motivation, illustrating the agenda-confirmation-with-additions structure.Source: §Q.1 · p. 22

Q.2 Stages 3–4 (S3–S4): Mood Check and Recent Update

• In S3–S4, the counselor elicits a numeric mood rating, then asks for a recent contributing event, yielding distinct information.The client reports a missed deadline and resulting self-criticism rather than repeating the mood rating.Source: §Q.2 · p. 22

Q.3 Stages 5–6 (S5–S6): Functional Impact and Problem Identification

• S5 checks the client’s understanding of how self-critical thoughts relate to emotion, and S6 narrows discussion to a specific recurring problem.The client identifies academic self-comparison in class; these stages assess daily functioning and identify a problem area.Source: §Q.3 · p. 22

Q.4 Stages 7–8 (S7–S8): Cognitive Model and Application to an Event

• S7 elicits a thought–emotion–behavior chain, and S8 applies that pattern to a concrete recent presentation event.The client links anticipated failure to anxiety and an urge to avoid, then traces the same pattern in the presentation example.Source: §Q.4 · p. 22

• Across S9–S11, the counselor consolidates observations, identifies a manageable between-session action, and invites reflection on supporting follow-through.Source: §Q.5 · p. 22
• In S9, the counselor asks the client to summarize key insights, including how isolation worsens mood and fear of judgment inhibits connection; the client affirms them.This fulfills the confirmation function fconfirm.Source: §Q.5 · p. 22
• In S10, the counselor proposes texting a friend as a concrete, low-effort between-session task, and the client’s agreement satisfies the homework criterion.The passage identifies this as fhomework.Source: §Q.5 · p. 22
• In S11, the counselor asks what would ease follow-through, and the client offers feedback by reframing vulnerability and lowering expectations for the friend’s reaction.The reflective response signals engagement with the plan and completes the session.Source: §Q.5 · p. 22