Source-linked AI summary
PsychAgent: An Experience-Driven Lifelong Learning Agent for Self-Evolving Psychological Counselor
Yutao Yang, Junsong Li, Qianjun Pan, Jie Zhou, Kai Chen, Qin Chen, Jingyuan Zhao, Ningning Zhou, Xin Li, Liang He
TL;DR
Many AI counseling models remain static after training and lack persistent multi-session context. PsychAgent combines longitudinal memory and planning with therapeutic-skill evolution and rejection fine-tuning; experiments report stronger performance than general and psychology-specific baselines across multiple therapeutic schools.
Problem
Many counseling models remain fixed after training, lack persistent context for multi-session coherence, and have limited evidence of internalizing learned behaviors in model parameters.
Source: §1 · p. 2Method
PsychAgent combines memory-augmented planning, practice-grounded skill evolution, and rejection fine-tuning to integrate evolved skills into model parameters.
Source: §1 · p. 2Results
Takeaways & Limitations
In these experiments, the framework is associated with more consistent, professional, and effective multi-session counseling support.
Source: §5 · p. 8The authors identify evaluation in more realistic counseling settings and stronger safety and privacy protections as future work.
Source: §5 · p. 8
Abstract
from arXiv · showhide
Existing methods for AI psychological counselors predominantly rely on supervised fine-tuning using static dialogue datasets. However, this contrasts with human experts, who continuously refine their proficiency through clinical practice and accumulated experience. To bridge this gap, we propose an Experience-Driven Lifelong Learning Agent (\texttt{PsychAgent}) for psychological counseling. First, we establish a Memory-Augmented Planning Engine tailored for longitudinal multi-session interactions, which ensures therapeutic continuity through persistent memory and strategic planning. Second, to support self-evolution, we design a Skill Evolution Engine that extracts new practice-grounded skills from historical counseling trajectories. Finally, we introduce a Reinforced Internalization Engine that integrates the evolved skills into the model via rejection fine-tuning, aiming to improve performance across diverse scenarios. Comparative analysis shows that our approach achieves higher scores than strong general LLMs (e.g., GPT-5.4, Gemini-3) and domain-specific baselines across all reported evaluation dimensions. These results suggest that lifelong learning can improve the consistency and overall quality of multi-session counseling responses.
1 Introduction
AI counseling tools are emerging amid unmet demand for mental-health support, but many remain static and lack persistent multi-session context. PsychAgent addresses these gaps with experience-driven learning and reports stronger performance than general and domain-specific baselines.
1 Introduction
- Demand for mental-health support exceeds the supply of qualified professionals in many settings, while LLMs show promise for empathetic support and preliminary assistance.Examples include GPT-4o, Llama, and DeepSeek.Source: §1 · p. 1
- Existing counseling models often lack persistent memory for longitudinal coherence and provide limited evidence that learned behaviors become internalized in model parameters.The introduction identifies these as challenges alongside fixed strategies that do not update from interaction history.Source: §1 · pp. 1–2
- PsychAgent achieves higher scores than strong general LLMs and domain-specific baselines, with experiments also reporting superior generalization and consistency in counseling.The framework combines memory-augmented planning, skill evolution, and reinforced internalization.Source: §1 · p. 2
- PsychAgent frames counseling as experience-driven lifelong learning, drawing on human counselors’ practice-based refinement of techniques and clinical intuition.The proposed engines support longitudinal continuity, skill extraction from counseling trajectories, and integration of evolved skills into model parameters.Source: §1 · p. 2
2 Related Work
Prior work has adapted LLMs for counseling and developed systems for structured, multi-session support, while continual-learning and self-evolving-agent research explores adaptation from experience. PsychAgent applies this perspective to counseling through a cycle of clinical experience, therapeutic-skill abstraction, and training-based internalization.
2.1 LLMs for Psychological Counseling
- Mental-health counseling LLM work has moved from domain-specific data and supervision aimed at empathetic, supportive dialogue toward systems incorporating therapeutic structure, while evaluation increasingly covers counseling ability and ethical risks.Source: §2.1 · p. 2
- Multi-session datasets and agents address longitudinal counseling, yet prior systems do not explicitly extract therapeutic skills from accumulated interactions and internalize them into model parameters.MusPsy [8], PsyDial, and PsychEval [21] move toward multi-session data and evaluation; TheraMind incorporates cross-session planning and adaptation.Source: §2.1 · p. 2
2.2 Experience-Driven Lifelong Learning
- Continual-learning research has shifted toward dynamic LLM adaptation, but many approaches emphasize data retention over high-level skill acquisition in open-ended tasks.Earlier approaches addressed stability-plasticity through regularization such as EWC [24] or experience replay [25].Source: §2.2 · p. 2
- Self-evolving agents use recursive experience and reflection, while many rely on transient in-context learning or external retrieval rather than autonomous skill acquisition, refinement, and use.PsychAgent models counseling as accumulating clinical experience, abstracting therapeutic skills, and internalizing them through training.Source: §2.2 · pp. 2–3
- PsychAgent’s architecture links longitudinal memory and planning, hierarchical skill retrieval and abstraction, and rejection fine-tuning of selected trajectories.The three engines are presented as a closed loop for continuity, skill evolution, and internalization.Source: Figure 1 · p. 3
3 Our Methods
PsychAgent combines persistent client memory and strategic planning with a dynamic therapeutic-skill repository and rejection fine-tuning of selected counseling trajectories. Its components support longitudinal continuity, context-aware skill use and evolution, and internalization of successful experience.
3 Our Methods
- PsychAgent frames counseling as a longitudinal decision process in which memory and therapeutic strategies condition responses, while selected trajectories update model parameters across sessions.The framework combines memory-augmented planning and skill retrieval within sessions with reinforcement-based internalization between sessions.Source: §3 · pp. 3–4
3.1 Memory-Augmented Planning Engine
- The Memory-Augmented Planning Engine updates client profiles and episodic session summaries, then uses that memory to plan therapeutic stages and actionable session objectives.The evolving profile captures new client attributes, while summaries preserve emotional shifts and intervention outcomes.Source: §3.1 · p. 4
3.2 Skill Evolution Engine
- The Skill Evolution Engine organizes therapeutic techniques hierarchically and retrieves an atomic skill for the current dialogue, prepending it as a directive for response generation.The hierarchy spans counseling paradigms, therapeutic stages, meta-skills, and concrete atomic skills.Source: §3.2 · pp. 4–5
- After a high-reward trajectory, targeted extraction abstracts an atomic skill under a relevant meta-skill, while adaptive management appends or merges it into the skill tree.This update is designed to prevent redundancy while refining existing techniques.Source: §3.2 · p. 5
3.3 Reinforced Internalization Engine
- The Reinforced Internalization Engine generates parallel session-level rollouts, scores completed candidates, and carries forward only the maximum-reward trajectory and its updated memory.Rejection fine-tuning then maximizes the likelihood of the selected “golden” trajectory, treating lower-reward sessions as implicit negative samples and distilling explicit skills and planning logic into model weights.Source: §3.3 · pp. 5–6
4 Experimental Results
Across the reported evaluation dimensions, PsychAgent outperforms general-purpose and psychology-specific baselines; human ratings and ablations also support its overall advantage and the contributions of its components.
4.1 Experimental Setups
- Experiments use PsychEval multi-session counseling data and compare PsychAgent with general-purpose and psychology-specific model baselines.For each therapeutic school, 140 client profiles are sampled: 120 for training and 20 for evaluation.Source: §4.1 · p. 6
4.2 Main Results
- PsychAgent scores higher than general-purpose and psychology-specific baselines across all four aggregated evaluation dimensions.Against Qwen3-Max, gains are +1.44/+0.17 on counselor-level shared/specific scores and +0.51/+0.43 on client-level shared/specific scores.Source: §4.2 · p. 6, Table 1 · p. 7
4.3 Human Evaluation
- Across 522 matched multi-session dialogues, PsychAgent ranks first with both human raters and the LLM rater.Two human annotators and Gemini-3 rated Ethics, Interaction, Intervention, and Perception.Source: §4.3 · p. 7
4.4 Ablation Studies
- Removing any of the three components lowers counselor-level and client-level results, with the largest ablation drop when Skill Evolution is removed.Removing Reinforced Internalization causes a noticeable loss; removing Memory-Augmented Planning causes a smaller but consistent decline.Source: §4.4 · p. 7, Table 3 · p. 7
4.5 Further Analysis
- Generic memory architectures add limited benefit to DeepSeek-V3.2 and sometimes lower performance, while PsychAgent scores higher across all metrics.The authors suggest that counseling may require tracking evolving mental states, therapeutic progress, and prior intervention rationales, in addition to recalling facts.Source: §4.5 · p. 8, Table 4 · p. 8
- Case studies report practice-grounded skill extraction, recombination of micro-skills into functional skills, and partial reuse across therapeutic orientations.The reported cross-orientation reuse is partial, not therapy-agnostic.Source: §4.5 · p. 8
5 Conclusions and Further Work
PsychAgent combines memory-augmented planning, skill evolution, and reinforced internalization for experience-driven lifelong learning in AI psychological counseling. Across multiple therapeutic schools, the framework outperforms general-purpose and psychology-specific baselines; future work targets realistic settings and stronger safety and privacy protections.
5 Conclusions and Further Work
- PsychAgent integrates memory-augmented planning, skill evolution, and reinforced internalization in an experience-driven lifelong learning framework for AI psychological counseling.The authors describe it as moving beyond static fine-tuning toward improvement through accumulated experience.Source: §5 · p. 8
- Across multiple therapeutic schools, PsychAgent consistently outperforms strong general-purpose LLMs and psychology-specific baselines.The authors describe the aim as more consistent, professional, and effective multi-session support.Source: §5 · p. 8
- Future work will evaluate the framework in more realistic counseling settings and strengthen safety and privacy protections for real-world deployment.Source: §5 · p. 8
A Detailed Results Across Counselor- and Client-Level Therapy Metrics
Figure 3 compares counselor- and client-level therapy metrics across therapy-shared and therapy-specific dimensions. The reported pattern is broad, with especially clear gains in counselor-side planning and intervention measures and improvements also extending to client-side outcomes.
A Detailed Results Across Counselor- and Client-Level Therapy Metrics
- Figure 3 shows consistently strong performance across counselor- and client-level shared and therapy-specific metrics, rather than gains isolated to a few metrics.Source: A Detailed Results Across Counselor-… · p. 13
- The largest gains appear on counselor-side planning and intervention dimensions, especially DIALOGUE_PLANNING and CTRS.The passage connects these gains with structuring therapeutic progress and delivering theory-consistent interventions.Source: A Detailed Results Across Counselor-… · p. 13
- Improvements also extend to client-side outcome measures such as CCT and SFBT.The passage characterizes these as downstream client-level therapeutic progress.Source: A Detailed Results Across Counselor-… · p. 13
B Human Evaluation
The human evaluation rates counselor performance across four dimensions using a conservative, evidence-grounded rubric. Its matched sample spans 522 dialogues, with QWK used to report agreement among raters.
B.1 Raters and Annotation Protocol
- Raters score counselor performance, rather than client state, on Ethics, Interaction, Intervention, and Perception using a 1–5 ordinal scale.Two psychology-trained graduate students and one LLM evaluator rated the dialogues.Source: §B.1 · p. 13
- The rubric requires explicit dialogue evidence, defaults to 3 when evidence is limited, and reserves scores of 4–5 for clear support.Unsafe responses, boundary violations, and misreadings lower scores; abilities are not penalized when the relevant scenario is absent.Source: §B.1 · p. 14
- The authors say the evidence-grounded rubric reduces annotation ambiguity and makes human–LLM comparisons more interpretable and reproducible.Source: §B.1 · p. 14
- The matched set contains 522 fully rated dialogues from 25 cases and 188 case-session units, balanced across the three counselor models.The sample yields 6,264 dimension-level observations.Source: §B.1 · p. 14
B.2 Inter-Rater Agreement
The three counselor models retain the same ranking across raters, although rating distributions and agreement vary by rater and dimension.
B.2 Inter-Rater Agreement
- PsychAgent ranks first, Qwen3-Max second, and TheraMind last across raters, despite a concentration of ratings at 4–5 and some ceiling effect, especially for Perception.Ratings of 4–5 account for 78.8%–81.7% of dimension-level ratings.Source: §B.2 · p. 14
- LLM–human agreement exceeds human–human agreement: QWK is 0.770 for Gemini-3 vs Human Rater A and 0.877 vs Human Rater B, versus 0.675 between the human raters.Exact agreement is 78.7% for LLM vs Human A and 90.3% for LLM vs Human B, compared with 74.2% for Human A vs Human B; nearly all disagreements are within one rating point.Source: §B.2 · pp. 14–15
- Agreement varies by dimension: Intervention has the highest QWK across all three pairings, while Perception has the weakest human–human agreement.Intervention QWK values are 0.747, 0.812, and 0.907; Perception’s human–human QWK is 0.546, with its upper-score concentration likely making estimates more sensitive to small rating-style shifts.Source: §B.2 · p. 15
B.3 Systematic Bias of the LLM Rater
Gemini-3’s rating bias is not a simple global tendency to inflate scores; it varies by human rater, counselor model, and dimension.
B.3 Systematic Bias of the LLM Rater
- Gemini-3’s mean bias is −0.022 relative to Human Rater A and −0.043 relative to Human Rater B, while it is harsher on TheraMind (−0.170) and differs most on Ethics versus Human B (−0.102).Relative to Human A, the LLM is slightly more favorable on PsychAgent and Qwen3-Max; relative to Human B, it is more consistently conservative overall.Source: §B.3 · p. 15
- The authors view Gemini-3 as a scalable complement rather than a replacement for multi-rater human judgment because its alignment varies across dimensions and counselor models.Its judgments are closer to Human Rater B than Human Rater A, and Ethics, Perception, and lower-performing TheraMind remain more sensitive to rating-style differences.Source: §B.3 · p. 15
C.1 Extracting New Skills from Conversations
PsychAgent’s extraction pipeline is assessed by whether it distills practice-grounded additions beyond existing fine-grained skills, rather than merely restating the library. The reported additions mainly make known interventions more executable through thresholds, sequencing, monitoring, and reusable templates.
C.1 Extracting New Skills from Conversations
- The authors define a practice-grounded skill unit as adding non-trivial practical structure beyond the closest existing micro-skills in the same therapeutic orientation and stage.The v3 library is the comparison baseline; a skill counts as novel only when it adds practical structure not already captured by the nearest skills.Source: §C.1 · p. 15
- Across 715 dialogue records, extraction yields 9,701 atomic skills, including 5,923 practice-grounded units; their proportions range from 51.61% in PDT to 73.29% in BT.The reported proportions are 73.29% in BT, 69.77% in CBT, 56.0% in HET, 51.61% in PDT, and 63.49% in PMT.Source: §C.1 · p. 15
- Across orientations, extracted skills add operational sequencing, minimum viable actions, quantitative thresholds, structured templates, and monitoring or recording rules rather than entirely new therapy techniques.The global audit found no sufficiently supported candidates for entirely novel techniques; even the strongest cases are repackagings of existing techniques with practical additions.Source: §C.1 · pp. 15, 16, 18
- BT’s micro-contract case adds a red-light threshold, fallback sequence, minimum practice duration, and client-preferred logging format to a generic contract.The extracted unit adapts logging to the client’s preferred emoji-based format.Source: §C.1 · p. 16
- CBT’s process-card case turns tools into a sequenced workflow with a low activation threshold, optional low-burden recording, and a cue in the client’s own language.The interpretation describes the cue and sequencing as making tools deployable when the client hesitates to begin.Source: §C.1 · p. 16
- HET’s extracted unit translates broad self-care wishes into prioritized, measurable actions with SMART specification and lightweight monitoring.The unit combines a prioritized domain, a SMART target, and a small logging procedure.Source: §C.1 · p. 17
- PDT’s goal-setting unit reduces adherence demands through a minimum daily target, a low-burden record, timing choice, and explicit permission to pause.The interpretation frames this as preserving adherence when reflective work itself becomes aversive.Source: §C.1 · p. 17
- PMT’s strength-card unit turns resource-focused work into a pre-session routine with a minimum quota, evidence categories, and a reusable recording template.The routine is described as gathering material that can feed later force-field analysis.Source: §C.1 · pp. 17–18
C.2 Operational New Functional Skills
After internalization, PsychAgent increasingly organizes existing micro-skills into recurrent, functionally coherent turn-level packages, some of which are reused across therapies. The authors interpret these as operationally new functional units, not newly invented canonical techniques.
C.2 Operational New Functional Skills
- Internalization is associated with more stable, functionally coherent turn-level packages and cross-therapy reuse, rather than simply more multi-skill turns.The authors interpret the pattern as transferable counseling principles and skill-level compositional generalization beyond the original inventory.Source: §C.2 · pp. 18, 20
- The authors caution that these packages are not newly invented canonical techniques, but recurrent compressions of known micro-skills into operationally useful units.They argue that package quality and stability, rather than raw bundle counts, provide the strongest evidence for compositional generalization.Source: §C.2 · p. 18
- CBT’s goal-to-record bridge combines near-term goal negotiation with structured follow-up monitoring in one executable loop.The reported frequency is Internalization: 30; SFT: 58.Source: §C.2 · p. 19
- CBT’s rated reappraisal package completes alternative-thought replacement and immediate feedback within the same turn.The example combines generating and evaluating alternative thoughts with before-and-after ratings of belief and emotion.Source: §C.2 · p. 19
- BT’s dose-specified practice scaffold packages practice parameters, recording rules, and a continuation-versus-pause procedure into an adherence support unit.The dialogue specifies exposure timing, response prevention, a pause condition, and trial ratings; the interpretation characterizes the package as a structured adherence scaffold.Source: §C.2 · p. 19
- PDT’s validation-wrapped pacing package first reduces shame and threat, then reframes the client’s experience for further work.The reported frequency is Internalization: 6; SFT: 0.Source: §C.2 · pp. 19–20
- PMT’s alternative-story package opens a preferred narrative and consolidates it through meaning and competence, forming a narrative-strengthening unit.The reported frequency is Internalization: 241; SFT: 1338.Source: §C.2 · p. 20
- HET’s resource-to-action package translates identified resources into an action ladder, self-positioning language, and a monitoring routine.The interpretation describes the result as a complete practice scaffold rather than an abstract resource identification.Source: §C.2 · p. 20
C.3 Cross-Therapy Reuse
Internalization supports reuse of some higher-order counseling skills across therapies, primarily as functional families rather than frequent exact mixed-therapy bundles.
C.3 Cross-Therapy Reuse
- Goal-to-Practice Bridge translates insight into concrete, reviewable between-session practice across BT, CBT, HET, and PMT.Its recurring principle is to connect discussion or formulation to practice that can be reviewed and adjusted.Source: §C.3 · pp. 21–22
- Progress Consolidation Package recurs across BT, HET, PDT, and PMT by affirming progress, reviewing strategies, and orienting future work.The counselor names progress as a replicable capacity and turns it into a maintainable future direction.Source: §C.3 · p. 21
- Affect Attunement Before Deeper Work recurs across CBT, HET, and PDT, stabilizing or naming emotion before deeper cognitive or experiential work.The reusable structure is to address the emotional experience first, then move toward more challenging exploration.Source: §C.3 · p. 22
- Across these families, what transfers is an intervention principle, not merely a shared skill label.The examples include translating insight into practice, consolidating gains, and preparing clients for deeper work through affect attunement.Source: §C.3 · pp. 20, 22
- Some higher-order functional skills recur across therapeutic contexts, but the evidence does not establish a therapy-agnostic counselor or frequent exact mixed-therapy bundles.The reported portability is at the functional-family level, where intervention principles recur across therapy-specific contexts.Source: §C.3 · pp. 20, 22, Table 9 · p. 21