PaperlayerView exact revision on arXiv

Source-linked AI summary

ESCRAG-R1: Retrieval-Augmented Reinforcement Learning for Emotional Support Conversation

Weichu Liu, Yuxuan Hu, Yirong Sun, Ningning Mao, Ziyun Zhang, Jian Chen, Mingyang Xu, Qishan Zhong, Chengming Li

arXiv:2608.21925v1cs.AI

TL;DR

ESC systems struggle to combine structured, stage-aware reasoning with seamless empathy and professional grounding. ESCRAG-R1 integrates retrieval-based psychological guidance into GRPO and builds ESC-PREFERENCE for empathy-aware reward supervision. The framework improves emotional support quality across model scales and achieves stronger empathy-expertise alignment than general-purpose and specialized ESC models.

  • Problem

    Existing ESC methods struggle to simultaneously achieve structured, stage-aware reasoning and seamless empathy-expertise alignment.

  • Method

    ESCRAG-R1 integrates retrieval-based psychological guidance into GRPO, using ESC-PREFERENCE and a Client–Counselor–Judge framework for reward modeling and retrieval grounding.

  • Results

    ESCRAG-R1 improves emotional support quality across model scales and achieves stronger empathy-expertise alignment than general-purpose and specialized ESC models.

  • Takeaways & Limitations

    Retrieval-augmented reinforcement learning provides a unified framework for integrating professional therapeutic strategies with empathetic expression in ESC.

  • Takeaways & Limitations

    Experiments focus mainly on 3B and 7B policy models and text-only emotional support, leaving larger-scale and multimodal settings underexplored.

Abstract

from arXiv · show

Emotional Support Conversation (ESC) systems aim to provide holistic support by balancing professional therapeutic competence with natural empathy. However, existing methods struggle to simultaneously achieve structured, stage-aware reasoning and seamless empathy-expertise alignment, often resulting in an artificial splicing of clinical strategies and generic reassurance. To overcome these limitations, we propose ESCRAG-R1, a unified framework that integrates retrieval-based psychological guidance into Group Relative Policy Optimization (GRPO). By incorporating retrieval into the reinforcement learning loop, ESCRAG-R1 transforms external knowledge into a robust learning signal that stimulates explicit internal reasoning prior to generation and fundamentally reshapes the model's internal policy. To provide the reliable supervision required for this optimization, we construct ESC-Preference, a high-quality dataset based on a Client--Counselor--Judge evaluation framework that delivers precise, empathy-aware reward signals. Extensive experiments demonstrate that ESCRAG-R1 significantly outperforms existing baselines by mitigating superficial splicing and realizing a natural integration of professional guidance and empathetic expression. Code and datasets are released at https://github.com/Matcha-Liu/ESCRAG-R1.

1 Introduction

ESC systems must combine structured, stage-aware psychological reasoning with carefully calibrated empathy, but existing approaches struggle to integrate these capabilities. ESCRAG-R1 addresses this gap by embedding retrieved psychological guidance into GRPO and using ESC-PREFERENCE for empathy-aware reward supervision.

  • Existing ESC systems struggle to jointly maintain rigorous psychological reasoning and naturally calibrated empathetic language.Superficial reassurance can lack clinical structure, while technically accurate responses can feel affectively detached.
  • Parameter-based methods often rely on surface imitation without explicit guidance for adapting to a client’s evolving psychological state.
  • Inference-augmented methods inject psychological knowledge during inference, but this information often remains peripheral to the model’s core decision-making.
  • ESCRAG-R1 integrates retrieval-based psychological guidance into GRPO to stimulate explicit stage-aware reasoning before response generation.The framework uses retrieved counseling strategies to support internal psychological assessment rather than surface-level imitation.
  • ESC-PREFERENCE uses a three-dimensional Client–Counselor–Judge evaluation framework to provide reward supervision and retrieval-grounding exemplars.Its preferred responses serve as practice-grounded examples for training and inference.
  • Experiments show improved empathy-expertise alignment and more grounded, supportive, and therapeutically appropriate responses.

2 Related Work

Related ESC research primarily uses parameter-based optimization or inference-time augmentation to improve supportive behavior. These paradigms provide useful training or external guidance but leave limitations in jointly integrating empathy, psychological knowledge, and model decision-making.

  • Parameter-based ESC systems internalize supportive behaviors through supervised fine-tuning or reinforcement learning on specialized counseling datasets.
  • Inference-augmented systems introduce external knowledge or structured reasoning during inference without modifying model parameters.
  • Dialogue-agent frameworks and psychological theories are used to make ESC systems more empathetic, helpful, and grounded in clinically validated principles.
  • Retrieval-augmented generation incorporates external knowledge or relevant dialogue exemplars during inference, while the augmented information often remains peripheral to core decision-making.

3 Method

ESCRAG-R1 combines preference supervision, structured supervised initialization, and retrieval-augmented GRPO to train an emotional-support policy. Retrieved counseling exemplars guide both response generation and policy updates, while multi-perspective evaluation supplies reward signals.

  • ESC-Preference Construction: ESC-Preference provides preference pairs for reward modeling and counseling exemplars for retrieval grounding.The dataset is built from simulated multi-turn client–counselor interactions and uses stage-wise selection with final preference filtering.
  • ESC-Preference Construction: Candidate responses are produced through three counseling stages: understanding context, assessing psychological state, and selecting an intervention strategy.This design grounds generation in counseling-oriented reasoning rather than only surface dialogue context.
  • ESC-Preference Construction: Client, counselor, and judge perspectives evaluate emotional support quality, therapeutic coherence, professional grounding, state fit, and problem targeting.Dominance-based filtering uses these evaluation results to obtain contrasting preference pairs.
  • Supervised Pattern Initialization: The policy is initialized with 500 GPT-4o cold-start demonstrations containing structured reasoning and final responses.Supervised fine-tuning teaches the reasoning–response format and provides a stable initialization for subsequent GRPO.
  • Retrieval-Augmented GRPO: Retrieval-augmented GRPO exposes the policy to counseling exemplars during rollouts, allowing retrieval guidance to affect response generation and policy updates.An initial response queries the retrieval corpus, and the policy then generates a final response conditioned on the dialogue state, initial response, and retrieved exemplars.

4 Experiment

Experiments evaluate ESCRAG-R1 across model scales, perspectives, framework components, retrieval counts, and human preferences. Results show strong overall ESC performance, improved empathy–expertise balance, benefits from joint retrieval-policy optimization, and sensitivity to retrieval quantity.

  • Main Results: ESCRAG-R1-7B with RAG achieves the highest average score of 8.63, while ESCRAG-R1-3B with RAG reaches 8.50 across Client, Counselor, and Judge perspectives.The results indicate effectiveness across both evaluated policy-model scales.
  • Main Results: Compared with GPT-4o with RAG, ESCRAG-R1-7B scores 9.60 vs. 8.96 on Non-Lecturing Empathy and 8.68 vs. 8.56 on Therapeutic Alliance.It also records 8.98 on Strategic Coherence and 8.09 on Theoretical Application.
  • Main Results: ESCRAG-R1-7B improves on PsyLLM-8B in all three Client-side metrics and raises Strategic Coherence from 8.68 to 8.98 and Theoretical Application from 7.87 to 8.09.ChatCounselor-7B obtains a lower average score of 6.12.
  • Ablation Analysis: The ablation average rises from 7.12 for Qwen-2.5-3B-Instruct to 7.30 with RAG, 8.14 after Cold-Start SFT, 8.35 with Vanilla-GRPO, 8.44 with RAG Inference, and 8.50 for full ESCRAG-R1-3B.The full framework further improves Emotional Impact, Therapeutic Alliance, Strategy–Client Fit, and Problem Targeting over Vanilla-GRPO + RAG Inference.
  • Ablation Analysis: Retrieving n = 3 exemplars raises ESCRAG-R1-7B’s average score from 8.44 at n = 2 to 8.63, while n = 4 slightly decreases it to 8.61.The reported pattern suggests that additional exemplars can become redundant or less focused.
  • Human Evaluation: On 100 ESConv test contexts, ESCRAG-R1-7B with RAG records 57 wins against GPT-4o with RAG, 54 against PsyLLM-8B without RAG, and 63 against Llama-3.1-8B-Instruct with RAG.Human annotators prefer ESCRAG-R1 in more cases against each listed baseline.
  • Case Study: Retrieval-guided responses provide warmer validation, avoid premature advice, and encourage exploration rather than directly prescribing solutions.During GRPO training, rewards for these responses are used to shape the policy beyond inference-time retrieval.

5 Conclusion

The paper presents ESCRAG-R1 as a retrieval-augmented reinforcement-learning framework for emotional support conversation and introduces ESC-PREFERENCE for reward modeling and retrieval grounding. Experiments report improved support quality across model scales and stronger empathy–expertise alignment than general-purpose and specialized ESC models.

  • Conclusion: ESCRAG-R1 integrates retrieval-augmented reinforcement learning with ESC-PREFERENCE, a multi-perspective dataset supporting reward modeling and retrieval grounding.The framework targets emotional support conversation.
  • Conclusion: Experiments show improved emotional support quality across model scales and stronger empathy–expertise alignment than general-purpose and specialized ESC models.This is the paper’s reported overall conclusion.

6 Limitations

The study evaluates ESCRAG-R1 mainly on 3B and 7B policy models and only in text-only emotional support conversation. Larger-scale models and multimodal emotional signals remain outside the investigated scope.

  • Limitations: Experiments are mainly conducted on 3B and 7B policy models, leaving performance on larger-scale LLMs insufficiently investigated.The authors attribute this scope to computational resource constraints.
  • Limitations: The work focuses on text-only ESC, excluding multimodal signals such as facial expressions, speech tone, and other behavioral cues.These signals may provide important information about users’ emotional states.

7 Ethical Considerations

The framework uses role-playing prompts to simulate clients, evaluate responses, and structure counselor reasoning across counseling stages and psychological states. It is intended for academic research, with safeguards and human oversight required before any practical deployment.

  • Ethical scope: The work uses publicly available, open-source data and models without private user data or personally identifiable information.
  • Safety measures: The dataset pipeline uses prompts for client simulation, counselor reasoning, and response evaluation to reduce harmful or inappropriate content.
  • Deployment boundary: The authors warn that automatically generated emotional-support responses may pose risks without safeguards and should not substitute for professional services.
  • Deployment boundary: Future deployment should include strict safety mechanisms, human oversight, and instructions to seek professional help in high-risk situations.
  • Role-playing prompts: Client role-playing prompts enforce consistency with emotional and situational context, natural therapist interaction, and one short sentence per turn.
  • Role-playing prompts: Evaluator prompts use client, counselor, and judge roles to assess candidate responses for empathetic quality and professional appropriateness.
  • Role-playing prompts: Counselor reasoning is decomposed into counseling-stage identification, psychological-state assessment, and therapeutic-strategy selection.

B Prompt Design for ESCRAG-R1 Training Framework

The training framework combines reward-model evaluation with a structured policy prompt. It asks the model to reason before responding and to incorporate retrieved guidance while preserving empathy, helpfulness, and natural conversation.

  • B.1 Reward Model Prompt: The reward model evaluates whether a counselor response addresses the client’s emotional problem given the conversation history.
  • B.1 Reward Model Prompt: The reward-model instruction supplies conversation history and a candidate response for evaluation.
  • B.2 Policy Model Prompt: The policy model generates responses in a structured reasoning–response format during retrieval-augmented generation.
  • B.2 Policy Model Prompt: The system prompt assigns the model the role of a therapist helping a patient reduce emotional distress and work through challenges.
  • B.2 Policy Model Prompt: The model is instructed to think through its reasoning before producing the patient-facing response.
  • B.2 Policy Model Prompt: The user prompt provides dialogue context, an initial response, and retrieved external-library guidelines for generating the final response.

C Parameter Setting

The reported settings specify the hardware and base model used for reward-model training and Retrieval-Augmented GRPO. ESCRAG-R1-3B uses two A800 GPUs, while ESCRAG-R1-7B uses four.

  • Parameter settings: The reward model uses Qwen-2.5-7B and is trained on two A800 GPUs.
  • Parameter settings: ESCRAG-R1-3B is trained on two A800 GPUs, whereas ESCRAG-R1-7B is trained on four A800 GPUs.
  • Parameter settings: Table 4 summarizes the hyperparameter settings for reward-model training and Retrieval-Augmented GRPO.

D Retrieval-Augmented GRPO and RAG Inference

ESCRAG-R1 uses retrieved counseling exemplars during GRPO rollouts, updates the policy with rewards from retrieval-guided responses, and applies retrieval again during inference without parameter updates.

  • Overall procedure: Algorithm 1 covers retrieval-augmented GRPO during training and retrieval-augmented generation during inference.
  • Retrieval-Augmented GRPO: During training, retrieved counseling exemplars are incorporated into rollouts, and retrieval-guided response rewards update the policy.
  • Retrieval-Augmented GRPO: Each training step samples a dialogue state, generates an initial response, produces a retrieval-augmented response, and computes its reward.
  • Retrieval-Augmented GRPO: GRPO computes group-relative advantages and a clipped loss before updating the policy parameters.
  • RAG Inference: At inference, the optimized policy generates an initial response, retrieves relevant exemplars, and generates the final response without parameter updates.

E Human Evaluation

The human evaluation uses trained psychology-background annotators and blind pairwise comparisons to assess generated responses across three support-oriented dimensions.

  • Three psychology-background students receive training on the evaluation criteria and scoring prompt before annotation.The training is intended to clarify each evaluation dimension and promote consistent application of the criteria.
  • Annotators evaluate blind pairs of model responses with model identities hidden to reduce potential bias.Each dialogue context is assessed using responses generated by different models.
  • The Client–Counselor–Judge evaluation covers emotional support quality, therapeutic coherence, and contextual appropriateness.Scores are assigned from each of the three specified perspectives.
  • Model preference is determined by comparing annotators’ scores for the two responses in each blind pair.
Loading 2608.21925v1…