Source-linked AI summary
Towards Emotional Support Dialog Systems
Siyang Liu, Chujie Zheng, Orianna Demasi, Sahand Sabour, Yu Li, Zhou Yu, Yong Jiang, Minlie Huang
TL;DR
Dialog systems need emotional support capabilities, but research has lacked a well-designed task and effective-support corpora. This paper defines ESC, introduces a theory-grounded framework and ESConv dataset, and evaluates support-capable dialog models. The results emphasize support strategies and indicate ESConv’s utility for improving emotional support systems.
Problem
Research on data-driven emotional-support dialog systems is limited by the lack of a well-designed task and corpora demonstrating diverse support skills.
Method
The paper defines ESC, adapts Helping Skills Theory into a three-stage framework, constructs richly annotated ESConv conversations, and applies training and quality-control procedures.
Results
Experimental results show the importance of support strategies for effective emotional support and ESConv’s potential utility for improving dialog systems’ support ability.
Takeaways & Limitations
Strategy selection and realization, user-state modeling, and support-effectiveness evaluation are important directions for future research.
Takeaways & Limitations
The appropriate levels of emotional support for dialog systems remain uncertain, and some forms of support may be inappropriate, unrealistic, or risky to deliver.
Abstract
from arXiv · showhide
Emotional support is a crucial ability for many conversation scenarios, including social interactions, mental health support, and customer service chats. Following reasonable procedures and using various support skills can help to effectively provide support. However, due to the lack of a well-designed task and corpora of effective emotional support conversations, research on building emotional support into dialog systems remains untouched. In this paper, we define the Emotional Support Conversation (ESC) task and propose an ESC Framework, which is grounded on the Helping Skills Theory. We construct an Emotion Support Conversation dataset (ESConv) with rich annotation (especially support strategy) in a help-seeker and supporter mode. To ensure a corpus of high-quality conversations that provide examples of effective emotional support, we take extensive effort to design training tutorials for supporters and several mechanisms for quality control during data collection. Finally, we evaluate state-of-the-art dialog models with respect to the ability to provide emotional support. Our results show the importance of support strategies in providing effective emotional support and the utility of ESConv in training more emotional support systems.
1 Introduction
Emotional support aims to reduce distress and help people work through challenges, but dialog-system research lacks a well-designed task and effective-support corpora. The paper defines ESC, proposes a three-stage framework, constructs ESConv, and evaluates dialog models.
- Emotional support reduces individuals’ distress and helps them understand and work through their challenges.
- Emotional support is relevant to social interactions, mental health support, and customer service chats.
- Existing emotional-chatting and empathetic-responding systems lack many support skills used for effective emotional support.
- The paper defines ESC and proposes a Helping Skills Theory-based framework with Exploration, Comforting, and Action stages.
- ESConv uses help-seeker and supporter roles, rich annotations, trained supporters, and manual or automatic quality-control mechanisms.
2 Related Work
Related work distinguishes emotional support from emotional chatting and empathetic responding, while existing support datasets often lack sufficiently long, multi-turn interactions.
- Emotional chatting focuses on expressing designated emotions, whereas emotional support aims to reduce users’ distress through support skills.
- Empathetic responding focuses on understanding users’ feelings and replying accordingly, while emotional support has a broader support-oriented aim.
- Social-context support datasets include Twitter, TalkLife, and mental-health-forum posts or responses annotated with supportive categories or communication mechanisms.
- Post-response pairs or very short conversations limit effective emotional support, which often requires many interaction turns.
- Some conversational systems provide support through human-crafted rules, supportive-act selection, or predefined reply candidates.
3 Emotional Support Conversation
The ESC task models support as a conversation in which a system tracks a help-seeker’s emotional state and selects appropriate support strategies. Its framework organizes support into flexible Exploration, Comforting, and Action stages.
- 3 Emotional Support Conversation: The help-seeker has an emotion, intensity level, and underlying challenge, while the supporter must lower emotional intensity through conversation.
- 3 Emotional Support Conversation: ESC includes strategy selection and constrained response generation, emotion-state modeling, and evaluation of support effectiveness.
- 3.2 ESC Framework: Support conversations generally proceed from Exploration to Comforting to Action but can flexibly adjust stages to the help-seeker’s needs.
- 3.2 ESC Framework: The ESC Framework adapts Helping Skills Theory into three stages with corresponding support strategies for dialog-system social interactions.
- 3.2 ESC Framework: The framework extracts seven methods from Helping Skills Theory, plus an “Others” category, as task strategies.
4 Data Collection
ESConv data collection combines supporter training, seeker surveys and feedback, and layered filtering to obtain high-quality emotional support conversations.
- Participant preparation: Only trained supporters who passed the examination were admitted, while help-seekers reported problems, emotions, situations, and experience origins before chatting.The pre-chat survey included five problem categories, seven emotion categories, intensity from 1 to 5, and open-text situation descriptions.
- Supporter training: Supporters learned the ESC Framework through eleven tutorial sub-tasks covering its three stages and eight support strategies.Each sub-task included an example conversation excerpt and quiz question.
- Feedback collection: Help-seekers rated supporter helpfulness after every two new supporter utterances, and phase-average feedback scores were 4.03, 4.30, and 4.44.The scores were calculated across the conversation’s three phases.
- Quality control: Quality control filtered unfinished or short conversations, applied post-chat survey and utterance-length criteria, and manually corrected annotation errors.The initial 2,472 conversations were filtered before automatic approval and subsequent annotation correction.
- Quality control: 1,053 of 1,342 considered conversations were qualified after automatic approval criteria were applied.The qualified set represented 78.5% of the conversations considered after preliminary filtering.
5 Data Characteristics
ESConv contains relatively long, richly annotated support conversations whose strategy use changes across conversation progress while generally following the framework’s stage order.
- Dataset statistics: ESConv contains 1,053 examples with relatively long conversations averaging 29.8 utterances.The paper links this length to the many turns often required for effective emotional support.
- Dataset statistics: The dataset annotations cover help-seekers’ problems, emotions, feedback, and support strategies.These annotations were released to facilitate further research.
- Seeker characteristics: Depression and job crisis were the most commonly stated problems, while depression and anxiety were the most commonly noted emotions.The paper attributes these distributions partly to the period of the COVID-19 outbreak.
- Strategy distribution: Strategy proportions were computed across six conversation-progress intervals and plotted at six corresponding points.The intervals divide normalized progress into consecutive fifth-based ranges.
- Strategy distribution: Supporters generally follow the ESC Framework’s stage order but flexibly adjust stages and strategies during conversations.Early turns tend to use exploratory strategies such as Question, whereas later turns tend to include Providing Suggestions.
6 Experiments
The experiments compare strategy-aware and strategy-free dialog variants using automatic and human evaluations. Support strategies improve model performance when available, while predicted strategy timing and selection remain important for effective support.
- Automatic Evaluation: Oracle models significantly outperform Vanilla models on all automatic metrics, demonstrating the utility of support strategies.The evaluated metrics are perplexity, BLEU-2, ROUGE-L, and BOW Embedding-based Extrema matching.
- Automatic Evaluation: Joint models score slightly below Vanilla models automatically because incorrect predicted strategies can produce responses unlike the references.Despite this automatic disadvantage, strategy prediction is needed when ground-truth labels are unavailable.
- Automatic Evaluation: BlenderBot variants consistently outperform DialoGPT variants, making BlenderBot the backbone for subsequent human evaluation.The comparison covers both strategy-aware and strategy-free variants.
- Human Interactive Evaluation: Fine-tuning BlenderBot on ESConv significantly improves its emotional-support capability across all human-evaluation metrics.The human interactive evaluation compares models through conversations with recruited participants.
- Human Interactive Evaluation: Strategy use improves comfort, while proper strategy timing helps users identify problems and receive effective suggestions.The human evaluation compares Joint with Vanilla and Random variants across 100 conversations per model pair.
- Further Analysis: The Joint model adopts a strategy distribution similar to ESConv’s truth distribution and produces more supportive, skill-diverse responses than unfine-tuned BlenderBot.The analysis links strategy imitation with observed differences in conversational supportiveness.
7 Conclusion
The paper defines Emotional Support Conversation, adapts Helping Skills Theory into an ESC Framework, and constructs ESConv for dialog-system research. Experiments indicate that ESConv can improve systems’ ability to provide effective emotional support and support future work.
- Conclusion: The paper defines the Emotional Support Conversation task and presents an ESC Framework adapted from Helping Skills Theory for dialog systems.The framework characterizes three stages with corresponding support strategies.
- Conclusion: ESConv is constructed as an Emotional Support Conversation dataset, with data collection and quality-control mechanisms designed to support effective conversations.The paper evaluates state-of-the-art dialog models using the dataset.
- Conclusion: Experimental results show ESConv’s potential utility for improving dialog systems’ ability to provide effective emotional support.The paper identifies strategy selection and realization, user-state modeling, and task evaluation as directions for further research.
Ethical Considerations
The paper limits its framework to support elements considered relevant and appropriate for dialog systems, while recognizing that suitable support levels require further study.
- Ethical Considerations: Some forms of human support may be inappropriate, unrealistic, or too risky for dialog systems to deliver.The framework omits elements viewed as clear oversteps and targets reasonably modest levels of support.
- Ethical Considerations: Determining appropriate levels of support for systems remains an open area requiring considerable additional work.The corpus is intended to support future examination of the ethical extent of system-provided support.
A Data Example from ESConv
ESConv records structured seeker context, supporter strategies, and seeker feedback within annotated support conversations. The example illustrates how these annotations represent both the seeker’s situation and the supporter’s responses.
- Conversation Annotations: Each conversation’s pre-chat survey includes a problem category, emotion category, emotion intensity, and brief situation description.The example describes academic pressure, anxiety, intensity 5, and school closure during the pandemic.
- Conversation Annotations: Supporter utterances are labeled with the strategies used during the conversation.The example includes Questions, Affirmation and Reassurance, and Self-disclosure.
- Conversation Annotations: The dataset provides a seeker feedback score after every two supporter utterances to evaluate helpfulness.Figure 6 marks these feedback scores with stars and displays the evaluation question in orange text.
- Support Strategies: Question Asking requests information to help the help-seeker articulate the issues they face.Open-ended questions are preferred, while closed questions can obtain specific information.
- Support Strategies: Restatement or Paraphrasing rephrases the help-seeker’s statements concisely to clarify the situation.Reflection of Feelings describes the help-seeker’s feelings, while Self-disclosure shares similar experiences or emotions to express empathy.
C Implementation Details
The models use a standardized Transformer-based training and inference setup, while ESConv quality-control thresholds are calibrated against human annotators using agreement scores.
- Model training: ESConv is split into training, validation, and test sets in proportions of 6:2:2, with each dialogue divided into five-utterance pieces.Each piece contains one supporter response and the preceding four utterances.
- Model training: All models are trained with Adam using a learning rate of 5e−5, and validation examples with the lowest perplexity are selected for evaluation.
- Inference: During inference, strategy tokens are sampled first, and responses are decoded with Top-k and Top-p sampling using p = 0.9, k = 30, τ = 0.7, and repetition penalty 1.03.Random variants sample strategies according to the strategy distribution in ESConv.
- Quality control: Three trained annotators judge 100 randomly sampled conversations, whose post-survey results and utterance lengths inform thresholds for the automatic filtering rules.
- Quality control: The selected thresholds maximize the second-highest agreement with human annotators rather than the highest agreement because the strictest rules would filter out many conversations.
E Interface of Data Collection Platform
The data-collection platform presents interfaces for supporter training, surveys, and both participants’ chat experiences. A case-study view also links the help-seeker’s survey, predicted strategies, and strategy-corresponding utterances while marking self-focused model utterances.
- Data-collection workflow: Figure 8 covers support strategy training, the help-seeker’s pre-chat survey and chat interface, the supporter’s chat interface, and the post-chat survey.
- Data-collection workflow: The platform diagram is intended to give readers an intuitive view of important data-collection steps, including supporter chatting and post-survey activities.
- Case-study visualization: The case-study display uses blue text for the help-seeker’s pre-chat survey and red text for strategies predicted by the model.
- Case-study visualization: Green backgrounds identify utterances corresponding to the used strategies, while yellow backgrounds identify model utterances that talk about itself and are not user-centered.