Source-linked AI summary
Towards Empathetic Open-domain Conversation Models: a New Benchmark and Dataset
Hannah Rashkin, Eric Michael Smith, Margaret Li, Y-Lan Boureau
TL;DR
Dialogue agents need to recognize and respond to partners’ feelings, but suitable benchmarks and datasets for empathetic responding are scarce. The paper introduces a benchmark and the 25k-conversation EmpatheticDialogues dataset, then evaluates retrieval and fine-tuning adaptations. Models using the dataset are evaluated as more empathetic than models trained only on large-scale internet conversation data.
Problem
Dialogue systems lack suitable publicly available benchmarks and datasets for evaluating and training empathetic responses to emotional situations.
Method
The paper introduces EmpatheticDialogues, a publicly released dataset of about 25k emotion-grounded conversations, and tests retrieval candidates, fine-tuning, and external-predictor information.
Results
Models using EmpatheticDialogues through retrieval or fine-tuning produce responses evaluated as more empathetic than models trained on spontaneous internet conversation data.
Takeaways & Limitations
EmpatheticDialogues provides a benchmark and training resource for studying empathetic responding in open-domain dialogue.
Takeaways & Limitations
More extensive experiments with large models are required to confirm whether larger capacity makes additional external supervision effective for this task.
Abstract
from arXiv · showhide
One challenge for dialogue agents is recognizing feelings in the conversation partner and replying accordingly, a key communicative skill. While it is straightforward for humans to recognize and acknowledge others' feelings in a conversation, this is a significant challenge for AI systems due to the paucity of suitable publicly-available datasets for training and evaluation. This work proposes a new benchmark for empathetic dialogue generation and EmpatheticDialogues, a novel dataset of 25k conversations grounded in emotional situations. Our experiments indicate that dialogue models that use our dataset are perceived to be more empathetic by human evaluators, compared to models merely trained on large-scale Internet conversation data. We also present empirical comparisons of dialogue model adaptations for empathetic responding, leveraging existing models or datasets without requiring lengthy re-training of the full model.
1 Introduction
Empathetic responding requires dialogue agents to infer and acknowledge feelings, but existing benchmarks and broadly trained models provide limited support for evaluating this ability. The paper introduces EmpatheticDialogues and tests dataset-based methods for improving empathetic responses.
- Empathetic responding means understanding and acknowledging implied feelings when people describe personal experiences.The paper contrasts an emotionally acknowledging response with one that is merely topically relevant.
- Existing dialogue benchmarks do not assess whether agents can converse with empathy, while internet-trained models may reproduce aggressive or callous responses.
- EmpatheticDialogues provides about 25k crowdsourced one-on-one conversations grounded in emotional situations and balanced across a broad set of emotions.The dataset was publicly released with code to reproduce the paper’s main experiments.
- The experiments test retrieval candidates from EmpatheticDialogues, fine-tuning on the task, and combinations of information from related tasks.These approaches are intended to improve models originally trained on spontaneous internet conversation data.
- Training on EmpatheticDialogues improves end-to-end dialogue-system performance on empathetic dialogue, while baseline large-capacity models are not rated very empathetic.
2 Related Work
Prior work covers emotion classification, controllable affective generation, and personal-context chit-chat, but these resources generally do not directly target empathetic responding in emotionally grounded personal conversations.
- Emotion data: Emotion datasets use varied schemas, and this work combines labels from several sources to represent a broad range of positive and negative emotions.The paper emphasizes that emotions inferred from situations matter in dialogue scenarios.
- Emotion data: Public social-media content can yield imbalanced emotional coverage, so the dataset targets a more balanced domain closer to training agents for any emotion.
- Controllable language generation: Controllable-generation studies target specified emotions or affect levels, evaluating emotion matching rather than empathetic responding.
- Related chit-chat data: Personal-context chit-chat resources focus on personal facts, while DailyDialog contains about 13k educational dialogues with only approximately 5% of utterances labeled beyond “none” or “happy”.This work instead focuses explicitly on emotionally grounded personal situations.
3 Talking about Personal Situations
EmpatheticDialogues is an open-domain, one-on-one dataset in which speakers discuss personal situations associated with emotion labels and listeners respond using conversational cues.
- Each conversation is grounded in one situation associated with one of 32 emotion labels spanning positive and negative emotions.The labels were aggregated from several emotion-prediction datasets.
- The speaker writes a situation description, initiates discussion about it, and exchanges up to six additional turns with a listener.Models are evaluated as listeners responding to speakers without seeing the situation description or emotion label.
- Crowdsourced collection used ParlAI and Amazon Mechanical Turk, with 810 US workers producing situation descriptions and paired conversations.
- Workers respond solely to conversational cues because they cannot see the other participant’s emotion label or situation description.Conversations contain 4–8 utterances on average, with an average of 4.31 utterances per conversation.
- Emotion-selection balancing encourages coverage of less frequently chosen categories so models can handle emotions beyond the most common ones.
- The dataset contains 24,850 conversations about situation descriptions, collected from 810 participants and split into approximately 80% train, 10% validation, and 10% test partitions.The split prevents overlap of discussed situations across partitions by grouping conversations from the same initial situation.
4 Empathetic Response Generation
The section describes retrieval-based and generative dialogue architectures, then adapts pretrained models using EmpatheticDialogues data and external supervised predictors. These adaptations include candidate-pool expansion, fine-tuning, and prepended emotion or topic labels.
- Base Architecture: The experiments use both retrieval-based models that select candidates by context–candidate similarity and generative models that decode responses from encoded context.Retrieval models encode context and candidates separately, while generative models use an encoder–decoder architecture trained on target-response likelihood.
- Training details: Pretrained models learn from 1.7 billion Reddit conversations, with Transformer architectures trained from scratch or BERT-based architectures initialized from released BERT models.Pretrained models are evaluated without fine-tuning on EmpatheticDialogues, and contexts and responses are limited to 100 word tokens.
- 4.2 Leveraging the Training Data from ED: EmpatheticDialogues is used either to add empathetic training candidates at inference time or to fine-tune pretrained models on next-utterance prediction.Candidate-pool expansion applies to retrieval models without ED fine-tuning, while fine-tuning uses a context window of four previous utterances.
- 4.3 Adding Information from External Predictors: External supervised information is incorporated by prepending predicted emotion or topic labels to context and candidate sequences before encoding.The prepend setup requires no architecture modification and can use black-box classifiers; the top predicted label is prepended to the token sequence.
- 4.3 Adding Information from External Predictors: Multitask and feature-concatenation variants were tested but did not provide consistent improvements, while additional top-3 and top-5 prepend variants showed similar result patterns.The top-3 and top-5 experiments are reported in the appendix, and classifier training on utterances produced similar results.
- Evaluation: Table 1 reports BLEU, perplexity, and P@1,100, and states that in-domain fine-tuning clearly improves all automatic metrics relative to pretrained models, whereas other metrics are inconsistent.The table compares pretrained, fine-tuned, and externally augmented models across Reddit, EmpatheticDialogues, and DailyDialog candidate sources.
5 Experimental Evaluation
The evaluation measures empathetic listener responses with automated metrics and human ratings, comparing pretrained, retrieval, fine-tuned, and externally supervised models. Using EmpatheticDialogues candidates or fine-tuning generally improves empathy, often with minimal additional resources, while broader generalization and large-model effects remain qualified.
- Evaluation setup: Human evaluation rates empathy, relevance, and fluency on a five-point Likert scale, while automated evaluation uses BLEU, perplexity, and retrieval accuracy.Human raters specifically judge whether responses acknowledge the conversation partner’s feelings.
- Pretrained models baseline: Pretrained Reddit-based models receive poor empathy ratings; larger BERT or Transformer models improve ratings but remain far below human performance and require more resources.The baseline includes both retrieval and generative systems.
- Using EMPATHETICDIALOGUES for candidate selection: Using EmpatheticDialogues candidates substantially improves pretrained retrieval models on human metrics, especially Empathy.The candidate pool also improves retrieval BLEU scores.
- Using EMPATHETICDIALOGUES for fine-tuning: Fine-tuning on EmpatheticDialogues improves automated metrics and generally improves human metrics for both retrieval and generative systems.On other corpora, fine-tuning raises performance on DailyDialog but slightly decreases it on Reddit, where the pretrained model was trained directly.
- Augmenting conversation models with external pretrained classifiers: Prepending emotion or topic predictions may bring high-capacity BERT-based models close to human Empathy ratings, but more large-model experiments are needed to confirm the effect.The improvement is reported for high-capacity models rather than smaller models.
- Resources and capacity: EmpatheticDialogues adaptation makes smaller models outperform larger ones with minimal additional resource usage.The resource comparison covers retrieval with ED candidates and fine-tuning pretrained conversation models.
6 Conclusion
The paper releases a 25k-dialogue resource grounded in emotion-labeled situations and reports that using it for retrieval candidates or fine-tuning yields responses judged more empathetic. It also identifies integrating empathy with topicality and information provision as an open challenge.
- 6 Conclusion: EmpatheticDialogues contains 25k dialogues grounded in situations prompted by specific emotion labels.The dataset is presented as a resource for empathetic dialogue generation.
- 6 Conclusion: Using EmpatheticDialogues for retrieval candidates or fine-tuning produces responses evaluated as more empathetic.The conclusion summarizes both adaptation strategies as effective within the reported evaluation.
- 6 Conclusion: A remaining challenge is integrating empathetic responding with staying on topic or providing information.The paper presents this as a direction for further research.
A Data Examples
The appendix provides ten randomly selected training-set dialogues as examples of the dataset.
- A Data Examples: Ten randomly selected dialogues from the training set are included as examples.
B Human Evaluation Crowdsourcing Task
Human ratings were collected through a structured Mechanical Turk task in which workers scored empathy, relevance, and fluency for model outputs.
- B Human Evaluation Crowdsourcing Task: Each worker rated ten randomly subsampled test examples from different models for empathy, relevance, and fluency.At least 100 ratings were collected per model from 221 US workers.
C Next utterance prediction on other datasets
Fine-tuning on EmpatheticDialogues improves next-utterance prediction on DailyDialog, while the experiment evaluates transfer using contexts and candidates from external datasets.
- Performance on DAILYDIALOG improves after fine-tuning on EmpatheticDialogues data.The evaluation uses both context and candidate responses from DailyDialog.
- The transfer experiment evaluates next-utterance prediction on REDDIT and DAILYDIALOG.For each dataset, both contexts and candidates come from that same external dataset.
- The models use Adamax throughout, with 0% dropout except 20% in the emotion-label linear layer of the multitask objective.The four-layer Transformer models use an 8e−4 learning rate, while five-layer generative models use 8e−5.
D.2 Additional Experimental Set-Ups
Additional adaptations incorporate emotion or topic supervision through multitask learning, prepended classifier labels, or ensemble encoders. Human-evaluation gains over fine-tuned models appear for some variants but remain inconsistent, while top-1 and top-3 prepending do not improve generative scores.
- Multitask with Emotion labels: The multitask model adds an emotion-classification objective alongside next-utterance prediction.A linear layer and softmax predict the conversation emotion from encoded context, and the objectives average their negative log-likelihoods.
- Prepend-3, Prepend-5: Prepend-3 and Prepend-5 supply the classifier’s top-3 or top-5 predicted emotion or topic labels instead of only the top-1 label.
- Ensemble of Encoders: Ensemble encoders incorporate latent representations from pretrained supervised architectures while freezing the base Transformer encoder and pretrained classifier.The trainable components are the linear layers and, for generative systems, the decoder.
- Additional Experimental Set-Ups: Multitask, prepend-5, and ensemble setups may improve human-rated empathy over fine-tuned generative models, but the evidence is too inconsistent to be conclusive.Top-1 and top-3 prepending do not improve generative model scores.
D.4 Emotion Classification Results
EmpatheticDialogues can support emotion-classifier training or fine-tuning, and its classification results are reported alongside established benchmarks. The broader model comparison covers automatic, human, and resource-based evaluations.
- Emotion Classification: EmpatheticDialogues can be used to train or fine-tune an emotion classifier in the PREPEND-K and ENSEM-DM+ setups.
- Evaluation Scope: The broader experimental comparison includes automatic metrics, human ratings, and training-resource measurements.Table 7 covers automatic evaluation, Table 8 covers human evaluation, and Table 9 reports empathy, relevance, fluency, and resource comparisons.
- Classification Results: Table 10 reports emotion-label classification on the situation descriptions, including ED and ED-CUT conditions alongside Felbo et al. (2017) benchmarks.ED-CUT removes situation descriptions where the target label was present.