Source-linked AI summary
MoEL: Mixture of Empathetic Listeners
Zhaojiang Lin, Andrea Madotto, Jamin Shin, Peng Xu, Pascale Fung
TL;DR
Empathetic dialogue systems need to understand user emotions rather than merely generate emotional responses, while existing approaches have important assumptions. MoEL tracks emotions, combines emotion-specialized listeners, and generates responses; human evaluations report stronger empathy and relevance, with analyses indicating interpretability.
Problem
Existing empathetic dialogue approaches can use a single decoder without explicit reaction specialization or assume the appropriate conditioning emotion is given, although that emotion is often unknown.
Method
MoEL encodes dialogue context to estimate an emotion distribution, then softly combines outputs from listeners optimized for different emotions to generate a response.
Results
Human evaluations report that MoEL outperforms multitask and other baseline settings in empathy and relevance, while listener analyses show appropriate listener selection and emotion-specific reactions.
Takeaways & Limitations
MoEL provides an end-to-end empathetic response model with an interpretable generative process based on automatically selecting an emotional decoder.
Takeaways & Limitations
The work identifies persona integration and task-oriented dialogue integration as future extensions for more consistent, personalized, and versatile responses.
Abstract
from arXiv · showhide
Previous research on empathetic dialogue systems has mostly focused on generating responses given certain emotions. However, being empathetic not only requires the ability of generating emotional responses, but more importantly, requires the understanding of user emotions and replying appropriately. In this paper, we propose a novel end-to-end approach for modeling empathy in dialogue systems: Mixture of Empathetic Listeners (MoEL). Our model first captures the user emotions and outputs an emotion distribution. Based on this, MoEL will softly combine the output states of the appropriate Listener(s), which are each optimized to react to certain emotions, and generate an empathetic response. Human evaluations on empathetic-dialogues (Rashkin et al., 2018) dataset confirm that MoEL outperforms multitask training baseline in terms of empathy, relevance, and fluency. Furthermore, the case study on generated responses of different Listeners shows high interpretability of our model.
1 Introduction
Empathetic dialogue requires recognizing a user's emotion and responding appropriately, but existing approaches either rely on a single decoder or assume the appropriate emotion is given. MoEL addresses this with emotion-specific listeners whose outputs are softly combined, outperforming baselines in human evaluations and supporting interpretability.
- Motivation: Empathetic dialogue agents must recognize users’ emotions and respond with an appropriate emotion, a capability that remains challenging to train.The empathetic-dialogues examples center on understanding a speaker’s feelings and replying accordingly.
- Related approaches: Prior work either jointly predicts user emotion and generates a response or conditions generation on a fixed emotion.These are the two main lines of empathetic dialogue response generation identified in the introduction.
- Research gap: A single decoder can lose interpretability and encourage generic responses, while fixed-emotion conditioning assumes an emotion that may be unknown.These limitations concern the assumptions behind the two existing approaches.
- Proposed model: MoEL tracks the user’s emotion, uses emotion-specialized listeners, and softly combines their decoder states through a Meta-listener to generate the response.The listeners are optimized to react to particular context emotions, while the combination follows the emotion classification distribution.
- Results: Human evaluations show that MoEL outperforms competitive baselines in empathy and relevance, while listener analysis supports a more interpretable generation process.The analysis reports attention to the appropriate listener and emotion-specific listener reactions.
2 Related Work
Related work spans open-domain, personalized, retrieval-augmented, emotional, empathetic, and mixture-of-experts dialogue systems. MoEL draws specifically on specialized-expert architectures to model reactions associated with different emotions.
- Conversational models: Open-domain conversational research has studied scalable conversation models, persona-conditioned personalization, and retrieval-generation combinations for response diversity.PersonaChat and ConvAI 2 support personalized responses, while other work combines retrieval with generation.
- Emotion and empathy: Fewer dialogue-system studies have focused specifically on emotion and empathy, although several works control the sentiment or emotion of generated responses.The cited emotional-dialogue approaches introduce frameworks for controlling generated sentiment and emotion.
- Mixture of Experts: Mixture-of-experts research uses specialized parameters or experts across architectures including SVMs, Gaussian processes, Dirichlet processes, and hierarchical experts.The paper situates its approach within a broad literature on independent specialized components.
- Mixture of Experts: Recent mixture-of-experts models place many experts between LSTM layers, inspiring MoEL to use separate experts for reactions to different emotions.This connection motivates modeling each emotion with an independent specialized listener.
3 Mixture of Empathetic Listeners
MoEL tracks the speaker’s emotional state and uses that distribution to combine specialized listeners before generating an empathetic response. Its architecture jointly models emotion recognition, listener selection, and response generation, with evaluation emphasizing empathy, relevance, and fluency.
- MoEL represents dialogue as alternating speaker and listener utterances and aims to track the current speaker emotion before generating a response.
- Embedding: The context embedding sums word, positional, and dialogue-state embeddings so the encoder can distinguish tokens and speaker turns.
- Emotion Tracker: The emotion tracker encodes the dialogue context with a Transformer encoder and produces a representation used to generate the emotion distribution.A query token is added to the flattened context sequence to compute the context representation.
- Emotion Aware Listeners: Emotion-aware listeners include one shared listener and independently parameterized Transformer decoders specialized to particular emotional states.The shared listener provides a general representation, while other listeners learn emotion-specific responses.
- Emotion Aware Listeners: A key-value memory network uses the encoder query to score listener representations, and those scores weight their contribution to the combined output.During training, listener weights are supervised using the speaker’s emotion state.
- Meta Listener: The Meta Listener transforms the combined listener representation into the final response, while joint training optimizes listener selection and response generation.The evaluation compares MoEL with baseline models using response metrics and human A/B tests.
4 Experiment
The experiment uses the empathetic-dialogues dataset and compares MoEL with Transformer and multitask Transformer baselines through automatic and human evaluation.
- Dataset: The empathetic-dialogues dataset contains 25k one-to-one open-domain conversations grounded in emotional situations, with 32 evenly distributed emotion labels.
- Baselines: MoEL is compared against a standard Transformer trained with MLE and a multitask Transformer incorporating supervised emotion information.
- Automatic evaluation: BLEU is reported only as a reference because it is not considered a good measure of open-domain dialogue generation quality.
- Human evaluation: Human evaluators score empathy, relevance, and fluency independently on a 1-to-5 scale for sampled model responses.The evaluation samples 100 dialogues and assigns three annotators to each response.
- Human evaluation: A separate A/B test compares MoEL responses directly with each baseline, allowing judges to select either response or a tie.The task uses 100 dialogues per comparison and three workers per item.
5 Results
MoEL detects user emotions and uses listener specialization to generate responses, outperforming the baselines on empathy and relevance while supporting interpretable listener behavior.
- Emotion detection: 38%, 63%, and 74% are MoEL’s top-1, top-3, and top-5 emotion-detection accuracies over 32 emotions.The majority of emotions reach around 80% accuracy with top-5 detection.
- Response evaluation: MoEL achieves the highest empathy and relevance scores, while TRS achieves the highest BLEU and fluency scores.The models receive close BLEU scores overall.
- Response analysis: Table 4 compares TRS, Multi-TRS, and MoEL on two user emotion states and compares responses from different listeners.The listener comparison uses hard attention on Terrified, Sad, Excited, and Proud listeners.
- Response evaluation: Human A/B tests confirm that human judges prefer responses from MoEL over the baseline responses.
6 Analysis
MoEL is analyzed through response comparisons, listener-specific behavior, and emotion-distribution visualizations. These analyses show emotion-sensitive responses and listener mixtures, while exposing failures when emotion weights are confused.
- Model response comparison: MoEL captures speaker emotions more appropriately than the baselines in example responses, including comforting a disappointed speaker after an interview failure.In another example, it identifies disgust about cleaning dog poop rather than producing generic alternatives such as “horrible” or “scary”.
- Visualization of Emotion Distribution: Figure 4 visualizes attention over 32 listeners for each context and generated response, showing how MoEL selects emotional decoders.The visualization examines the emotion distribution p used to choose listeners.
- Listener analysis: Different listeners generate empathetic and relevant responses when their learned emotional reactions match the speaker’s emotion.The sad listener comforts sad speakers, while excited and proud listeners share positive emotions.
- Listener analysis: Emotionally mismatched listeners produce inappropriate responses, although a sad listener can still offer a meaningful encouragement when earlier context conveys sadness.An excited listener may remain relevant but unsympathetic in the same example.
- Visualization of Emotion Distribution: MoEL can combine multiple listeners to address multiple emotions, such as annoyance and fear after a neighbor is shot.Other emotions receiving non-negligible weights can smooth the distribution and confuse the meta-listener, impairing response generation.
7 Conclusion & Future Work
MoEL generates empathetic dialogue by tracking user feelings and learning emotion-specific listeners, achieving competitive performance with greater interpretability. The authors identify persona and task-oriented dialogue integration as future extensions.
- Conclusion: MoEL tracks user feelings and uses specific listeners for each emotion to generate empathetic dialogue responses.It is benchmarked on the empathetic-dialogues multi-turn, open-domain corpus grounded in emotional situations.
- Conclusion: MoEL achieves competitive performance on empathetic-dialogues while offering greater interpretability than conventional models.The authors also report automatic selection of the correct emotional decoder and effective empathetic response generation.
- Future Work: Future work includes combining MoEL with persona and task-oriented dialogue systems for more consistent, personalized, and versatile conversations.The proposed extensions are intended to add personalization and task-oriented conversational capabilities.
A Supplemental Material
Table 5 presents generated responses from TRS, Multi-TRS, and MoEL across different user emotion states, with goal responses provided for reference.
- Supplemental Material: Table 5 compares generated responses from TRS, Multi-TRS, and MoEL across different user emotion states.Reference goal responses are included for comparison.