Source-linked AI summary
DialogueRNN: An Attentive RNN for Emotion Detection in Conversations
Navonil Majumder, Soujanya Poria, Devamanyu Hazarika, Rada Mihalcea, Alexander Gelbukh, Erik Cambria
TL;DR
Conversational emotion detection needs to account for the speaker-specific and temporal nature of utterances, but existing systems do not meaningfully distinguish parties. DialogueRNN uses recurrent networks to track party states and conversational context for classification, outperforming state-of-the-art baselines on two datasets. Its main remaining weakness is emotion-shift turns, where performance is substantially lower than on turns without such shifts.
Problem
Existing conversational emotion systems do not meaningfully distinguish parties, despite emotion depending on speaker, context, and preceding emotions.
Method
DialogueRNN uses interconnected GRUs to track party states, shared conversational context, and emotion representations for utterance classification.
Results
DialogueRNN outperforms baseline methods, including state-of-the-art CMN, on both datasets on average.
Takeaways & Limitations
Speaker-aware party states provide finer utterance context, and the architecture is designed to scale to conversations with more than two speakers.
Takeaways & Limitations
Emotion-shift turns remain difficult: the model correctly predicts 47.5% of shifts versus 69.2% of turns without an emotional shift.
Abstract
from arXiv · showhide
Emotion detection in conversations is a necessary step for a number of applications, including opinion mining over chat history, social media threads, debates, argumentation mining, understanding consumer feedback in live conversations, etc. Currently, systems do not treat the parties in the conversation individually by adapting to the speaker of each utterance. In this paper, we describe a new method based on recurrent neural networks that keeps track of the individual party states throughout the conversation and uses this information for emotion classification. Our model outperforms the state of the art by a significant margin on two different datasets.
1 Introduction
DialogueRNN addresses speaker-insensitive conversational emotion detection by tracking individual party states alongside utterance context and prior emotions. Its recurrent architecture models these factors through interconnected GRUs for emotion classification.
- Conversational emotion detection supports applications including opinion mining over chat history and social media threads.
- Existing systems do not meaningfully distinguish conversation parties or identify the speaker of each utterance.
- DialogueRNN models speaker, preceding context, and preceding emotions using individual party states updated as the conversation unfolds.
- Three GRUs update global context, party state, and emotion representation for classification.
- The global and emotion GRUs model inter-party relations, while the party GRU models sequential states of the same party.
2 Related Work
Prior work spans multimodal and contextual emotion recognition, while conversational emotion is framed as an interpersonal, temporally structured phenomenon. DialogueRNN builds on this framing by incorporating party interactions and recurrent temporal modeling.
- Emotion recognition research has used facial, acoustic, visual, textual, and contextual information across several fields and settings.
- Emotional dynamics in conversation are described as an interpersonal phenomenon.
- DialogueRNN incorporates interpersonal interactions and the temporal nature of conversation through recurrent networks.
- Memory networks have been applied to emotion recognition in dyadic conversations, where two distinct memory networks enable inter-speaker interaction.
3.1 Problem Definition
The task is to predict emotion labels for each utterance in a conversation while representing which party produced each utterance. The formulation uses M parties, with M = 2 for the datasets used.
- The task predicts happy, sad, neutral, angry, excited, or frustrated for each constituent utterance.
- Each utterance is associated with its speaking party through a mapping from utterances to party indices.
- The formulation allows M parties, with M = 2 for the datasets used.
3.2 Unimodal Feature Extraction
The paper uses standardized feature extraction procedures for fair comparison with CMN, combining CNN-based textual representations with 3D-CNN visual and openSMILE acoustic features.
- Feature extraction follows procedures identical to CMN for a fair comparison.
- A CNN extracts textual utterance representations from 3-, 4-, and 5-gram features using 50 feature-maps per filter size.
- The textual CNN applies max-pooling and ReLU activation before concatenating outputs into a 100-dimensional dense representation.
- Visual and acoustic features are extracted with 3D-CNN and openSMILE, respectively.
3.3 Our Model
DialogueRNN models conversational emotion through party-specific states, shared context, and preceding emotions. GRU-based updates produce an emotion representation that is classified with a softmax layer.
- DialogueRNN assumes utterance emotion depends on the speaker, preceding context, and preceding emotions.
- Speaker-state modeling: Each party has a recurrent state, while a shared global state jointly encodes preceding utterances and party states for context.
- Speaker-state modeling: GRU cells update the global, party, and emotion representations recurrently from current utterances and relevant previous states.The global update uses the current utterance and the current speaker’s previous party state.
- Emotion Representation: The emotion representation combines the current speaker state with the previous emotion representation to model links among parties’ preceding states.The emotion GRU is described as connecting the speaker state with information from other party states.
- Emotion Classification: A two-layer perceptron and softmax layer convert the emotion representation into probabilities over 6 emotion classes and select the most likely label.Training uses categorical cross-entropy with L2 regularization and Adam optimization.
3.4 DialogueRNN Variants
The paper uses DialogueRNN as the basis for subsequent model variants.
- DialogueRNN serves as the basis for the following model variants.
DialogueRNN + Listener State Update (DialogueRNNl):
DialogueRNN_l updates listener states from the resulting speaker state, while bidirectional and attention variants incorporate broader dialogue context.
- DialogueRNN + Listener State Update: DialogueRNN_l updates each listener state using the resulting speaker state for the current utterance.
- Bidirectional DialogueRNN: BiDialogueRNN runs separate forward and backward DialogueRNNs so emotion representations include past and future utterances.The two directional outputs are concatenated at sequence level.
- Emotional Attention: Attention over BiDialogueRNN emotion representations selects context from relevant preceding and future utterances before classification.The attended representations are then passed to the two-layer perceptron.
4 Experimental Setting
DialogueRNN is evaluated on IEMOCAP and AVEC using speaker-disjoint splits and compared with contextual, noncontextual, and multimodal baselines.
- Datasets Used: DialogueRNN is evaluated on IEMOCAP and AVEC with train/test partitions of roughly 80/20 that do not share speakers.Table 1 reports the dataset split.
- Datasets Used: IEMOCAP contains dyadic conversations from ten speakers, with utterances labeled as happy, sad, neutral, angry, excited, or frustrated.
- Datasets Used: AVEC contains human-agent interactions annotated with valence, arousal, expectancy, and power, averaged over each utterance.The attributes are real-valued and originate from annotations available every 0.2 seconds.
- Baselines: The evaluation compares DialogueRNN with baseline methods including contextual speaker-aware CMN, context-aware c-LSTM, and models without surrounding-utterance context.The baselines also include multimodal TFN and MFN, while the primary evaluation uses textual modality with additional multimodal experiments.
- Experimental Scope: The study primarily evaluates textual features and additionally experiments with multimodal features to examine performance in a multimodal setting.
5 Results and Discussion
DialogueRNN generally outperforms CMN and other baselines across textual and multimodal emotion benchmarks. Its party-aware recurrent structure, attention mechanisms, and emotion-state modeling support stronger performance, while emotion shifts remain difficult.
- Comparison with baselines: DialogueRNN outperforms all baseline methods, including CMN, on average across both textual datasets.The comparison uses textual modality results in Table 2.
- Comparison with the State of the Art: 2.77% accuracy and 3.76% f1-score are the average IEMOCAP gains over CMN.DialogueRNN surpasses CMN in five of six emotion classes, but trails by 1.23% f1-score for frustrated.
- Comparison with the State of the Art: DialogueRNN outperforms CMN on valence, arousal, expectancy, and power in AVEC.It achieves lower MAE and higher Pearson correlation coefficient (r) for all four attributes.
- DialogueRNN variants: Explicit listener-state updates slightly worsen general performance, whereas BiDialogueRNN improves over DialogueRNN and attention over both past and future context improves further.The listener-state exception is happy on IEMOCAP, where DialogueRNNl gains 1.71% f1-score; DialogueRNN+Attn improves over BiDialogueRNN by 1.23% f1-score on IEMOCAP.
- DialogueRNN variants: 6.62% higher f1-score than CMN and 2.86% higher f1-score than vanilla DialogueRNN are achieved by the best-attention setting on IEMOCAP.The same setting performs best overall on both datasets and across all four AVEC attributes.
- Attention analysis: Focused attention highlights relevant self- and inter-party context, while approximately 18% of correctly predicted IEMOCAP utterances depend on context 20–40 turns away.The attention analysis associates distant context with long-term emotional dependencies.
- Error analysis: Emotion-shift regions are substantially harder than stable regions: the model correctly predicts 47.5% of shifts versus 69.2% of non-shifts.Errors also cluster among related emotions such as happy–excited and anger–frustrated.
- Ablation Study: Removing party state lowers performance by 4.33%, while removing the emotion GRU lowers it by 2.51% on IEMOCAP.The ablation study identifies party state as the more impactful of these two components.
6 Conclusion
DialogueRNN is an RNN architecture that incorporates each speaker’s characteristics when processing utterances. It outperforms the current state of the art on two datasets in textual and multimodal settings.
- DialogueRNN treats each incoming utterance using the speaker’s characteristics, providing finer context.
- The model outperforms the current state of the art on two distinct datasets.
- DialogueRNN achieves this result in both textual and multimodal settings.