Source-linked AI summary
Emotion Recognition in Conversation: Research Challenges, Datasets, and Recent Advances
Soujanya Poria, Navonil Majumder, Rada Mihalcea, Eduard Hovy
TL;DR
ERC must identify emotions in utterances while modeling conversational context, speaker information, and changing interpersonal dynamics. This paper surveys the task, research challenges, datasets, benchmarks, and recent advances, finding that current models benefit from contextual history but remain weak on emotional shifts. It concludes that emotion-shift recognition and stronger context encoders are important directions, alongside speaker-specific, multiparty, and sarcasm-focused ERC.
Problem
ERC requires context, speaker information, and temporal modeling because vanilla utterance-level emotion recognition ignores conversation-specific factors.
Method
The paper synthesizes ERC research by discussing its task definition, emotion dynamics, research challenges, datasets, benchmarks, recent models, and shortcomings.
Results
47.5% versus 69.2%: DialogueRNN correctly predicts utterances with emotion shifts less often than utterances without emotional shifts.
Takeaways & Limitations
Effective emotion-shift recognition and context encoding may improve chit-chat dialogue and some aspects of task-oriented dialogue, while speaker-specific, multiparty, and sarcasm-focused ERC remain research directions.
Takeaways & Limitations
Current ERC networks perform poorly on utterances involving emotion shifts, particularly when the target emotion differs from the previous utterance.
Abstract
from arXiv · showhide
Emotion is intrinsic to humans and consequently emotion understanding is a key part of human-like artificial intelligence (AI). Emotion recognition in conversation (ERC) is becoming increasingly popular as a new research frontier in natural language processing (NLP) due to its ability to mine opinions from the plethora of publicly available conversational data in platforms such as Facebook, Youtube, Reddit, Twitter, and others. Moreover, it has potential applications in health-care systems (as a tool for psychological analysis), education (understanding student frustration) and more. Additionally, ERC is also extremely important for generating emotion-aware dialogues that require an understanding of the user's emotions. Catering to these needs calls for effective and scalable conversational emotion-recognition algorithms. However, it is a strenuous problem to solve because of several research challenges. In this paper, we discuss these challenges and shed light on the recent research in this field. We also describe the drawbacks of these approaches and discuss the reasons why they fail to successfully overcome the research challenges in ERC.
I. INTRODUCTION
Emotion recognition in conversation (ERC) identifies each utterance’s emotion while accounting for conversational context, speaker information, and interacting pragmatic variables. The paper frames ERC as an emerging NLP field with applications in conversation analysis and emotion-aware dialogue generation, then surveys its challenges, datasets, and recent progress.
- Motivation: ERC has applications in social-media analysis, real-time conversation analysis, opinion mining, recommender systems, health-care, and emotion-aware dialogue generation.The paper connects ERC to psychological analysis, student frustration, and dialogue systems that need to understand users’ emotions.
- Motivation: Unlike sentence-level emotion recognition, ERC must model preceding utterances, temporal turn order, contextual cues, and speaker-specific information.Lexicon-based and deep learning approaches that ignore these conversation-specific factors perform poorly on ERC datasets.
- Task definition: ERC predicts a predefined emotion label for every utterance using the conversation transcript and speaker information.The input is a sequence of utterances paired with their speakers, and the output is one emotion label per utterance.
- Conversation factors: Conversation dynamics reflect topic, personality, interlocutor state, intent, emotion, and utterances, with pragmatic factors shaping speakers’ responses.The framework distinguishes hidden and observed variables in dyadic interaction and can extend to multiparty conversations.
- Conversation factors: Self-dependency and interpersonal influence create emotional dynamics in which speakers maintain, induce, or mirror emotional states during dialogue.These dependencies help explain how one participant’s response can intensify another participant’s emotion.
- Scope of the paper: The paper surveys ERC’s research challenges, datasets, benchmarks, and recent advances to clarify shortcomings and suggest future directions.Its stated aim is to support researchers’ understanding of the field and identify possible research directions.
II. RESEARCH CHALLENGES
ERC faces challenges in representing emotion categories, conversational context, speaker and listener dynamics, emotion shifts, fine-grained meaning, multiparty interaction, sarcasm, reasoning, and annotation. These difficulties arise because emotions depend on discourse, personal characteristics, evolving interactions, and the limitations of available taxonomies and datasets.
- Categorization of emotions: ERC must choose between categorical and dimensional emotion models, each offering different representations and comparison properties.Categorical models use discrete labels, whereas dimensional models represent valence and arousal continuously, making vector-based comparison easier.
- Categorization of emotions: Available datasets use inconsistent taxonomies, and simpler labels or short contexts can make ERC less challenging while limiting complexity.IEMOCAP uses both categorical and dimensional models, whereas DailyDialogue and EmoContext use categorical labels; EmoContext reports higher agreement but reduced difficulty.
- Emotional dynamics: Emotional dynamics combine self-dependency, or emotional inertia, with interpersonal influence between conversation participants.Speakers may maintain their emotional state, mirror counterparts, or change in response to others; these patterns also interact with topic, personality, intent, and discourse structure.
- Conversational context modeling: Context modeling must capture local and distant history because short utterances can express different emotions depending on discourse.Distant context is especially useful when speakers refer to earlier utterances, while local context is generally more important.
- Speaker and listener modeling: Speaker-specific modeling is needed because sarcasm and emotional expression vary across individuals, while listener modeling may require visual reactions and is most useful for continuous recognition.Sarcasm also depends on conversational context and discourse, and listener reactions may already be conveyed by subsequent utterances.
- Advanced ERC challenges: Emotion shifts, fine-grained meanings, multiparty structure, sarcasm, and emotion reasoning remain difficult because they require tracking changing states, perspectives, relationships, and causes.Emotion reasoning differs from context modeling by determining how contextual utterances function in producing the target emotion; suitable richly annotated datasets are unavailable.
III. DATASETS
The paper surveys publicly available ERC datasets and their annotation schemes, emphasizing both their diversity and important limitations for emotion reasoning.
- ERC datasets include IEMOCAP, SEMAINE, EmotionLines, MELD, DailyDialog, and EmoContext.
- SEMAINE uses continuous valence, arousal, expectancy, and power annotations, whereas other datasets primarily use categorical emotion labels.SEMAINE values are real-valued; its ranges include valence, arousal, and expectancy in [−1,1], and power in [0,∞).
- EmoContext assigns an emotion label only to the final utterance of each dialogue.
- Available datasets lack the annotation detail needed for emotion reasoning and do not provide fine-grained, topic-level emotion annotations.
IV. RECENT ADVANCES
Recent ERC advances model conversational context, speaker-specific states, and inter-speaker influence, but performance remains limited for emotion shifts, long-range context, and multiparty settings.
- Benchmarks and their drawbacks: DialogueRNN uses a hierarchical multi-stage RNN with attention and speaker information to model self- and inter-speaker emotional influence.
- Benchmarks and their drawbacks: ERC research compares approaches that model contextual history, speaker-specific states, and self- and inter-speaker emotional influence.The surveyed approaches include CMN, ICON, IANN, and DialogueRNN, alongside context-generation methods using RNNs.
- Benchmarks and their drawbacks: Nearby utterances are generally more context-rich, while future utterances can improve ERC performance but are unavailable in real-time applications.For real-time use, the paper identifies fixed-context-window variants of CMN, ICON, and DialogueRNN as suitable alternatives.
- Benchmarks and their drawbacks: 47.5% is DialogueRNN’s success rate when the target emotion differs from the previous utterance, versus 69.2% without an emotional shift.The surveyed CMN, ICON, IANN, and DialogueRNN networks all perform poorly on emotion-shift utterances.
- Benchmarks and their drawbacks: Only DialogueRNN handles large-scale multiparty conversations, yet it yields little improvement over bc-LSTM on MELD.CMN and ICON target dyadic dialogues, and adapting them to multiparty data may create scalability issues when test-time speaker counts exceed training-time counts.
- Benchmarks and their drawbacks: RNN-based context representations perform poorly at capturing long-distance contextual information.
- EmoContext advances: EmoContext contains three-utterance conversations and labels only the third utterance, making speaker information potentially inconsequential because of its short context history.
- EmoContext advances: HRLCE combines an utterance encoder using ELMo, GloVe, and Deepmoji with a bc-LSTM-based context encoder and multi-head attention.HRLCE was applied only to EmoContext, although the paper states that it can be adapted to other ERC datasets.
V. CONCLUSION
The paper summarizes ERC challenges, datasets, and recent advances while identifying emotion-shift recognition and context encoding as important directions. It also highlights topic-level, speaker-specific, multiparty, sarcasm, and continuous-emotion challenges for future work.
- Effective emotion-shift recognition and context encoders are identified as promising routes for improving conversational emotion recognition.
- Future research directions include topic-level speaker-specific emotion recognition, multiparty ERC, conversational sarcasm detection, and fine-grained continuous emotion tracking.
- Addressing the outlined challenges is expected to enhance AI-enabled conversation understanding and dialogue-system performance using affective information.