Source-linked AI summary
COSMIC: COmmonSense knowledge for eMotion Identification in Conversations
Deepanway Ghosal, Navonil Majumder, Alexander Gelbukh, Rada Mihalcea, Soujanya Poria
TL;DR
Emotion recognition in conversations requires modeling contextual and latent participant information that ordinary approaches may miss. COSMIC represents commonsense mental states, events, intents, and causal relations to model interlocutor interactions, achieving new state-of-the-art results across benchmark datasets. The improvements are consistent across datasets but not very substantial.
Problem
Conversational emotion recognition must infer emotional dynamics from utterances while accounting for contextual cues and latent participant states.
Method
COSMIC extracts continuous commonsense representations and models speaker- and listener-related intents, effects, reactions, mental states, events, and causal relations.
Results
COSMIC achieves new state-of-the-art emotion-recognition results across several benchmark conversational datasets, including scores of 65.28 on IEMOCAP and 51.05 and 58.48 on DailyDialog.
Takeaways & Limitations
Commonsense representations help address emotion-shift detection and confusion between related emotion classes in conversation.
Takeaways & Limitations
Although commonsense knowledge improves performance across datasets, the improvement is not very substantial.
Abstract
from arXiv · showhide
In this paper, we address the task of utterance level emotion recognition in conversations using commonsense knowledge. We propose COSMIC, a new framework that incorporates different elements of commonsense such as mental states, events, and causal relations, and build upon them to learn interactions between interlocutors participating in a conversation. Current state-of-the-art methods often encounter difficulties in context propagation, emotion shift detection, and differentiating between related emotion classes. By learning distinct commonsense representations, COSMIC addresses these challenges and achieves new state-of-the-art results for emotion recognition on four different benchmark conversational datasets. Our code is available at https://github.com/declare-lab/conv-emotion.
1 Introduction
Emotion recognition in conversations is important for fine-grained dialogue understanding, but conversational context, latent participant states, and emotional dynamics make it difficult. COSMIC addresses these challenges by explicitly incorporating commonsense knowledge into emotion identification.
- Emotion recognition supports downstream applications including emotion-aware chat agents, visual question answering, and health conversations.
- Conversational emotions depend on latent speaker states and intents that must be inferred from observable utterances.
- Commonsense knowledge helps model dialogue reasoning, participant interactions, and emotional dynamics.
- Vanilla emotion-recognition approaches overlook contextual cues, turn temporality, and speaker-specific information in conversation.
- COSMIC uses a commonsense-guided framework and evaluations across four conversation datasets to improve emotion identification.
2 Related Work
Conversational emotion recognition builds on broader emotion-recognition research but focuses on contextual modeling of utterances in dialogue. Its recent growth has been enabled by publicly available conversational datasets from social media and scripted settings.
- Emotion recognition has been studied across machine learning, signal processing, and social and cognitive psychology.
- Earlier emotion-recognition systems used facial expressions, acoustic information, and visual cues.
- Conversational emotion recognition gained popularity with publicly available datasets from social media, movies, and television shows.
- The main approach to conversational emotion recognition is contextual modeling in textual or multimodal settings with deep-learning algorithms.
- Related conversational-emotion work includes models and methods from Hazarika, Zadeh, Chen, and other researchers.
3 Methodology
COSMIC models utterance-level conversational emotion by combining RoBERTa utterance features with commonsense representations and participant states. Its architecture uses sequential context modeling, attention, and GRU-based updates for internal, external, and intent states.
- Task formulation: The ERC task predicts each utterance’s emotion label from a conversation transcript and speaker information.
- Feature extraction: COSMIC extracts context-independent utterance features with fine-tuned RoBERTa and commonsense features from COMET.The RoBERTa utterance vector has dimension 1024, while COMET produces five 768-dimensional commonsense vectors per utterance.
- Feature extraction: The five commonsense relations represent speaker intent, speaker effect, speaker reaction, listener effect, and listener reaction.ATOMIC relations cover mental states, events, and causal relations involving speakers and listeners.
- Conversational modeling: COSMIC combines context and commonsense features to model sequential dependencies and interactions among participants.The shared context state and attention vector propagate information across utterances, while participant states capture different mental states, actions, and events.
- Conversational modeling: The framework uses GRU cells to update context, internal, external, and intent states before emotion classification.Internal state captures concealed feelings and perceived effects, while intent state is updated from speaker intent and internal speaker state.
4 Experimental Setup
COSMIC is evaluated on four conversational emotion-recognition datasets spanning two-party and multi-party conversations, using dataset-specific class schemes and metrics. The setup also includes textual inputs and RoBERTa-based feature extraction.
- Evaluation: MELD and EmoryNLP are evaluated with both three-class broad and seven-class fine-grained classification, while Table 3 reports dataset splits and metrics.
- Datasets: COSMIC is benchmarked on IEMOCAP, MELD, DailyDialog, and EmoryNLP.
- Datasets: IEMOCAP contains two-person conversations among ten speakers, with six annotated emotions.
- Datasets: DailyDialog contains seven emotion labels, with neutral utterances comprising over 83% of the dataset and excluded from Micro-F1 evaluation.
- Datasets: MELD contains over 1,400 dialogues and 13,000 utterances from Friends, annotated with seven emotion and three sentiment classes.
- Datasets: EmoryNLP uses seven emotion classes and derives three broader classes by grouping positive and negative emotions.
5 Results and Analysis
COSMIC improves conversational emotion recognition across multiple datasets by incorporating commonsense representations, with particularly strong gains on fine-grained classification and evidence that speaker-specific knowledge is important. A case study suggests commonsense propagation helps handle emotion shifts and distinguish closely related classes, while additional feature strategies provide limited benefit.
- Comparison with the State-of-the-Art Methods: COSMIC improves over all evaluated models on IEMOCAP and DailyDialog, achieving state-of-the-art scores of 65.28, 51.05, and 58.48.The improvement is larger on DailyDialog than on IEMOCAP.
- Comparison with the State-of-the-Art Methods: 73.20 and 56.51 weighted F1 scores are achieved for three-class classification on MELD and EmoryNLP, respectively.For seven-class classification, the corresponding scores are 65.21 and 38.11.
- Comparison with the State-of-the-Art Methods: 65.21 and 38.11 weighted F1 scores are achieved for seven-class classification on MELD and EmoryNLP, respectively.COSMIC brings a large improvement over other models in this fine-grained setup for both datasets.
- The Role of Commonsense: Speaker-specific commonsense contributes more to COSMIC performance than listener-specific commonsense in the ablation studies.Removing speaker-specific components causes a sharp drop in most cases, while listener ablation causes a smaller drop and slightly improves EmoryNLP.
- Case Study: Commonsense propagation helps COSMIC handle sudden emotion transitions and distinguish closely related classes such as angry and frustrated.The case study describes propagation from speaker effects and listener reactions as supporting these predictions.
- Strategies to Incorporate Commonsense: Several additional COMET-derived features do not improve performance by a significant margin, including vectorized speaker attributes and auxiliary predictions of needs and wants.The paper reports that these strategies add complexity without much improvement in emotion classification.
- Strategies to Incorporate Commonsense: The authors report that commonsense improves performance across datasets, but the improvement is not very substantial.They propose seeking better commonsense sources and more efficient integration into deep learning models.
6 Conclusion
COSMIC models commonsense knowledge for conversational emotion recognition by representing mental states, events, actions, and cause-effect relations. The framework alleviates emotion-shift and related-emotion classification difficulties and achieves state-of-the-art results across benchmark datasets.
- COSMIC models mental states, events, actions, and cause-effect relations for emotion recognition in conversations.
- Commonsense representations alleviate difficulties detecting emotion shifts and distinguishing between related emotion classes.
- COSMIC achieves new state-of-the-art results across several benchmark emotion-recognition datasets.