Source-linked AI summary
EmoCaps: Emotion Capsule based Model for Conversational Emotion Recognition
Zaijing Li, Fengxiao Tang, Ming Zhao, Yusen Zhu
TL;DR
ERC research has focused on contextual modeling while giving less attention to representing an utterance's emotional tendency across modalities. The paper proposes Emoformer and EmoCaps to extract and fuse multimodal emotion vectors with sentence vectors, and reports better performance than existing state-of-the-art models on two benchmark datasets.
Problem
Existing ERC work mainly focuses on context modeling, while emotional tendency and multimodal utterance information remain insufficiently represented.
Method
Emoformer extracts emotion vectors from textual, audio, and visual features, and EmoCaps fuses them with sentence vectors before context-based emotion classification.
Results
EmoCaps achieves better performance than existing state-of-the-art models on the MELD and IEMOCAP benchmark datasets.
Takeaways & Limitations
The reported experiments support the paper's hypothesis that modeling multimodal emotional tendency is useful for conversational emotion recognition.
Takeaways & Limitations
Audio features may not distinguish emotions with similar frequency and amplitude, while visual features can be limited when speakers hide their expressions; speaker modeling also consumes more computing resources and time.
Abstract
from arXiv · showhide
Emotion recognition in conversation (ERC) aims to analyze the speaker's state and identify their emotion in the conversation. Recent works in ERC focus on context modeling but ignore the representation of contextual emotional tendency. In order to extract multi-modal information and the emotional tendency of the utterance effectively, we propose a new structure named Emoformer to extract multi-modal emotion vectors from different modalities and fuse them with sentence vector to be an emotion capsule. Furthermore, we design an end-to-end ERC model called EmoCaps, which extracts emotion vectors through the Emoformer structure and obtain the emotion classification results from a context analysis model. Through the experiments with two benchmark datasets, our model shows better performance than the existing state-of-the-art models.
1 Introduction
The paper addresses ERC's limited modeling of utterance-level emotional tendency alongside contextual information by proposing Emoformer and the end-to-end EmoCaps model. EmoCaps fuses multimodal emotion vectors with sentence vectors and achieves the best reported performance on MELD and IEMOCAP.
- Motivation: Existing ERC research emphasizes contextual information modeling but has difficulty extracting grammatical and semantic utterance information and representing emotional tendency.Prior emotion-vector work described in the paper is limited to textual modality, while emotional expression also involves audio and visual cues.
- Motivation: Emotional tendency is described as an offset that gives a neutral utterance an emotional direction and can help explain emotional reversals caused by context.In ERC, the tendency and the final recognized emotion may diverge because contextual information affects classification.
- Proposed model: The paper presents Emoformer as a Transformer-based encoder without a decoder that extracts emotional tendency from modal features.Its extraction uses multi-head self-attention and feed-forward layers.
- Proposed model: Emoformer extracts emotion vectors from textual, audio, and visual modalities and merges them with sentence vectors into an emotion capsule.The resulting capsule is passed to a context analysis model for final emotion classification.
- Experimental outcome: EmoCaps achieves the best performance on both the MELD and IEMOCAP datasets against existing state-of-the-art models.The comparison covers multimodal and text-modality settings under the reported benchmark evaluation.
2 Related Work
Related work covers contextual ERC models, multimodal emotion-recognition approaches, and Transformer-based representation learning. These studies provide precedents for speaker-aware context modeling, multimodal fusion, and long-distance contextual representation.
- Emotion Recognition in Conversation: ERC research has modeled conversational context with recurrent, graph-based, commonsense, directed-acyclic, and cognitive-context architectures.The cited approaches differ in how they represent speaker states, interlocutor interactions, conversation graphs, and contextual dependencies.
- Multi-modal Emotion Recognition: Multimodal emotion-recognition research has used tensor outer products, multi-level attention, hierarchical fusion, and graph convolutional networks.Some cited multimodal methods are not applied to ERC, while CMN and MMGCN explicitly address multimodal conversational emotion recognition.
- Transformer: Transformer research introduced self-attention-based representations that obtain long-distance contextual information without sequence-based recurrence.The paper situates Transformer applications across computer vision, audio processing, and language modeling.
3 Methodology
EmoCaps defines ERC over multimodal utterances and combines modality-specific emotional tendency vectors with sentence representations before contextual classification. Its Emoformer uses Transformer-style attention and feed-forward processing, while a Bi-LSTM analyzes the resulting emotion capsules.
- Problem definition: ERC maps each utterance in a dialogue to an emotion label from a predefined emotion-label set.A dialogue is represented as u1, u2, ..., un, with one classification assigned to each utterance.
- Unimodal feature extraction: Text features come from BERT, audio features from OpenSMILE acoustic descriptors, and visual features from a 3D-CNN capturing facial-expression changes.BERT produces 512-dimensional sentence vectors; the visual extractor also produces a 512-dimensional vector after fully connected layers.
- EmoCaps overview: EmoCaps extracts textual, audio, and visual features, obtains modality-specific emotion vectors with Emoformer, fuses them with the sentence vector, and classifies emotions contextually.The model processes the fused emotion capsules with a context analysis model.
- Emoformer block: Emoformer applies Transformer-style multi-head self-attention, normalization, feed-forward processing, residual connections, and a mapping network to produce emotion vectors.The mapping network consists of five fully connected layers, and the resulting vectors represent emotional tendencies for different modalities.
- Emotion capsule: The emotion capsule combines sentence semantics with modality-specific emotional cues because audio and visual features can supplement text when its emotional inclination is insufficient.The fused capsule is passed to contextual analysis to determine the speaker’s emotion.
- Context modeling: A bidirectional LSTM models each dialogue’s sequence of emotion capsules, using preceding and following utterances to derive contextual information for final classification.The forward direction traces emotional clues from previous moments, while the backward direction supports reasoning from subsequent moments.
4 Experiment Setting
The experiments evaluate EmoCaps on IEMOCAP and MELD using established ERC baselines, specified training settings, and multimodal configurations.
- Datasets: IEMOCAP contains 7,433 utterances in 151 dialogues with text and audio, while MELD contains 13,708 utterances in 1,433 conversations from Friends.
- Baseline Models: The baseline set includes recurrent, graph-based, transformer-based, and multimodal context models such as DialogueRNN, DialogueGCN, DialogXL, and MMGCN.
- Feature Extraction: Textual features use BERT, while audio and visual features use Emoformer to obtain modality-specific emotion vectors.
- Parameter Settings: Both datasets use 80 epochs, a 0.0001 learning rate, and a 0.1 dropout rate, following Li et al. (2021).
- Evaluation Settings: Table 4 reports EmoCaps F1 scores under textual, audio, visual, and combined multimodal settings.
5 Results and Analysis
EmoCaps achieves the best reported performance on both benchmark datasets, while analyses show that multimodal combinations and emotion vectors contribute unevenly across settings.
- 5.1 Compare with Other Baseline Models: EmoCaps achieves the best performance on both IEMOCAP and MELD under the same parameter conditions as the compared state-of-the-art models.
- 5.2 Various Modality: Textual input performs better than audio or visual input individually, and multimodal input outperforms single-modal input on both datasets.
- 5.3 Error Analysis: Audio features may confuse emotions with similar frequency and amplitude, while hidden facial expressions limit visual-only emotion recognition.
- 5.2 Various Modality: Adding textual modality significantly improves performance, while audio and visual modalities provide additional accuracy gains.
- 5.4 Impact of Speaker Embedding: DialogueRNN-based context modeling performs better than LSTM on MELD, but speaker modeling is insignificant on the two-person IEMOCAP dialogues and costs more resources.
- 5.3 Error Analysis: Emotion vectors capture contextual emotional tendency when sentence vectors remain neutral, helping explain emotion reversals in the dialogue example.
6 Conclusion
The paper concludes that Emoformer extracts multimodal emotion vectors for EmoCaps, whose emotion capsule combines them with sentence vectors before context-based classification.
- EmoCaps combines textual, audio, and visual emotion vectors with sentence vectors into an emotion capsule, then uses context analysis for emotion recognition.
- Experiments on two benchmark datasets show better performance than existing state-of-the-art models and support the proposed hypothesis.