Source-linked AI summary
MMGCN: Multimodal Fusion via Deep Graph Convolution Network for Emotion Recognition in Conversation
Jingwen Hu, Yuchen Liu, Jinming Zhao, Qin Jin
TL;DR
ERC needs better ways to model multimodal dependencies and long-distance conversational context beyond text-focused modeling or feature concatenation. MMGCN uses a multimodal fused graph convolutional network with speaker information, and it significantly outperforms other models on IEMOCAP and MELD under multimodal settings.
Problem
ERC research has limited modeling of multimodal dependencies and long-distance context because prior work focuses mainly on text or concatenates multimodal features.
Method
MMGCN constructs modality-specific fully connected graphs, links corresponding utterances across modalities, injects speaker embeddings, and applies deep spectral-domain graph convolution.
Results
MMGCN significantly outperforms other models on IEMOCAP and MELD under the multimodal conversation setting.
Takeaways & Limitations
MMGCN provides a more effective way to utilize multimodal and long-distance contextual information while modeling intra-speaker and inter-modality dependencies.
Abstract
from arXiv · showhide
Emotion recognition in conversation (ERC) is a crucial component in affective dialogue systems, which helps the system understand users' emotions and generate empathetic responses. However, most works focus on modeling speaker and contextual information primarily on the textual modality or simply leveraging multimodal information through feature concatenation. In order to explore a more effective way of utilizing both multimodal and long-distance contextual information, we propose a new model based on multimodal fused graph convolutional network, MMGCN, in this work. MMGCN can not only make use of multimodal dependencies effectively, but also leverage speaker information to model inter-speaker and intra-speaker dependency. We evaluate our proposed model on two public benchmark datasets, IEMOCAP and MELD, and the results prove the effectiveness of MMGCN, which outperforms other SOTA methods by a significant margin under the multimodal conversation setting.
1 Introduction
ERC identifies and tracks speakers’ emotional status during dialogue, where context includes preceding utterances, conversational timing, and speaker information. MMGCN addresses limitations in jointly capturing multimodal and long-distance conversational dependencies.
- ERC identifies and tracks speakers’ emotional status during dialogue for applications including conversation analysis, e-health services, and emotional human-computer interaction.
- Unlike isolated-utterance emotion recognition, ERC requires modeling context from preceding utterances, conversation-turn temporality, and speaker information.
- Long-distance context helps predict emotions, but previous models struggled to capture speaker and long-distance dialogue information simultaneously in multi-speaker conversations.
- MMGCN constructs fully connected modality graphs, links corresponding cross-modal utterance nodes, injects speaker embeddings, and uses deep spectral-domain graph convolution.
- MMGCN is evaluated on IEMOCAP and MELD, significantly outperforming other models on both multimodal benchmark datasets.
2 Related Work
ERC research has expanded with conversational datasets and increasingly models contextual and speaker information. Related multimodal approaches often concatenate modality features, while graph convolutional networks provide a framework for non-Euclidean relational data.
- Open-source conversation datasets including IEMOCAP, AVEC, and MELD have supported growing research attention to ERC.
- Prior ERC models capture interaction history, speaker-specific context, or speaker and sequential relations using LSTMs, memory networks, GRUs, and DialogueGCN.
- Most recent ERC studies emphasize text, while multimodal methods commonly concatenate acoustic, visual, and textual features without modeling cross-modal interaction.
- Graph convolutional networks address non-Euclidean data and are broadly categorized into spectral-domain and non-spectral-domain methods.
3 Method
MMGCN models multimodal conversational context with a deep graph that links utterances within modalities and across modalities, while incorporating speaker information. Its pipeline encodes modality-specific context, constructs and learns the graph, fuses utterance features, and predicts emotion labels.
- 3.1 Modality Encoder: Each utterance provides acoustic, visual, and textual data, and the task predicts its emotional status using all three modalities.
- 3.1 Modality Encoder: A bidirectional LSTM encodes sequential textual context, while fully connected networks encode acoustic and visual features.
- 3.2 Multimodal Fused Graph Convolutional Network: MMGCN uses a spectral-domain deep graph convolutional network to encode multimodal utterance dependencies and learned speaker embeddings to encode speaker-level context.
- 3.2.2 Graph Construction: The dialogue graph contains 3N modality-specific utterance nodes, connects all nodes within each modality, and links nodes representing the same utterance across modalities.
- 3.2.2 Graph Construction: Same-modality and cross-modality edges use distinct angular-similarity weighting strategies, with a hyperparameter scaling cross-modality interactions.
- 3.2.3 Graph Learning and Emotion Classification: Residual connections and adaptive weight decay support deeper GCN stacking, after which modality features are concatenated and passed to an MLP emotion classifier.
4 Experiment Setups
The experiments evaluate MMGCN on the multimodal IEMOCAP and MELD benchmarks, using aligned textual, acoustic, and visual inputs. Performance is measured against established ERC models with weighted average F1-score.
- Datasets: MMGCN is evaluated on IEMOCAP and MELD, two multimodal benchmarks containing aligned acoustic, visual, and textual utterance information.IEMOCAP contains dyadic conversations, while MELD contains multi-speaker conversations.
- Datasets: IEMOCAP contains 7,433 utterances from 151 dyadic dialogues, with six emotion classes annotated per utterance.The videos comprise 12 hours of two-way conversations from ten speakers.
- Datasets: MELD contains 13,708 utterances across 1,433 conversations and 304 speakers, with seven annotated emotion classes.Unlike IEMOCAP, MELD includes conversations with three or more speakers.
- Feature extraction: Textual, acoustic, and visual raw features are extracted with TextCNN, OpenSmile using IS10, and a FER+-pretrained DenseNet, respectively.These extractors provide the three modality-specific inputs used in the multimodal experiments.
- Evaluation and training: Weighted average F1-score is the evaluation metric, and paired t-tests assess performance improvements at a 0.05 significance level.For MELD, focal loss is used during MMGCN training to address class imbalance, and layer normalization follows speaker embedding.
- Compared models: The comparison includes contextual ERC baselines such as BC-LSTM, CMN, DialogueRNN, and DialogueGCN.The baselines differ in how they model dialogue context and speaker information; BC-LSTM omits speaker information, while DialogueRNN uses separate GRUs.
5 Results and Discussions
Experiments on IEMOCAP and MELD show that MMGCN improves multimodal ERC through graph-based fusion, modality combinations, speaker embeddings, and long-distance contextual aggregation.
- Comparison with other models: 1.18% on IEMOCAP and 0.42% on MELD: MMGCN improves F1-score over DialogueGCN under multimodal settings, with p-value < 0.05.DialogueGCN performs best with textual inputs alone, while concatenated multimodal features provide only slight improvement.
- MMGCN under various modality settings: Textual inputs provide the best single-modality performance, visual inputs the worst, and adding acoustic and visual modalities improves over text alone.These results are reported for MMGCN across both benchmark datasets.
- Comparison with other fusion methods: MMGCN achieves higher performance than all compared multimodal fusion methods using graph-based multimodal fusion.The comparison includes early fusion, late fusion, gated attention, MFN, and MulT-based alternatives.
- MMGCN with different layers: Four graph-convolution layers yield MMGCN’s best performance on both IEMOCAP and MELD.Changing the number of layers affects ERC recognition performance.
- Impact of Speaker Embedding: Removing speaker embeddings significantly degrades MMGCN performance, with the difference supported by a t-test at p<0.05.Speaker embeddings differentiate input features from different speakers.
- Case Study: MMGCN attends to both nearby and long-distance contextual utterances, aggregating relevant information such as the 3rd utterance for the 20th utterance.The heatmap case study contrasts this behavior with simple sequential models.
6 Conclusion
MMGCN is a multimodal graph convolutional network for emotion recognition in conversation. It captures intra-speaker and inter-modality dependencies, and significantly outperforms other state-of-the-art methods on IEMOCAP and MELD.
- MMGCN provides a more effective way to use multimodal and long-distance contextual information for multimodal emotion recognition in conversation.
- Its graph captures both intra-speaker context dependency and inter-modality dependency.
- Residual connections enable deeper MMGCN layers to further improve recognition performance.
- MMGCN significantly outperforms other state-of-the-art methods under the multimodal conversation setting on IEMOCAP and MELD.