Source-linked AI summary
MM-DFN: Multimodal Dynamic Fusion Network for Emotion Recognition in Conversations
Dou Hu, Xiaolong Hou, Lingwei Wei, Lianxin Jiang, Yang Mo
TL;DR
Multimodal ERC needs stronger conversational context understanding and fusion across modalities because existing graph-based methods accumulate redundant information. MM-DFN introduces stacked graph-based dynamic fusion modules that model contextual dynamics across semantic spaces, and experiments on two benchmark datasets demonstrate its effectiveness and superiority.
Problem
Existing multimodal ERC methods have difficulty capturing complex cross-modal and cross-utterance interactions, while graph-based fusion can accumulate redundant contextual information across layers.
Method
MM-DFN uses modality encoders and a gated Graph-based Dynamic Fusion module stacked to fuse intra- and inter-modality conversational context in dynamic semantic spaces.
Results
MM-DFN consistently obtains the best performance over comparison methods on the IEMOCAP and MELD benchmark datasets.
Takeaways & Limitations
Modeling contextual dynamics across semantic spaces reduces redundancy and enhances complementarity between modalities for multimodal ERC.
Takeaways & Limitations
The formulation represents each conversation as a sequence of N utterances, defining the task around utterance-level conversational inputs.
Abstract
from arXiv · showhide
Emotion Recognition in Conversations (ERC) has considerable prospects for developing empathetic machines. For multimodal ERC, it is vital to understand context and fuse modality information in conversations. Recent graph-based fusion methods generally aggregate multimodal information by exploring unimodal and cross-modal interactions in a graph. However, they accumulate redundant information at each layer, limiting the context understanding between modalities. In this paper, we propose a novel Multimodal Dynamic Fusion Network (MM-DFN) to recognize emotions by fully understanding multimodal conversational context. Specifically, we design a new graph-based dynamic fusion module to fuse multimodal contextual features in a conversation. The module reduces redundancy and enhances complementarity between modalities by capturing the dynamics of contextual information in different semantic spaces. Extensive experiments on two public benchmark datasets demonstrate the effectiveness and superiority of MM-DFN.
1. INTRODUCTION
MM-DFN addresses multimodal ERC by modeling conversational context and dynamically fusing modalities, targeting redundancy in graph-based fusion and insufficient cross-utterance interaction. Experiments on IEMOCAP and MELD report consistent superiority over comparison methods.
- Multimodal ERC requires accurate context understanding and fusion of acoustic, visual, and textual information to detect utterance-level emotions.
- Early or late fusion struggles to capture contextual interactions between modalities, limiting the use of multimodal information.Hybrid methods also focus on isolated or sequential utterances and insufficiently exploit interactions across conversation utterances.
- Graph-based fusion models unimodal and cross-modal interactions but can accumulate redundant information in a specific semantic space at each layer.The paper frames contextual information as views with distinct representation spaces and dynamics.
- MM-DFN uses modality encoders and stacked Graph-based Dynamic Fusion modules to fuse multimodal conversational context in dynamic semantic spaces.GDF combines graph convolution over intra- and inter-modality context with gating mechanisms that control information flow between layers.
- The proposed module reduces redundancy and enhances complementarity between modalities for multimodal context understanding.
- Experiments on two public benchmark datasets, IEMOCAP and MELD, consistently show MM-DFN outperforming comparison methods.
2. METHODOLOGY
MM-DFN encodes modality-, speaker-, and conversation-level context before constructing a multimodal graph. Its gated dynamic fusion module selectively retains contextual information while graph convolution integrates intra- and inter-modality relations for utterance-level emotion prediction.
- Modality Encoder: MM-DFN represents each utterance with acoustic, visual, and textual features, while modeling speaker-specific utterance sequences with shared-parameter BiGRUs.The modality encoder combines context features with speaker embeddings before graph construction.
- Graph Construction: The conversation graph contains 3N modality nodes for N utterances, connecting same-modality nodes across a conversation and cross-modality nodes within each utterance.Node initialization uses both context and speaker embeddings; edge weights are computed from cosine similarity.
- Dynamic Fusion Module: Modified graph convolution aggregates intra- and inter-modality context at each layer while combining transformed representations with initial features through residual-style updates.The convolution uses a renormalized graph matrix and hyperparameters α and β^k.
- Dynamic Fusion Module: Gated updates learn sequential patterns across semantic spaces by controlling information written to memory, retaining relevant context, and filtering redundant information.The update, forget, and output gates regulate memory and the information passed to graph convolution.
- Emotion Classifier: After K fusion layers refine the three modality representations for each utterance, a classifier predicts its emotion and training uses cross-entropy loss with L2 regularization.The classifier parameters are trainable, and the loss is defined over utterance emotion probabilities and one-hot labels.
3. EXPERIMENTS
Experiments on IEMOCAP and MELD evaluate MM-DFN against prior methods, fusion modules, ablations, and modality settings. MM-DFN consistently performs best, while GDF improves fusion by reducing redundancy and modeling intra- and inter-modality interactions.
- Datasets: IEMOCAP contains 7,433 utterances in 151 dyadic dialogues with six emotion labels, while MELD contains 13,708 utterances in 1,433 multi-party conversations with seven labels.IEMOCAP uses four sessions for training, the last session for testing, and 10% of training dialogues for validation.
- Overall Results: MM-DFN consistently obtains the best performance on both benchmark datasets in the overall comparison.The evaluation reports Acc, weighted-average F1, and per-class F1, with Fear and Disgust omitted on MELD because of limited training samples.
- Ablation Study: Removing GDF or speaker embedding significantly decreases results on both datasets, and additionally removing context embedding decreases them further.These ablations support the effectiveness of GDF, speaker embedding, and context embedding.
- Fusion Modules: GDF and graph-based fusion outperform the other tested fusion modules because they capture intra- and inter-modality interactions in conversations.The compared alternatives include Concat/Gate Fusion, Tensor/Memory Fusion, Early/Late Fusion plus GCN, and graph-based fusion.
- Fusion Modules: Ignoring either intra-modal or inter-modal graph edges hurts performance, while GDF outperforms graph-based fusion across all tested edge variants.The results indicate that both interaction types contribute to multimodal context fusion and that GDF reduces both redundancy types.
- Modality Settings: Bimodal and trimodal models outperform corresponding unimodal models on both datasets, with textual modality strongest among unimodal settings.Among bimodal settings, acoustic plus textual modalities perform best, while GDF particularly improves acoustic plus visual fusion over graph-based fusion.
4. CONCLUSION
The paper concludes that MM-DFN uses graph-based dynamic fusion to understand multimodal conversational context for multimodal ERC. Its stacked GDF modules learn contextual dynamics across semantic spaces, reducing redundancy and enhancing modality complementarity, and experiments on two benchmark datasets demonstrate effectiveness and superiority.
- 4. CONCLUSION: MM-DFN targets multimodal ERC by fully understanding conversational context through graph-based dynamic fusion.The GDF module fuses multimodal features in a conversation.
- 4. CONCLUSION: A stack of GDF modules learns contextual-information dynamics in different semantic spaces, reducing redundancy and enhancing complementarity between modalities.
- 4. CONCLUSION: Experiments on two benchmark datasets demonstrate the effectiveness and superiority of MM-DFN.