Source-linked AI summary
M2FNet: Multi-modal Fusion Network for Emotion Recognition in Conversation
Vishal Chudasama, Purbayan Kar, Ashish Gudmalwar, Nirmesh Shah, Pankaj Wasnik, Naoyuki Onoe
TL;DR
ERC research often emphasizes text despite conversational videos containing textual, visual, and acoustic cues. M2FNet learns emotion-relevant features across these modalities and fuses them with multi-head attention, achieving higher weighted average F1 than other methods on MELD and IEMOCAP. Its reported limitations include confusion between similar emotions and majority-class bias on highly imbalanced data.
Problem
Text-focused ERC methods underuse acoustic and visual information, although multimodal conversational data contain textual, visual, and acoustic cues.
Method
M2FNet learns audio-visual features with an adaptive margin-based triplet loss and combines them with text through multi-head fusion attention.
Results
M2FNet obtains a higher weighted average F1 score than other text-based and multimodal ERC methods on MELD and IEMOCAP.
Takeaways & Limitations
Encoding both scene and face information is essential for emotion recognition, while multimodal fusion helps leverage information across modalities.
Takeaways & Limitations
The model can confuse similar emotions and misclassify many samples as the majority class, especially Neutral in MELD.
Abstract
from arXiv · showhide
Emotion Recognition in Conversations (ERC) is crucial in developing sympathetic human-machine interaction. In conversational videos, emotion can be present in multiple modalities, i.e., audio, video, and transcript. However, due to the inherent characteristics of these modalities, multi-modal ERC has always been considered a challenging undertaking. Existing ERC research focuses mainly on using text information in a discussion, ignoring the other two modalities. We anticipate that emotion recognition accuracy can be improved by employing a multi-modal approach. Thus, in this study, we propose a Multi-modal Fusion Network (M2FNet) that extracts emotion-relevant features from visual, audio, and text modality. It employs a multi-head attention-based fusion mechanism to combine emotion-rich latent representations of the input data. We introduce a new feature extractor to extract latent features from the audio and visual modality. The proposed feature extractor is trained with a novel adaptive margin-based triplet loss function to learn emotion-relevant features from the audio and visual data. In the domain of ERC, the existing methods perform well on one benchmark dataset but not on others. Our results show that the proposed M2FNet architecture outperforms all other methods in terms of weighted average F1 score on well-known MELD and IEMOCAP datasets and sets a new state-of-the-art performance in ERC.
1. Introduction
M2FNet addresses the limits of text-focused ERC by fusing textual, visual, and acoustic information through attention-based multimodal representations. It introduces specialized audio-visual feature learning and evaluates the model on MELD and IEMOCAP.
- Text-focused ERC methods overlook acoustic and visual information even though ERC data contain all three modalities.
- M2FNet uses a multi-head attention-based fusion layer to combine modality-specific latent representations in the textual feature space.
- M2FNet obtains a higher weighted average F1 score than other text-based and multimodal methods on MELD and IEMOCAP.
- The model introduces a feature extractor and adaptive margin-based triplet loss to learn deeper emotion-relevant audio and visual representations.
- A weighted face model incorporates emotional information from multiple people present in a scene.
2. Related Works
Prior ERC work increasingly models textual context, but multimodal methods often use limited visual representations and simple fusion. This motivates approaches that retain facial and cross-modal information.
- 2.1. Text-based methods: Recent ERC methods increasingly treat the task as text-based, using contextual, transformer, graph, recurrent, or commonsense representations.
- 2.1. Text-based methods: Earlier dyadic methods use previous utterances from both speakers to provide conversational context for emotion prediction.
- 2.2. Multi-modal Methods: Many multimodal methods process whole frames without extracting distinct facial features that contribute to emotional context.
- 2.2. Multi-modal Methods: Existing multimodal methods often lack active fusion strategies beyond simple concatenation for acoustic and visual information.
3. Proposed Framework
M2FNet processes text, audio, and visual utterance signals through modality-specific and dialogue-level feature extraction, then fuses them with multi-head attention to predict emotions. Its audio-visual extractor uses adaptive-margin triplet learning, while the fusion design changes cross-modal keys to incorporate inter-modal information.
- Framework overview: M2FNet represents each utterance with text, audio, and visual components and predicts its emotion using the dialogue context.The framework separates utterance-level feature extraction from dialogue-level extraction.
- Utterance-level feature extraction: The framework independently extracts modality features before passing them to dialogue-level networks that learn inter-utterance context.Text uses a modified RoBERTa extractor, while audio and visual signals use corresponding feature extractors.
- Audio features: Audio is converted into augmented RGB Mel spectrograms, and the proposed audio extractor produces utterance-level audio embeddings.Augmentation includes time warping and additive white Gaussian noise before Mel-spectrogram computation.
- Visual features: The visual pathway jointly models scene context and facial expressions across 15 successive frames, using face detection, weighted aggregation, and temporal max pooling.Scene embeddings and facial features are concatenated into a comprehensive visual representation.
- Multi-modal fusion: The fusion module uses text features as Query and Value while visual and acoustic features serve as Keys to modulate attention across modalities.Unlike prior multi-head attention use described here, the strategy changes the key across modalities while keeping Query and Value the same.
- Feature extractor training: The audio-visual extractor addresses fixed-margin triplet-loss limitations with an adaptive margin intended to continue correcting insufficiently separated representations.The cited discussion describes zero loss cases where positive and negative distances do not provide adequate separation.
4. Experimental Analysis and Discussion
Experiments evaluate M2FNet on the MELD and IEMOCAP benchmarks through ablations and comparisons with text-based and multimodal ERC methods. Multimodal inputs, dual visual representations, and multi-head fusion contribute to performance, while M2FNet achieves higher reported results than prior methods.
- Datasets and evaluation: M2FNet is evaluated on the multimodal MELD and IEMOCAP benchmark datasets using weighted average F1 score and classification accuracy.MELD contains more than 1,400 conversations and 13,000 utterances with seven emotion labels; IEMOCAP contains approximately 12 hours of audiovisual dyadic-session data with six labels.
- Ablation studies: All three modalities produce higher accuracy and weighted average F1 than using only one or two modalities in the modality ablation.The fusion mechanism improves accuracy by 2.53% on IEMOCAP and 0.57% on MELD, while weighted average F1 improves by 2.74% on IEMOCAP but is slightly lower on MELD.
- Ablation studies: Using both scene and weighted face embeddings improves results, whereas either visual representation alone does not improve performance.The analysis indicates that scene context and information about people in the scene are both important for emotion recognition.
- Ablation studies: The best transformer-encoder setting differs by dataset: one encoder per modality performs best on MELD, while five perform best on IEMOCAP.The settings are NA = NV = NT = 1 for MELD and NA = NT = NV = 5 for IEMOCAP.
- Ablation studies: Five multi-head attention fusion modules achieve higher quantitative measures on both MELD and IEMOCAP than the tested alternatives.The proposed model therefore uses m = 5 fusion modules in this analysis.
- Comparative analysis: The model obtains its highest emotion-specific F1 score, 82.11%, for Sad on IEMOCAP and its highest MELD score for Neutral.These emotion-specific predictions are summarized for the MELD and IEMOCAP test sets.
- Comparative analysis: M2FNet exceeds previous text-based methods by 0.21% on MELD and 1.83% on IEMOCAP in weighted average F1 score.The comparison identifies EmotionFlow as the previous best on MELD and DAG-ERC as the previous best on IEMOCAP.
- Comparative analysis: Against previous multimodal methods, M2FNet gains 2.19% accuracy and 2.71% weighted average F1 on MELD, plus 0.77% accuracy and 0.63% weighted average F1 on IEMOCAP.The IEMOCAP comparison uses DialogueTRM as the previous best performance.
5. Limitations
M2FNet remains vulnerable to confusion between similar emotions and to majority-class predictions on highly imbalanced data.
- Limitations: M2FNet sometimes confuses similar emotions, including Frustration with Anger and Happy with Excited.The limitation concerns misclassification among emotionally close categories.
- Limitations: On highly imbalanced data, the model often misclassifies emotions as the most frequent class, especially Neutral on MELD.The reported MELD behavior is that many emotions are overwhelmingly predicted as Neutral.
6. Conclusion
The paper concludes that M2FNet combines multimodal fusion, audio-visual feature extraction, weighted face modeling, and scene context for ERC. Its analyses support using multimodal fusion and both scene- and face-related information, with quantitative validation on MELD and IEMOCAP.
- Conclusion: M2FNet uses multi-head fusion attention, an audio-visual feature extractor, adaptive margin triplet loss, and a weighted face model for ERC.The conclusion describes these components as part of the proposed robust multimodal fusion network.
- Conclusion: The analysis finds that multimodal fusion and combining scene and face-related information are important for emotion recognition.The conclusion states that encoding both scene and face-related information is essential and that multimodal fusion is necessary to leverage utterance modalities.
- Conclusion: Experiments quantitatively validate the proposed network on the MELD and IEMOCAP benchmark datasets.The conclusion presents this validation as the final experimental support for M2FNet.