COGMEN: COntextualized GNN based Multimodal Emotion recognitioN

1 Introduction

• Multimodal emotion recognition monitors emotions expressed through modalities such as video and audio, which can provide complementary cues.The paper motivates emotion recognition as important for AI technologies that interact directly with people.Source: §1 · p. 1
• In conversation, emotions can reflect another speaker's utterances and a speaker's own emotional state across utterances.The example describes Speaker-2 reacting to Speaker-1 at U2, U4, and U6, while maintaining curiosity at U8 and U10.Source: §1 · p. 1
• COGMEN uses a contextualized GNN architecture to predict each speaker's emotion per utterance by leveraging local and global conversational information.The model targets inter-speaker and intra-speaker dependencies as well as context.Source: §1 · p. 2
• COGMEN reports state-of-the-art results on IEMOCAP and MOSEI, with ablations showing the importance of local and global information.The analysis also examines the importance of the GNN component.Source: §1 · p. 2

2 Related Work

• Emotion recognition research spans applications such as emotion understanding, opinion mining, and generation; CMU-MOSEI and IEMOCAP accelerated progress.The review groups approaches into unimodal and multimodal categories.Source: §2 · p. 2
• Unimodal approaches typically use text alone, while multimodal approaches combine modalities such as audio, video, and text.The review notes that some unimodal techniques also apply to the paper's setting.Source: §2 · p. 2
• Text-based methods include commonsense classification in COSMIC (Yu et al., 2019), emotional consistency in CESTa (Wang et al., 2020), and graph models for context propagation.The graph-based examples include DialogueGCN (Ghosal et al., 2019), RGAT, ConGCN, and SumAggGin.Source: §2 · p. 2
• Multimodal methods include MFN's gated memory for intra- and cross-view interactions (Zadeh et al., 2018a) and Graph-MFN's dynamic fusion graph (Bagher Zadeh et al., 2018).Other reviewed approaches use conversation-history memories, utterance attention, contextual RNNs, or joint multimodal transformers.Source: §2 · pp. 2–3

3 Proposed Model

• COGMEN combines dialogue context with nearby-utterance relations to model inter-speaker and intra-speaker dependencies for per-utterance emotion prediction.Source: §3 · p. 3
• A transformer encoder without positional encodings uses the entire dialogue context to generate utterance representations, allowing information to flow from all utterances during prediction.Source: §3 · p. 3
• Directed graph relations connect utterances by speaker and spoken order; an RGCN and GraphTransformer model their inter- and intra-speaker dependencies.Source: §3 · p. 3, §3.1 · pp. 4–5

3.1 Overall Architecture

• The architecture sends utterances through a context extractor, forms a speaker-interaction graph, applies an RGCN and GraphTransformer, then classifies emotions with two linear layers.Source: §3.1 · p. 3

4 Experiments

• IEMOCAP is evaluated in both four- and six-emotion settings, while MOSEI covers six emotion labels and seven sentiment levels.MOSEI emotion labels differ from IEMOCAP's.Source: §4 · p. 5
• IEMOCAP uses OpenSmile audio, Baltrusaitis video, and sBERT text features; MOSEI uses audio and video features from prior work and sBERT text.Audio and visual features are averaged from token-level features to utterance level.Source: §4 · p. 5
• Tables 2 and 3 report multimodal IEMOCAP results for the six-way and four-class settings, respectively.Table 2 reports weighted-average results; Table 3 reports weighted F1-score.Source: Table 2 · p. 6, Table 3 · p. 6
• Available audio, text, and video features are concatenated, and this fusion performed best among the explored mechanisms.The model is compared with multimodal baselines, including DialogueRNN on IEMOCAP and Multilogue-Net and TBJE on MOSEI.Source: §4 · p. 6

5 Results and Analysis

• 7.7% improvement over the previous SOTA model is reported for four-way IEMOCAP; COGMEN also beats prior baselines on six-way accuracy and F1-score.The six-way results show improvements in class-wise F1 for happy, sad, neutral, and excited emotions.Source: §5 · p. 6
• On MOSEI, COGMEN outperforms baselines in most cases, reaches 85% accuracy for A+T two-class sentiment, and performs comparably in seven-class sentiment.Unlike other multimodal approaches described, COGMEN shows a performance boost when visual modality is added.Source: §5 · pp. 6–7, Table 4 · p. 7
• Reducing dialogue context and removing the GNN lower performance, while making dialogue graph relations uniform also causes a drop.The context experiment uses IEMOCAP four-way dialogues; the GNN and relation ablations are reported on IEMOCAP.Source: §5 · p. 7, Table 5 · p. 7, Table 6 · p. 7
• COGMEN is better than or comparable to state-of-the-art unimodal architectures on IEMOCAP four-way, though text-only models are optimized for text.The paper states that adding other modalities helps improve COGMEN's performance.Source: Table 7 · p. 7, §5 · p. 8
• Emotion-shift examples are classified less accurately than examples without a shift, and similar emotions are often confused.Accuracy is 53.6% when emotion shifts versus 74.2% when it remains the same; other labels are also misclassified as neutral amid a larger neutral-example proportion.Source: §5 · p. 8, Figure 3 · p. 8
• After the GNN, IEMOCAP emotion clusters form better; masking utterances 9, 10, and 11 in one dialogue sharply drops its F1-score.The masked utterances coincide with an emotion shift, while masking the first four neutral utterances has significantly less effect.Source: §5 · p. 8, Figure 4 · p. 8, Figure 5 · p. 8

6 Discussion

• COGMEN uses transformers for global context and GraphTransformers for self and interpersonal dependencies in a unified architecture.A shared emotion classifier predicts from the resulting utterance-level features, and the authors report improved performance after adding modalities.Source: §6 · p. 9
• Using future utterances makes COGMEN and the compared baseline systems offline methods that assume the full dialogue is available during inference.The discussion identifies online emotion recognition for live telecasting and telecommunication as a direction for future work.Source: §6 · p. 9
• In IEMOCAP 4-way experiments, performance decreased as the number of utterances in the dialogue was reduced, underscoring the importance of context.The authors suggest smaller context buffers as a possible approach for online prediction, with real-time performance improvements left for future research.Source: §6 · p. 9

7 Conclusion and Future Work

• The authors describe their approach as using GNNs for multimodal emotion recognition and test COGMEN on IEMOCAP and MOSEI.They report that the model fuses modalities effectively to improve emotion-prediction performance.Source: §7 · p. 9
• COGMEN outperforms existing state-of-the-art multimodal emotion-recognition methods, with a 7.7% F1-score increase for IEMOCAP (4-way).The authors report that comprehensive analysis and ablation studies show the importance of different modules.Source: §7 · p. 9
• Capturing emotional shifts for fine-grained emotion prediction is identified as a focus for future work.Source: §7 · p. 9

A Hyperparameter Setting

• Hyperparameters for IEMOCAP and MOSEI are presented in Tables 8 and 9, respectively.The IEMOCAP model has 55,932,052 parameters and takes around 7 minutes to train for 50 epochs on an NVIDIA Tesla K80 GPU.Source: A Hyperparameter Setting · p. 14, Table 8 · p. 14, Table 9 · p. 14

B Dataset Analysis

• A high percentage of transitions between similar emotions is associated with models confusing similar emotion classes.The analysis also deduces poor emotion-recognition performance for transitions between opposite states, such as happy to sad.Source: B Dataset Analysis · p. 14
• The authors plan to target a model that performs better in fine-grained emotion recognition and is robust toward emotion shifts.Source: B Dataset Analysis · p. 14
• Figure 6 presents emotion transitions across speakers between consecutive utterances, while Figure 7 presents transitions across consecutive utterances of the same speaker.Source: Figure 6 · p. 14, Figure 7 · p. 14

C Evaluation Metrics

• F1 is the harmonic mean of precision and recall, with equal relative contribution from both.The score ranges from 0 as its worst value to 1 as its best value.Source: C Evaluation Metrics · p. 14
• Weighted F1 averages per-label metrics using each label's support, or number of true instances, as weights.Source: C Evaluation Metrics · p. 14
• Accuracy is the percentage of correct predictions in the test set.Source: C Evaluation Metrics · p. 14

• Table 11 reports results across modality combinations for IEMOCAP 4-way and 6-way classification.Source: D Results on Modality Combinations · p. 14, Table 11 · p. 15
• Figure 8 presents the confusion matrix for IEMOCAP 4-way predictions.Source: D Results on Modality Combinations · p. 14, Figure 8 · p. 15
• The reported modality-combination results cover both IEMOCAP 4-way and 6-way datasets.Source: Table 11 · p. 15

E Additional Analysis

• UMAP visualizes IEMOCAP features before and after the GNN layer, and Figure 10 examines utterance importance.Source: E Additional Analysis · p. 15, Figure 9 · p. 15, Figure 10 · p. 15
• Masking utterances 5–15 affects future utterances, while masking high-emotion-shift utterances 15–30 drops dialogue F1.Source: E Additional Analysis · pp. 15–16

F Graph Formation

• Graph formation represents each utterance as a node and connects utterances through relations, including links to past and future utterances.Source: F Graph Formation · p. 16, Table 12 · p. 16

G.1 Modality Fusing Mechanisms

• Simple concatenation performs without significant improvement from attention-based fusion mechanisms and avoids their extra computation steps.Source: §G.1 · p. 16

G.2 Effect of window size in Graph Formation

• Window size is adjustable: larger windows suit longer-lived speaker dependencies, while smaller windows suit frequent topic changes and weaker cross-speaker influence.The authors plan to explore automatic, dialogue-dependent window-size selection.Source: §G.2 · p. 17