PaperlayerView exact revision on arXiv

Source-linked AI summary

GraphMFT: A Graph Network based Multimodal Fusion Technique for Emotion Recognition in Conversation

Jiang Li, Xiaoping Wang, Guoqing Lv, Zhigang Zeng

arXiv:2208.00339v5cs.MM

TL;DR

Multimodal emotion recognition in conversation has few existing studies, and prior approaches may neglect multimodal interaction or mishandle heterogeneous inputs. GraphMFT uses improved graph attention networks on pairwise modality graphs, and is reported to outperform all baselines on IEMOCAP and MELD, with strong IEMOCAP scores.

  • Problem

    Few multimodal ERC models exist, and prior recurrence- and graph-based approaches may neglect multimodal information or inter-modal interaction while graph fusion can face heterogeneity and noisy connections.

    Source: §1 · p. 1

  • Method

    GraphMFT models conversations with pairwise modality graphs and improved graph attention networks to capture intra-modal context and inter-modal complementary information.

    Source: §1 · p. 2

  • Results

    GraphMFT outperforms all baselines on IEMOCAP and MELD; on IEMOCAP it achieves 67.90% accuracy and 68.07% weighted-average F1.

    Source: §1 · p. 2, §5.1 · p. 7

  • Takeaways & Limitations

    The reported results support GraphMFT as an effective multimodal fusion approach on the two evaluated public benchmark datasets.

    Source: §1 · p. 2, §6 · p. 9

    GraphMFT still sometimes confuses similar emotions, including classifying Angry as Frustrated and Excited as Happy.

    Source: §5.7 · p. 9

Abstract

from arXiv · show

Multimodal machine learning is an emerging area of research, which has received a great deal of scholarly attention in recent years. Up to now, there are few studies on multimodal Emotion Recognition in Conversation (ERC). Since Graph Neural Networks (GNNs) possess the powerful capacity of relational modeling, they have an inherent advantage in the field of multimodal learning. GNNs leverage the graph constructed from multimodal data to perform intra- and inter-modal information interaction, which effectively facilitates the integration and complementation of multimodal data. In this work, we propose a novel Graph network based Multimodal Fusion Technique (GraphMFT) for emotion recognition in conversation. Multimodal data can be modeled as a graph, where each data object is regarded as a node, and both intra- and inter-modal dependencies existing between data objects can be regarded as edges. GraphMFT utilizes multiple improved graph attention networks to capture intra-modal contextual information and inter-modal complementary information. In addition, the proposed GraphMFT attempts to address the challenges of existing graph-based multimodal conversational emotion recognition models such as MMGCN. Empirical results on two public multimodal datasets reveal that our model outperforms the State-Of-The-Art (SOTA) approaches with the accuracy of 67.90% and 61.30%.

1. Introduction

Prior ERC models often miss long-distance context or multimodal interaction, while MMGCN introduces additional concerns about noisy connections and heterogeneous inputs. GraphMFT uses pairwise modality graphs and improved attention networks to model intra- and inter-modal information, and is reported to outperform all baselines.

1. Introduction

  • Recurrence-based models struggle with long-distance utterances, while prior recurrence- and graph-based approaches may neglect multimodal data or inter-modal interaction.MMGCN also coarsely connects utterances and treats a complex heterogeneous graph as homogeneous, potentially adding noise and complicating fusion.Source: §1 · pp. 1–2
  • GraphMFT applies multiple improved graph attention networks to capture intra-modal context and inter-modal complementary information.The paper presents it as a GNN-based multimodal fusion method for conversational emotion detection.Source: §1 · p. 2
  • Compared with MMGCN, GraphMFT uses variable context nodes, a spatial-based GNN with explicitly constructed edges, learned attention weights, and a new fusion network.MMGCN is described as connecting to all intra-modal utterances, using a spectral-based GNN and angular similarity for edge weights, and applying GCNII.Source: §1 · p. 2
  • The model represents a conversation with VA, VT, and AT graphs, each containing only two modalities and edges for intra- and inter-modal dependencies.The authors state that using pairwise modality graphs decreases the difficulty of multimodal fusion.Source: §1 · p. 2
  • GraphMFT outperforms all baseline methods in experiments on the IEMOCAP and MELD datasets.The introduction identifies both datasets as public benchmarks.Source: §1 · p. 2

2. Related works

Related work covers unimodal and multimodal emotion recognition, graph convolutional networks, and multimodal fusion. The discussion identifies limited multimodal ERC research and challenges in modeling interactions and heterogeneous data.

2.1. Emotion recognition in conversion

  • Multimodal ERC methods use complementary modalities, but existing approaches may miss inter-modal interaction or overlook multimodal heterogeneity; the authors note that relatively few multimodal ERC models exist.Unimodal approaches are described as offering limited performance gains in increasingly complex conversational scenarios.Source: §2.1 · pp. 2–3

2.2. Graph convolutional networks

  • Graph convolutional networks include spectral models defined through graph Fourier transforms and spatial models defined by aggregating neighboring nodes.The paper uses spatial-based GAT as its backbone, unlike MMGCN's spectral GCN.Source: §2.2 · p. 3

2.3. Multimodal machine learning

  • Multimodal machine learning includes fusion among its main components, yet the authors describe GNN-based multimodal fusion models as relatively scarce.The cited work spans cross-modal BERT, recurrent fusion, gating, and pairwise-modality approaches.Source: §2.3 · p. 3

3. The proposed architecture

The architecture combines unimodal encoding, graph-based multimodal fusion, and emotion prediction. It formulates the task as predicting each utterance’s emotion from intra- and inter-modal interactions across visual, acoustic, and textual modalities.

3. The proposed architecture

  • The proposed architecture has three stages: unimodal encoding, graph-based multimodal fusion, and emotion prediction.The fusion stage includes speaker embedding, graph construction, and multi-graph intra- and inter-modal interactions.Source: §3 · pp. 3–4, Figure 2 · p. 4

3.1. Problem definition

  • Each of m utterances has visual, acoustic, and textual expressions, yielding 3 × m utterance expressions in a conversation.The paper identifies V, A, and T as the visual, acoustic, and textual modalities.Source: §3.1 · p. 4
  • The task predicts each utterance’s emotion using intra- and inter-modal interaction learning in conversations with two or more speakers.The described datasets, IEMOCAP and MELD, have six and seven emotion labels, respectively.Source: §3.1 · p. 4

3.2. Unimodal encoding

Unimodal encoding uses separate fully connected networks for visual and acoustic utterances and a bidirectional LSTM for textual utterances.

3.2. Unimodal encoding

  • Separate fully connected networks encode visual and acoustic utterances to enhance the expressiveness of their features.The text describes visual and acoustic encoding separately.Source: §3.2 · p. 4
  • A bidirectional LSTM extracts contextual information from the current textual utterance.The textual modality uses a distinct encoder from the visual and acoustic modalities.Source: §3.2 · p. 4
  • The unimodal encoding process is represented with trainable parameters for the visual and acoustic encoders.The passage also identifies trainable parameters for the textual encoder.Source: §3.2 · p. 4

3.3. Graph network based multimodal fusion technique

GraphMFT combines speaker-aware utterance features with graphs that connect modalities within and across utterances. Improved graph attention networks process these graphs to extract contextual and complementary information for fusion.

3.3. Graph network based multimodal fusion technique

  • GraphMFT lets a current-modality utterance interact with both intra-modal and inter-modal utterances.The stated aim is to encode intra-modal contextual information and inter-modal complementary information.Source: §3.3 · p. 4

3.3.1. Speaker embedding

  • Speaker embeddings are added to each modality’s utterance features to carry speaker information.The speaker array and number of speakers are fed to an embedding layer.Source: §3.3.1 · p. 4

3.3.2. Graph construction

  • The method constructs V-A, V-T, and A-T graphs, with each modality’s utterance represented as a node.The V-A graph has 2 × m nodes for the visual and acoustic utterances in a conversation.Source: §3.3.2 · pp. 4–5
  • Each node connects to past or future contextual nodes of its modality and to the other modality’s node from the same utterance.A graph attention module calculates edge weights to reflect the importance of neighboring nodes.Source: §3.3.2 · p. 5

3.3.3. Multi-graph based intra- and inter-modal interactions

  • The improved GAT adds each layer’s output to the next layer’s output and concatenates outputs across layers.The modification is intended to alleviate GNN over-smoothing.Source: §3.3.3 · p. 5, Figure 3 · p. 5
  • Three improved GATs process the V-A, V-T, and A-T graphs, and their modality-specific features are combined for multimodal fusion.The resulting visual, acoustic, and textual feature matrices are concatenated to form the final feature matrix.Source: §3.3.3 · pp. 5–6

3.4. Emotion prediction

Emotion prediction concatenates the fused visual, acoustic, and textual feature matrices into a final feature matrix. Each utterance’s final feature vector is then passed through a fully connected network to predict its emotion.

3.4. Emotion prediction

  • The fused visual, acoustic, and textual feature matrices are concatenated to form the final feature matrix.A learnable parameter is applied in the stated concatenation operation.Source: §3.4 · p. 6
  • The final feature vector for each utterance is input to a fully connected network for emotion prediction.The vector is taken from the final feature matrix.Source: §3.4 · p. 6
  • The network produces a probability distribution over predicted emotions and a predicted emotion label for each utterance.The passage identifies Softmax as the function for the probability distribution.Source: §3.4 · p. 6

3.5. Objective training

The model is trained with cross-entropy loss as its objective. The loss uses ground-truth and predicted emotion distributions and includes an L2-regularization term.

  • Cross-entropy loss is the objective used to train the proposed model.Source: §3.5 · p. 6
  • The loss formalization uses ground-truth and predicted emotion distributions for each utterance, with an L2-regularizer weight and learnable parameters.Source: §3.5 · p. 6

4. Experiments

The experiments compare GraphMFT with multimodal and conversational baselines on MELD and IEMOCAP, using dataset-specific settings and weighted-average F1 and accuracy.

4.1. Baseline methods

  • The baseline set spans multimodal fusion, recurrent conversational models, graph-based context modeling, cognitive reasoning, and MMGCN.The comparisons include TFN, MFN, BC-LSTM, DialogueRNN, DialogueGCN, DialogueCRN, and MMGCN.Source: §4.1 · p. 6

4.2. Datasets

  • Experiments use the multimodal ERC benchmarks MELD and IEMOCAP; Table 1 summarizes dataset counts and emotion-class distributions.MELD has over 1,400 conversations and 13,700 utterances across seven emotions, while IEMOCAP has 151 conversations and 7,433 dyadic utterances across six emotions; MELD has especially few Fear and Disgust samples.Source: §4.2 · pp. 6–7, Table 1 · p. 7
  • MELD consists of multi-party conversations with three or more speakers, collected from the TV show Friends.Source: §4.2 · p. 7

4.3. Detailed settings

  • GraphMFT uses dataset-specific layer counts, learning rates, and batch sizes, and is evaluated with weighted-average F1 and accuracy.MELD uses 3 layers, a 0.00002 learning rate, and batch size 32; IEMOCAP uses 5 layers, a 0.00001 learning rate, and batch size 16.Source: §4.3 · p. 7

5. Results and analysis

GraphMFT outperforms baseline methods on IEMOCAP and MELD, and the experiments examine its attention module, speaker embeddings, modalities, and context settings. The analysis also identifies emotion-confusion problems that remain.

5.1. Overall results

  • 67.90% accuracy and 68.07% weighted-average F1 on IEMOCAP; GraphMFT exceeds MMGCN, beats DialogueCRN by 2.73% F1, and improves MELD accuracy over MMGCN by 1.99%.The overall-results table reports the models and evaluation metrics.Source: §5.1 · p. 7, Table 3 · p. 8

5.2. Comparison of vanilla and improved GATs

  • Replacing improved GAT with vanilla GAT lowers IEMOCAP weighted-average F1 by 1.60%; on MELD, vanilla GAT accuracy is 60.19%, 1.11% below improved GAT.These comparisons test the improved GAT module against vanilla GAT.Source: §5.2 · p. 7, Table 4 · p. 8

5.3. Impact of number of network layers

  • With more layers, vanilla GAT performance drops sharply and remains low, while improved GAT initially improves and later degrades only weakly.The comparison investigates whether the improved GAT alleviates over-smoothing; Figure 5 reports IEMOCAP results.Source: §5.3 · pp. 7–8, Figure 5 · p. 8

5.4. Impact of speaker embedding

  • 65.61% weighted-average F1 without speaker embeddings on IEMOCAP is 2.46% below the score with embeddings; adding embeddings raises accuracy by 2.16%.Speaker embeddings also improve performance on MELD.Source: §5.4 · p. 8, Table 5 · p. 8

5.5. Comparison of different modal settings

  • Three-modality models outperform two-modality models; among two-modality settings, visual-acoustic performs worst and acoustic-textual performs best.The reported comparison indicates acoustic and textual modalities contribute more to GraphMFT than the visual modality.Source: §5.5 · p. 9, Table 6 · p. 9

5.6. Impact of number of contextual nodes

  • IEMOCAP performance fluctuates after the (16, 16) context setting; optimal performance uses 16 contextual nodes there and 5 on MELD.The authors suggest MELD’s neighboring utterances may not be contiguous in the actual scenario, so long-range context modeling is not required.Source: §5.6 · p. 9

5.7. Limitations

  • GraphMFT still misclassifies similar emotions and non-Neutral utterances, but versus MMGCN reduces Angry→Frustrated from 29.41% to 26.47% and Excited→Happy from 14.38% to 12.71%.The paper identifies similar-emotion confusion and predictions of Neutral for non-Neutral emotions as persistent ERC challenges.Source: §5.7 · p. 9

6. Summary

The paper summarizes GraphMFT as a multimodal graph-fusion method evaluated on two benchmark datasets. It reports superior performance and points to multimodal dialogue generation as a future direction.

6. Summary

  • GraphMFT models each conversation as VA, V-T, and A-T heterogeneous graphs, with each graph containing data from two modalities to reduce fusion difficulty.The method also improves vanilla GAT, inspired by ResGCN, to address over-smoothing.Source: §6 · p. 9
  • On two widely used benchmark datasets, GraphMFT performs intra- and inter-modal interactions to extract contextual and complementary information and outperforms the previous baseline.Source: §6 · p. 9
  • The authors suggest extending graph-based models to multimodal dialogue generation as a future research direction.Source: §6 · p. 9
  • Figure 6 depicts performance across past/future contextual-node combinations; Figure 7 compares GraphMFT and MMGCN confusion matrices, with both facing similar-emotion and non-Neutral emotion problems.Source: Figure 6 · p. 10, Figure 7 · p. 10

CRediT authorship contribution statement

The contribution statement assigns conceptual, methodological, data, software, validation, analysis, investigation, visualization, writing, project, supervision, and funding roles among four authors.

CRediT authorship contribution statement

Related papers

Loading 2208.00339v5…