PaperlayerView exact revision on arXiv

Source-linked AI summary

GA2MIF: Graph and Attention Based Two-Stage Multi-Source Information Fusion for Conversational Emotion Detection

Jiang Li, Xiaoping Wang, Guoqing Lv, Zhigang Zeng

arXiv:2207.11900v6cs.MM

TL;DR

Multimodal ERC faces challenges from heterogeneous graph inputs and redundant graph connections. GA2MIF combines directed graph attention for contextual modeling with pairwise cross-modal attention. It achieves optimal accuracy and weighted-average F1 on IEMOCAP and MELD.

  • Problem

    Graph-based multimodal ERC faces a heterogeneous-input challenge, while connecting all utterances in each modality can introduce noisy, useless connections.

    Source: §I · p. 2

  • Method

    GA2MIF uses MDGATs to model intra-modal context and MPCATs to model pairwise cross-modal interactions using MDGAT outputs.

    Source: §I · p. 2, 4) Attention-Based Cross-Modal Modeling · p. 6

  • Results

    GA2MIF achieves optimal accuracy and weighted-average F1 on IEMOCAP and MELD; on IEMOCAP, accuracy is 69.75%, 4.19% above MMGCN.

    Source: A. Comparison With Baseline Models · p. 9

  • Takeaways & Limitations

    The experiments show that GA2MIF captures intra-modal local and long-range context as well as inter-modal complementary information.

    Source: §I · p. 2

Abstract

from arXiv · show

Multimodal Emotion Recognition in Conversation (ERC) plays an influential role in the field of human-computer interaction and conversational robotics since it can motivate machines to provide empathetic services. Multimodal data modeling is an up-and-coming research area in recent years, which is inspired by human capability to integrate multiple senses. Several graph-based approaches claim to capture interactive information between modalities, but the heterogeneity of multimodal data makes these methods prohibit optimal solutions. In this work, we introduce a multimodal fusion approach named Graph and Attention based Two-stage Multi-source Information Fusion (GA2MIF) for emotion detection in conversation. Our proposed method circumvents the problem of taking heterogeneous graph as input to the model while eliminating complex redundant connections in the construction of graph. GA2MIF focuses on contextual modeling and cross-modal modeling through leveraging Multi-head Directed Graph ATtention networks (MDGATs) and Multi-head Pairwise Cross-modal ATtention networks (MPCATs), respectively. Extensive experiments on two public datasets (i.e., IEMOCAP and MELD) demonstrate that the proposed GA2MIF has the capacity to validly capture intra-modal long-range contextual information and inter-modal complementary information, as well as outperforms the prevalent State-Of-The-Art (SOTA) models by a remarkable margin.

I. INTRODUCTION

Conversational emotion recognition matters for empathetic interaction, but unimodal approaches can miss emotional cues and existing multimodal methods have limitations in cross-modal interaction and graph construction.

A. Emotion Recognition

  • ERC detects a speaker’s emotional state from expressed signals and has potential applications in diagnosis, opinion mining, conversation generation, and recommender systems.The paper links ERC systems to customized services and more empathetic interactions.Source: §I · p. 1

1) Unimodal-Based Emotion Recognition:

  • Unimodal ERC work includes text-based conversational models and acoustic speech-emotion methods, but relying on one signal source may limit recognition when that modality’s signal conflicts with the emotional state.Examples include DialogueGCN, HiGRU, COSMIC, and DialogueCRN for text, and LSTM- and CNN-based approaches for speech.Source: §I · pp. 1–2

I. INTRODUCTION

  • Multimodal ERC methods have used direct feature concatenation, while MMGCN models modalities in a heterogeneous graph and connects all utterances, introducing redundant connections.The paper notes that GNNs can perform suboptimally on heterogeneous networks.Source: §I · p. 2

A. Emotion Recognition

  • Emotion recognition is studied across emotion-understanding systems, opinion mining, and emotion generation, with work broadly divided into unimodal- and multimodal-based approaches.Recognizing emotional states is presented as important for empathic responses and human-computer interaction.Source: §I · p. 1, A. Emotion Recognition · p. 2

2) Multimodal-Based Emotion Recognition:

Prior multimodal ERC systems combine modalities in varied ways, while this background section introduces graph neural networks and attention as tools for modeling contextual and cross-modal information.

2) Multimodal-Based Emotion Recognition:

  • Multimodal ERC approaches include sequence, memory, recurrent, and graph models, with examples modeling context, speaker characteristics, or multimodal dependencies.The cited approaches include BC-LSTM, CMN, DialogueRNN, ConGCN, and MMGCN.Source: 2) Multimodal-Based Emotion Recognition · p. 3

B. Machine Learning Methods

1) Graph Neural Networks:

  • Graph neural networks update graph-structured representations through message passing, and graph attention networks weight neighboring nodes according to their importance to the current node.GAT contrasts with methods that assign identical or predefined neighbor contributions.Source: 1) Graph Neural Networks · pp. 3–4

2) Multi-Head Attention Mechanism:

The paper introduces multi-head attention, outlines GA2MIF’s processing stages, and defines conversational emotion recognition alongside intra-modal context and inter-modal complementary information.

2) Multi-Head Attention Mechanism:

  • Multi-head attention uses separate query, key, and value projections across heads, then concatenates their outputs into a learned representation.The section presents it as a mechanism that can enhance the stability and performance of single attention.Source: 2) Multi-Head Attention Mechanism · p. 4

III. METHODOLOGY

  • GA2MIF’s overall framework consists of unimodal pre-encoding, two-stage multi-source information fusion, and classification.Source: §III · p. 4

A. Problem Definitions

  • ERC maps each utterance to an emotion label; its context includes earlier and later utterances, while other modalities’ representations provide complementary information.Nearby utterances are local context and more distant utterances are long-range context; recurrence-based methods commonly have difficulty capturing the latter.Source: A. Problem Definitions · p. 4

B. Unimodal Pre-Encoding

The method pre-encodes text with a BiLSTM and acoustic and visual features with fully connected networks, then adds speaker embeddings to utterance features. These representations feed graph-based contextual and attention-based cross-modal modeling.

B. Unimodal Pre-Encoding

  • Text is encoded with a BiLSTM, while acoustic and visual modalities use fully connected networks.The paper uses the same unimodal encoding method as Hu et al. [15] for its comparison with MMGCN.Source: B. Unimodal Pre-Encoding · p. 4

C. Two-Stage Multi-Source Information Fusion

1) Characteristics of the Speaker:

  • Speaker embeddings are added to the corresponding utterance features to encode speaker characteristics.The speaker embedding is derived from the set of speakers and added to each modality’s pre-encoded feature matrix using a trade-off parameter.Source: 1) Characteristics of the Speaker · p. 5

2) Creation of Directed Graph:

GA2MIF builds a separate directed graph for each modality, connecting utterances to a bounded window of past and future context. MDGATs use attention-weighted message passing and layer updates to model contextual dependencies.

2) Creation of Directed Graph:

  • Each modality has a directed graph whose utterance nodes connect to past and future utterances within fixed context windows.The graph construction uses three modality-specific graphs; the window sizes determine the number of past and future neighbors.Source: 2) Creation of Directed Graph · p. 5
  • GATv2-based attention assigns edge weights, while fixed-size neighborhoods improve computational efficiency and model stability.The paper contrasts this scoring with standard GAT, whose consecutive linear transformations limit attention-function expressiveness.Source: 2) Creation of Directed Graph · p. 5

3) Graph-Based Contextual Modeling:

  • Three MDGATs aggregate neighboring utterance information with attention and update each layer using the prior representation and message-passing output.The graph model includes a skip connection from the previous layer input to the next layer input to alleviate GNN over-smoothing.Source: 3) Graph-Based Contextual Modeling · p. 6
  • The state-update options are sum, concatenation, and Sum-Product, which adds element-wise products to vector sums.The authors intend Sum-Product updating to pass more messages from similar contextual neighbors to the current node.Source: 3) Graph-Based Contextual Modeling · p. 6

4) Attention-Based Cross-Modal Modeling:

MPCATs take modality features produced by MDGATs and model cross-modal interactions with pairwise multi-head attention. Repeated interactions yield updated features for each modality.

4) Attention-Based Cross-Modal Modeling:

  • Each MPCAT receives two modalities’ MDGAT outputs and combines pairwise multi-head attention with a feedforward layer.The MPCAT structure contains two multi-head attention layers and one feedforward layer.Source: 4) Attention-Based Cross-Modal Modeling · p. 6
  • For textual features, acoustic and visual queries attend separately to textual keys and values, and the resulting interactions are added.Residual connections and normalization are applied before the feedforward layer and again afterward.Source: 4) Attention-Based Cross-Modal Modeling · pp. 6–7
  • The cross-modal interaction process is repeated across MPCAT layers, producing updated textual, acoustic, and visual feature matrices.Each layer’s outputs become the next layer’s inputs, and the three modality matrices are encoded by their respective MPCATs.Source: 4) Attention-Based Cross-Modal Modeling · p. 7

D. Multimodal Emotion Classification

After cross-modal encoding, the method concatenates the final textual, acoustic, and visual feature matrices into a joint representation. A classifier uses that representation to predict utterance emotions.

D. Multimodal Emotion Classification

  • The final textual, acoustic, and visual feature matrices are concatenated into a joint feature matrix for emotion classification.The joint representation is used as input to the multimodal classification module.Source: D. Multimodal Emotion Classification · p. 7
  • For each utterance, the classifier transforms its feature vector with ReLU and Softmax to produce an emotion distribution and predicted label.The predicted label is the class with the maximum probability.Source: D. Multimodal Emotion Classification · p. 7

E. Training Objective

GA2MIF is trained with a cross-entropy objective and L2 regularization, using predicted emotion distributions and ground-truth labels for utterances.

E. Training Objective

  • GA2MIF uses cross-entropy with L2 regularization as its training objective.The L2 regularizer is weighted by η and applies to the learnable parameters.Source: E. Training Objective · p. 7
  • The objective compares each utterance’s predicted emotion-label distribution with its ground-truth label.The passage defines these as p_ij and y_ij for utterance j in conversation i.Source: E. Training Objective · p. 7
  • The loss uses the number of utterances in each conversation and the total number of training conversations in its formulation.These quantities are denoted n(i) and N, respectively.Source: E. Training Objective · p. 7

A. Dataset Descriptions

The experiments use IEMOCAP and MELD, compare GA2MIF with unimodal and multimodal baselines, and report dataset-specific implementation settings.

A. Dataset Descriptions

  • Experiments evaluate multimodal ERC on IEMOCAP and MELD using text, acoustic, and visual features extracted with TextCNN, OpenSmile, and DenseNet, respectively.The feature-extraction setup follows Hu et al. [15].Source: A. Dataset Descriptions · p. 8
  • IEMOCAP contains 151 conversations and 7,433 utterances with six labels, whereas MELD has over 1,400 dialogues, over 13,000 utterances, and seven labels.IEMOCAP is dyadic; MELD has three or more participants per conversation and is drawn from Friends.Source: A. Dataset Descriptions · p. 8

B. Comparison Models

  • The baselines include unimodal and multimodal ERC models, including MFN, BC-LSTM, DialogueGCN, DialogueCRN, and MMGCN.The compared approaches span different strategies for fusion, contextual modeling, and multimodal interaction.Source: B. Comparison Models · p. 8

C. Implementation Details

  • All models use AdamW, a 0.00001 L2 factor, and 0.1 dropout, with a maximum of 100 epochs.GA2MIF uses layer normalization; its dataset-specific depths and speaker-embedding trade-off settings differ between IEMOCAP and MELD.Source: C. Implementation Details · p. 8
  • For IEMOCAP, GA2MIF uses 3 MDGAT and 4 MPCAT layers with λ = 1.6; for MELD it uses 2 of each with λ = 0.6.Both datasets use a learning rate of 0.00001; batch sizes are 8 for IEMOCAP and 32 for MELD.Source: C. Implementation Details · p. 8

V. COMPARISON AND ANALYSIS

GA2MIF is compared with baselines and analyzed through parameter, graph-window, and update-function variations. It achieves the strongest reported overall scores on both datasets, while the analyses show dataset- and setting-dependent effects.

A. Comparison With Baseline Models

  • On IEMOCAP, GA2MIF achieved 69.75% accuracy and 70.00% F1, exceeding MMGCN by 4.19% and 4.29%, respectively; on MELD, it improved accuracy and F1 by 2.34% and 1.12%, respectively.The authors note that neighboring MELD utterances may not be consecutive in real conversations, making it challenging to benefit from the model without a powerful feature extractor.Source: A. Comparison With Baseline Models · p. 9, Table III · p. 9
  • GA2MIF has the highest IEMOCAP F1 for most emotion labels except Happy, and its confusion-matrix accuracy exceeds MMGCN’s except for Frustrated.The paper reports 81.22% accuracy for GA2MIF on Sad.Source: A. Comparison With Baseline Models · p. 9, Figure 4 · p. 9

B. Impact of Different Trade-Off Parameters

  • Performance rises with speaker-embedding trade-off λ before declining beyond a threshold, with IEMOCAP’s threshold at λ = 1.6 and MELD improving within λ < 0.6.The reported measures are accuracy and weighted-average F1; the figure distinguishes results for the two datasets.Source: B. Impact of Different Trade-Off… · pp. 9–10, Figure 5 · p. 10

C. Impact of Different Window Sizes in Directed Graph

D. Impact of Different Updating Functions

E. Impact of Different Number of Network Layers

Across layer-depth, modality, and case-study analyses, GA2MIF benefits from both MDGATs and MPCATs, performs best with three modalities, and reduces a reported Sad-to-Frustrated confusion relative to MMGCN.

E. Impact of Different Number of Network Layers

F. Performance under Different Modality Settings

G. Case Studies

  • For difficult non-Neutral utterances, GA2MIF processes modalities separately, then models cross-modal interaction and uses a context window instead of fully connected graph edges.Source: G. Case Studies · p. 12
  • GA2MIF identifies Sad as Frustrated with 7.76% probability, versus 15.51% for MMGCN, a 7.75% reduction.Source: G. Case Studies · p. 12

VI. SUMMARY AND PROSPECT

  • 70.00% weighted-average F1 on IEMOCAP, with GA2MIF outperforming all baseline models.Source: §VI · p. 12
  • Future work will enhance acoustic and visual modality expressiveness, extend multimodal fusion to more scenarios, and address similar-emotion and emotion-shifting problems.Source: §VI · p. 12

Related papers

Loading 2207.11900v6…