PaperlayerView exact revision on arXiv

Source-linked AI summary

GraphCFC: A Directed Graph Based Cross-Modal Feature Complementation Approach for Multimodal Conversational Emotion Recognition

Jiang Li, Xiaoping Wang, Guoqing Lv, Zhigang Zeng

arXiv:2207.12261v4cs.CLcs.LGcs.MM

TL;DR

Existing multimodal fusion can add redundant information and risk losing diverse features, despite the value of contextual and cross-modal cues for conversational emotion recognition. GraphCFC uses a directed graph, multiple subspaces, PairCC, and GAT-MLP; it outperforms other SOTA methods on IEMOCAP and MELD.

  • Problem

    Directly placing modalities into a graph for fusion may add redundancy and risk losing diverse information because modality distributions are inconsistent.

    Source: §I · p. 2

  • Method

    GraphCFC uses a directed multimodal dialogue graph, multiple subspace mappings, PairCC, and the GAT-MLP network structure.

    Source: §I · p. 2, §VI · p. 11

  • Results

    GraphCFC outperforms other SOTA methods on the IEMOCAP and MELD datasets.

    Source: §VI · p. 12

  • Takeaways & Limitations

    The reported results indicate that GraphCFC models long-distance intra-modal context and inter-modal interaction on IEMOCAP and MELD.

    Source: §VI · p. 12

Abstract

from arXiv · show

Emotion Recognition in Conversation (ERC) plays a significant part in Human-Computer Interaction (HCI) systems since it can provide empathetic services. Multimodal ERC can mitigate the drawbacks of uni-modal approaches. Recently, Graph Neural Networks (GNNs) have been widely used in a variety of fields due to their superior performance in relation modeling. In multimodal ERC, GNNs are capable of extracting both long-distance contextual information and inter-modal interactive information. Unfortunately, since existing methods such as MMGCN directly fuse multiple modalities, redundant information may be generated and diverse information may be lost. In this work, we present a directed Graph based Cross-modal Feature Complementation (GraphCFC) module that can efficiently model contextual and interactive information. GraphCFC alleviates the problem of heterogeneity gap in multimodal fusion by utilizing multiple subspace extractors and Pair-wise Cross-modal Complementary (PairCC) strategy. We extract various types of edges from the constructed graph for encoding, thus enabling GNNs to extract crucial contextual and interactive information more accurately when performing message passing. Furthermore, we design a GNN structure called GAT-MLP, which can provide a new unified network framework for multimodal learning. The experimental results on two benchmark datasets show that our GraphCFC outperforms the state-of-the-art (SOTA) approaches.

I. INTRODUCTION

ERC predicts speakers’ emotions in conversation, and multimodal inputs offer textual, acoustic, and visual cues. GraphCFC addresses limitations in existing multimodal fusion with a directed graph-based approach and outperforms prior SOTA methods on two benchmark datasets.

A. Emotion Recognition in Conversation

  • ERC predicts a speaker’s emotional state during conversation and supports applications including empathetic services, medical diagnosis, opinion mining, and dialogue generation.The paper describes contextual ERC methods using graph and recurrent networks, alongside multimodal methods using multiple information sources.Source: §I · p. 1, A. Emotion Recognition in Conversation · pp. 2–3

I. INTRODUCTION

  • Single-modality ERC can fail when its signals lack a significant emotional signature, while recurrence-based approaches may struggle to gather long-distance information.The paper also notes that some multimodal approaches concatenate modalities without modeling their interaction.Source: §I · pp. 1–2
  • MMGCN captures long-distance context and inter-modal interaction, but connecting each node to all others may add redundancy and its two edge types may hinder selection of important information.Directly treating different modalities as the same node type may also add redundant information and risk losing diverse information.Source: §I · p. 2
  • GraphCFC models multimodal dialogue as a directed heterogeneous graph, uses multiple subspace extractors and PairCC, and proposes GAT-MLP for multimodal fusion.The approach extracts distinct edge types to select contextual and interactive information and is intended to alleviate the heterogeneity gap while preserving diverse features.Source: §I · p. 2
  • GraphCFC outperforms previous SOTA approaches in comparative experiments on two benchmark datasets.The reported results describe productive complementation and fusion of multimodal features.Source: §I · p. 2

B. Multimodal Fusion

Multimodal ERC combines conversational context with multiple modalities, while multimodal fusion faces heterogeneity between their feature distributions. The proposed method encodes each modality, complements cross-modal features with a graph-based module, and classifies emotions.

B. Multimodal Fusion

  • Multimodal fusion faces a heterogeneity gap between modalities, and multimodal ERC concerns conversations with two or more participants rather than utterance-level prediction alone.In this task, an utterance’s emotion is influenced by both the speaker’s expression and others’ expressions.Source: B. Multimodal Fusion · p. 3

C. Graph Neural Networks

  • Graph neural networks model non-Euclidean data through information propagation, with spatial approaches noted for their efficiency, flexibility, and generality.The paper identifies Graph-SAGE, GAT, and FastGCN as widely used techniques.Source: C. Graph Neural Networks · p. 3

III. PROPOSED METHODS

  • The proposed ERC approach comprises a general overview, uni-modal encoder, GraphCFC module, and multimodal emotion classifier.The section introduces this graph-based feature-fusion method in response to multimodal emotion-recognition challenges.Source: §III · p. 3

1) Problem Definition:

  • Multimodal ERC predicts an emotion label for each utterance using inter-modal interaction and intra-modal context.The paper describes textual, acoustic, and visual modalities and reports six IEMOCAP labels and seven MELD labels, with coarsened labels also explored.Source: 1) Problem Definition · pp. 3–4

2) Overall Architecture:

  • The pipeline encodes uni-modal features, uses a GNN-based GraphCFC module to collect long-distance intra-modal context and inter-modal interaction, then classifies emotions with multiple loss functions.The classifier is described as a multitask-learning model.Source: 2) Overall Architecture · p. 4, Figure 2 · p. 4

B. Uni-Modal Encoder

  • The textual modality uses a BiLSTM for context-aware feature pre-extraction, while acoustic and visual features use fully connected networks.The stated encoder choices follow MMGCN.Source: B. Uni-Modal Encoder · p. 4

C. Graph Based Cross-Modal Feature Complementation Module

GraphCFC combines intra-modal context with inter-modal interaction in a multimodal dialogue graph. Its edge types distinguish contextual links by speaker and cross-modal links within an utterance.

1) Graph Construction:

C. Graph Based Cross-Modal Feature Complementation Module

  • The module is designed to preserve multimodal consistency and diversity while selecting contextual and interactive information and addressing the heterogeneity gap.The paper also presents GraphCFC as a network model for visual, acoustic, and textual modalities.Source: C. Graph Based Cross-Modal Feature… · p. 4

1) Graph Construction:

  • Each utterance in each modality is represented as a node, and graph edges encode relational dependencies between nodes.With M modalities and n utterances, the graph has M × n nodes.Source: 1) Graph Construction · pp. 4–5
  • Intra-edges are subdivided by speaker identity, while inter-edges represent pairs of modalities within the same utterance.For D speakers, the graph has D×(D+1)/2 intra-edge relations; with M modalities, there are M × (M −1)/2 inter-edge relations per utterance.Source: 1) Graph Construction · p. 5

2) Multi-Subspace Extractor:

The multi-subspace extractor maps modalities into shared and separate representations to capture consistency and diversity. Speaker embeddings augment utterance features, while subspace losses keep extracted features aligned with the emotion-label task.

2) Multi-Subspace Extractor:

  • Speaker embeddings are added to utterance features to encode speaker identity.The speaker-embedding contribution is controlled by a ratio μ ∈[0, 1].Source: 2) Multi-Subspace Extractor · p. 5
  • Mapping modalities into a shared subspace is intended to address inconsistent feature distributions, while separate subspaces preserve modality-specific diversity for complementation.The paper notes that greater similarity between modality representations can mean less complementary information.Source: 2) Multi-Subspace Extractor · p. 5
  • Shared mappings use the same trainable parameters across modalities to capture consistency, while separate mappings use modality-specific parameters to capture diversity.The modalities are textual, acoustic, and visual.Source: 2) Multi-Subspace Extractor · p. 5
  • Four subspace loss functions constrain extracted features so they remain directed toward the emotion-label task.The passage introduces shared and separate subspace losses.Source: 2) Multi-Subspace Extractor · p. 6

3) GAT-MLP Layer:

GAT-MLP combines graph attention and a feedforward sublayer in a GNN layer inspired by ResNet and Transformer. The two sublayers are presented as complementary for contextual and feature-level encoding across modalities.

3) GAT-MLP Layer:

  • The design addresses the over-smoothing concern that repeated graph convolutions can make node features within a connected component similar.The paper describes model performance as deteriorating when GNN depth reaches a threshold.Source: 3) GAT-MLP Layer · p. 6
  • GAT-MLP combines a multi-head graph attention network with a feedforward sublayer, drawing on ideas from ResNet and Transformer.The layer can place normalization before MultiGAT and FeedForward.Source: 3) GAT-MLP Layer · pp. 6–7
  • MultiGAT captures sequence-level contextual information, while FeedForward compensates for limitations in efficiently capturing feature-level information.The paper proposes their combination to compensate for encoding disparities across modalities.Source: 3) GAT-MLP Layer · p. 7

4) GAT-MLP Based PairCC:

PairCC applies GAT-MLP encoding in stages, first combining visual and acoustic features, then introducing text, and finally combining the result with shared-subspace features.

4) GAT-MLP Based PairCC:

  • PairCC first encodes visual and acoustic features together, then treats their concatenation as a new modality to encode with text, and finally combines the result with shared-subspace features.The stated aim is cross-modal interaction and reducing the heterogeneity gap associated with direct concatenation.Source: 4) GAT-MLP Based PairCC · p. 7

5) SingleGAT:

SingleGAT aggregates neighboring utterance features with attention and updates each node through a GRU-based combination. Edge-type information can also shape attention, and multiple heads are averaged to produce the layer output.

5) SingleGAT:

  • Attention weights neighbor features to aggregate contextual and inter-modal information for each node.The neighbors are intra-modal contextual nodes or inter-modal interactive nodes.Source: 5) SingleGAT · p. 7
  • GATv2 learns edge weights from node representations to characterize the relevance of neighboring information.The attention coefficient is computed over a node and its neighboring nodes.Source: 5) SingleGAT · p. 7
  • Edge-type embeddings let attention account for relation types as well as node features when weighting neighbors.The paper hypothesizes that different relations carry distinct dependencies, such as same-speaker context and inter-modal complementarity.Source: 5) SingleGAT · pp. 7–8
  • A GRU combines aggregated neighbor information with the current node feature, and the paper reverses their input and hidden-state roles to better exploit neighbor information.The reversed arrangement uses the aggregated neighbor information as input and the current feature as hidden state.Source: 5) SingleGAT · p. 8
  • The multi-head graph-attention output is obtained by averaging the outputs of multiple single-head attentions.The number of heads is denoted by K.Source: 5) SingleGAT · p. 8

D. Multimodal Emotion Classifier

The classifier predicts an emotion label for each encoded utterance and trains with classification and subspace losses. Its final objective combines the shared-subspace, separate-subspace, and classification losses.

D. Multimodal Emotion Classifier

  • The encoded feature for each utterance is passed to a fully connected layer to predict its emotion label.The encoded utterance feature is denoted hi.Source: D. Multimodal Emotion Classifier · p. 8
  • Cross-entropy with L2 regularization is used as the classification loss.The loss is defined over dialogues and their utterances, using ground-truth labels and predicted emotion distributions.Source: D. Multimodal Emotion Classifier · p. 8
  • The final objective combines the shared-subspace loss, separate-subspace losses, and classification loss.Separate-subspace losses are included for the acoustic, visual, and textual modalities.Source: D. Multimodal Emotion Classifier · p. 8

A. Datasets and Evaluation Metrics

GraphCFC is evaluated on IEMOCAP and MELD against prior ERC baselines, using accuracy and weighted-average F1. It outperforms MMGCN on both datasets, with larger reported gains on IEMOCAP than MELD.

1) Datasets:

  • IEMOCAP contains 151 two-way conversations and 7,433 utterances from ten professional actors; MELD contains over 1,433 multi-party conversations and 13,708 utterances from 304 speakers.IEMOCAP uses six emotion labels and MELD seven; the IEMOCAP split uses the first 80% for training and the remainder for testing, with 10% of training used for validation.Source: 1) Datasets · p. 8

2) Metrics:

  • Weighted-average F1 is the principal comparison metric because of class imbalance, with per-class F1 and average accuracy also reported.The metric choice follows prior methods,.Source: 2) Metrics · p. 8

B. Baselines

  • The comparison includes bc-LSTM, CMN, ICON, DialogueRNN, DialogueGCN, DialogueCRN, and MMGCN.The listed methods include recurrent, graph-based, and multimodal baselines.Source: B. Baselines · pp. 8–9

C. Implementation Details

  • Experiments use PyTorch on an NVIDIA Tesla A100 with AdamW, dropout 0.1, and a learnable trade-off among loss terms.GraphCFC uses five GAT-MLP layers on IEMOCAP and three on MELD; Kendall et al. is used to set loss trade-off parameters as learnable parameters.Source: C. Implementation Details · p. 9

A. Overall Performance

  • On IEMOCAP, GraphCFC exceeds MMGCN by 3.57% in accuracy and 3.20% in weighted-average F1, and it outperforms other models on both metrics.The reported F1 scores for Sad and Excited are 84.99% and 78.86%, respectively; their scores are 7.46% and 6.82% higher than MMGCN's.Source: A. Overall Performance · p. 9
  • On MELD, GraphCFC's accuracy and weighted-average F1 improvements over MMGCN are 2.11% and 1.04%, respectively.The authors suggest non-continuous dialogue sequences and background noise as possible reasons the gains are modest; Sadness is a minority class with low F1.Source: A. Overall Performance · p. 9

B. Comparison Under Various Modality Settings

GraphCFC performs best with multimodal inputs, while component studies show benefits from the GAT-MLP sublayers, skip connections, subspace losses, and edge-type and speaker embeddings. The reported effects vary by component and setting.

B. Comparison Under Various Modality Settings

  • Three-modality fusion improves GraphCFC over uni-modal settings, while text performs best alone and text-acoustic performs best among two-modality settings.Visual-acoustic is the weakest two-modality combination, and visual-only is the weakest uni-modal setting.Source: B. Comparison Under Various Modality… · p. 9, Table III · p. 10

C. Effect of Various Components in GAT-MLP Layer

  • On IEMOCAP, removing MultiGAT lowers accuracy by 4.93% and weighted-average F1 by 4.36%; removing FeedForward lowers them by 2.09% and 1.67%, respectively.The authors conclude that MultiGAT has the larger effect; the study reports results for GraphCFC without either GAT-MLP sublayer.Source: C. Effect of Various Components in… · pp. 9–10, Table IV · p. 10
  • On IEMOCAP, performance drops sharply without skip connections as GAT-MLP depth passes a threshold, whereas it decreases more slowly when skip connections remain.The authors state that skip connections help mitigate over-smoothing to a certain extent.Source: C. Effect of Various Components in… · p. 10, Figure 4 · p. 10

D. Effect of Multi-Subspace Loss

E. Effect of Multi-speaker and Edge Types

  • Without edge-type embeddings, IEMOCAP weighted-average F1 falls to 65.91%; removing speaker embeddings causes only slight degradation, with results still above baseline models.The authors report that using both speaker and edge-type embeddings can improve GraphCFC performance.Source: E. Effect of Multi-speaker and Edge… · pp. 10–11, Table VI · p. 11

F. Effect of the Past j and Future k Utterance Nodes

The section examines how context size affects GraphCFC, reports three-emotion results and IEMOCAP case studies, and concludes that GraphCFC outperforms other SOTA methods on IEMOCAP and MELD.

F. Effect of the Past j and Future k Utterance Nodes

G. Overall Performance of Three-Emotion

  • On MELD, GraphCFC improves accuracy by 0.61% and weighted-average F1 by 1.20% relative to MMGCN in the three-emotion setting.The authors note slight improvements on IEMOCAP as well and suggest coarsened emotion labels make classification easier for most models.Source: G. Overall Performance of Three-Emotion · p. 11, Table VIII · p. 12

H. Case Studies

  • On IEMOCAP, visual and acoustic modalities can compensate when text-only models label non-Neutral utterances as Neutral; GraphCFC also identifies Excited in two emotional-shift cases.The cases cover emotional shifts within one speaker and between two speakers.Source: H. Case Studies · p. 11, Figure 6 · p. 12

VI. CONCLUSION

  • GraphCFC outperforms other SOTA methods on IEMOCAP and MELD while modeling long-distance intra-modal context and inter-modal interaction.The conclusion attributes these results to a directed graph with variable context, distinct edge types, multiple subspace mappings, and PairCC.Source: §VI · p. 12
  • The authors identify future work in evaluating GAT-MLP on other multimodal tasks and addressing class imbalance and background noise on MELD.Source: §VI · p. 12
Loading 2207.12261v4…