PaperlayerView exact revision on arXiv

Source-linked AI summary

GCNet: Graph Completion Network for Incomplete Multimodal Learning in Conversation

Zheng Lian, Lan Chen, Licai Sun, Bin Liu, Jianhua Tao

arXiv:2203.02177v2cs.LGcs.CL

TL;DR

Incomplete modalities hinder conversation understanding, and existing approaches mainly target individual utterances rather than conversational temporal and speaker structure. GCNet uses Speaker and Temporal GNNs with joint classification and reconstruction, and experiments on three benchmark datasets report superior classification and imputation performance.

  • Problem

    Existing incomplete-multimodal methods mainly address individual utterances or medical images, leaving conversational temporal and speaker information underused.

  • Method

    GCNet uses Speaker GNN and Temporal GNN modules to model speaker and temporal dependencies, jointly optimizing classification and reconstruction end to end.

  • Results

    GCNet consistently outperforms advanced approaches under varying missing rates, achieving the best classification and imputation performance on three benchmark datasets.

  • Takeaways & Limitations

    GCNet provides a graph-based framework for learning from modality-incomplete conversational data while exploiting speaker and temporal information.

  • Takeaways & Limitations

    The method assumes modality-complete data is available during training; collecting no such data is left for future work.

Abstract

from arXiv · show

Conversations have become a critical data format on social media platforms. Understanding conversation from emotion, content and other aspects also attracts increasing attention from researchers due to its widespread application in human-computer interaction. In real-world environments, we often encounter the problem of incomplete modalities, which has become a core issue of conversation understanding. To address this problem, researchers propose various methods. However, existing approaches are mainly designed for individual utterances rather than conversational data, which cannot fully exploit temporal and speaker information in conversations. To this end, we propose a novel framework for incomplete multimodal learning in conversations, called "Graph Complete Network (GCNet)", filling the gap of existing works. Our GCNet contains two well-designed graph neural network-based modules, "Speaker GNN" and "Temporal GNN", to capture temporal and speaker dependencies. To make full use of complete and incomplete data, we jointly optimize classification and reconstruction tasks in an end-to-end manner. To verify the effectiveness of our method, we conduct experiments on three benchmark conversational datasets. Experimental results demonstrate that our GCNet is superior to existing state-of-the-art approaches in incomplete multimodal learning. Code is available at https://github.com/zeroQiaoba/GCNet.

1 INTRODUCTION

Incomplete modalities make conversation understanding difficult, while existing methods largely target individual utterances or medical images and overlook conversational temporal and speaker information. GCNet addresses this gap with graph-based modeling and joint classification-reconstruction optimization.

  • Missing speech, text, or faces caused by noise, sensor failure, recognition errors, lighting, motion, or occlusion complicates conversation understanding.
  • Existing incomplete-modality approaches mainly target individual utterances or medical images rather than conversational data.
  • Conversational data contains temporal and speaker dependencies that existing works often fail to exploit.
  • GCNet uses Speaker GNN and Temporal GNN modules to capture speaker and temporal dependencies in incomplete conversations.The modules share edges but use different edge types.
  • GCNet jointly optimizes classification and reconstruction end to end and is evaluated on three benchmark conversational datasets.The authors report superiority to existing state-of-the-art approaches in incomplete multimodal learning.

2 RELATED WORKS

Related work on incomplete multimodal learning includes imputation and non-imputation strategies. These approaches estimate missing data, exploit multimodal correlations, partition incomplete samples, or learn representations without encoders.

  • Imputation methods estimate missing data from partially observed inputs before, or alongside, downstream learning.
  • Imputation Methods: Imputation methods include zero or average filling, low-rank imputation, and DNN-based models such as autoencoders, GANs, VAEs, and Transformers.
  • Non-imputation Methods: Grouping strategies partition incomplete data into complete subgroups, but the number of subgroups grows exponentially with the number of modalities.
  • Non-imputation Methods: Correlation-maximization methods constrain different modalities of the same sample to have related low-dimensional representations.
  • Non-imputation Methods: Encoderless models directly optimize latent representations to reconstruct modality-incomplete data regardless of missing patterns.

3 METHODOLOGY

GCNet models incomplete conversational multimodal data by representing utterances as graph nodes and separately capturing speaker and temporal dependencies. It jointly trains classification and modality reconstruction to use incomplete inputs while learning from complete data.

  • 3.1 Data Preparation: Each conversation is represented as a sequence of utterances with labels, speakers, and acoustic, lexical, and visual features.Missing modalities are randomly discarded while guaranteeing at least one available modality per utterance.
  • 3.2 Method Overview: GCNet uses three modules: Node Construction, Speaker & Temporal GNN, and Classification & Reconstruction.The framework targets classification on conversational data with missing modalities.
  • 3.2.1 Node Construction: Utterances become graph nodes whose incomplete multimodal features are processed by Bi-LSTM to produce contextual initial representations.The resulting node representations are then used by the graph modules.
  • 3.2.2 Speaker & Temporal GNN: SGNN captures speaker-sensitive dependencies, while TGNN captures temporal dependencies using distinct edge types over shared graph edges.Speaker types encode speaker relations, and temporal types encode past, present, or future relative positions.
  • 3.2.2 Speaker & Temporal GNN: A context window limits each node’s interactions to nearby utterances, with the implementation selecting w from {1, 2, 3, 4}.This avoids connecting every utterance to all conversation nodes.
  • 3.2.2 Speaker & Temporal GNN: R-GCN aggregates neighborhood information with parameters dependent on edge types for both SGNN and TGNN.Each node receives separate speaker-aware and temporal-aware representations from these modules.
  • 3.2.3 Classification & Reconstruction: The model feeds latent representations to a fully connected layer and softmax for classification, while modality-specific linear transformations reconstruct complete features.Reconstruction is intended to guide learning of missing-part semantics.
  • 3.2.4 Joint Optimization: GCNet jointly optimizes cross-entropy classification loss and reconstruction loss at missing positions in an end-to-end objective.The joint loss optimizes all trainable parameters and leverages both complete and incomplete data.

4 EXPERIMENTAL DATABASES AND SETUP

Experiments evaluate incomplete multimodal conversation understanding on three benchmark datasets, using varying missing rates, WAF classification, and imputation metrics. The setup extracts acoustic, lexical, and visual features and compares GCNet with established incomplete-learning baselines.

  • Corpus Description: Experiments use IEMOCAP, CMU-MOSI, and CMU-MOSEI as benchmark conversational datasets.IEMOCAP supports four-class and six-class labeling, while CMU-MOSI and CMU-MOSEI provide sentiment scores between -3 and +3.
  • Evaluation Metrics: IEMOCAP is evaluated with weighted average F1-score because its categorical classes are naturally imbalanced.CMU-MOSI and CMU-MOSEI are converted to positive/negative classification using the sign of each sentiment score and also evaluated with WAF.
  • Implementation Details: Missing modalities are randomly selected with probability η while ensuring every sample retains at least one available modality.For three modalities, η ranges from 0.0 to 0.7, with the same missing rate used during training, validation, and testing.
  • Multimodal Feature Extraction: Acoustic, lexical, and visual utterance features are extracted using wav2vec-large, DeBERTa-large, and MA-Net-based pipelines, respectively.The extracted acoustic features are 512-dimensional, while lexical and compressed visual utterance features are 1024-dimensional.
  • Baselines: GCNet is compared with correlation-based, autoencoder-based, and completeness- or structure-oriented incomplete multimodal learning baselines.Baselines include CCA, DCCA, DCCAE, AE, CRA, MMIN, and CPM-Net.

5 RESULTS AND DISCUSSION

GCNet achieves the strongest classification results across the evaluated datasets and remains more robust as modality missingness increases. Its imputation performance also exceeds baselines across missing rates, supporting the use of temporal and speaker dependencies.

  • Classification Performance: GCNet achieves the best average classification performance across all evaluated datasets.On IEMOCAP four-class, it surpasses advanced approaches by 10.29%; on IEMOCAP six-class, it reaches 56.18%, an absolute improvement of 9.03%.
  • Classification Performance: As missing rates increase from 0.0 to 0.7, IEMOCAP four-class performance declines 6.98% for GCNet versus 13.24%–37.70% for baselines.The GCNet–baseline performance gap increases from 2.10% to 15.94% over the same missing-rate range.
  • Classification Performance: At η = 0.0, GCNet improves over advanced approaches by 2.10% on IEMOCAP four-class and 0.46% on CMU-MOSEI.The method is slightly below baselines on CMU-MOSI and IEMOCAP six-class, where the difference is described as not significant.
  • Imputation Performance: GCNet consistently outperforms all baselines in imputation under every missing rate on all datasets.The figure uses MSE, where lower values indicate better imputation performance.

5.2 Imputation Performance

GCNet improves imputation and classification on incomplete conversational data by exploiting speaker and temporal dependencies and using modality-incomplete samples during training. Its advantage over baselines and the Lower Bound grows as missing rates increase.

  • Imputation comparison: GCNet consistently outperforms AE and CRA in imputation MSE across all missing rates and datasets.The comparison evaluates mean square error between original and estimated features at missing positions.
  • Imputation comparison: Imputation performance declines as the missing rate increases, while GCNet’s advantage over baselines becomes more significant.The authors attribute the decline to fewer observed data and describe GCNet as more robust to missing data.
  • Importance of incomplete data: GCNet consistently outperforms the Lower Bound at every missing rate on all datasets.GCNet uses both modality-complete and modality-incomplete data, whereas the Lower Bound uses only complete data.
  • Importance of incomplete data: The performance gap widens under severe missingness because the Lower Bound discards more training samples, whereas GCNet retains incomplete samples.Using more available training samples supports learning when missing rates are high.

5.4 Role of SGNN and TGNN

Ablation results support modeling speaker and temporal dependencies separately in GCNet. Temporal information improves classification in most cases, speaker information improves performance across missing rates, and separate graphs generally outperform their coupled alternative.

  • Temporal GNN: GCNet improves over GCNet-S in most cases, supporting the value of TGNN for temporal-sensitive modeling.GCNet-S omits TGNN and therefore ignores temporal information.
  • Speaker GNN: GCNet outperforms GCNet-T at all missing rates, with a 0.31%∼2.35% absolute improvement on IEMOCAP(six-class).GCNet-T omits SGNN and ignores speaker information.
  • Experimental setup: The ablation compares GCNet, variants without SGNN or TGNN, and a coupled speaker-temporal graph model under varying missing rates.Table 4 reports classification performance on IEMOCAP, with bold values marking the best performance.
  • Separate dependency modeling: GCNet outperforms GCNet-ST in most cases, while GCNet-ST jointly models speaker-temporal dependencies in one graph.The authors associate the coupled design with more relation types to optimize under limited labeled data.

5.5 Parameter Tuning

GCNet’s performance depends on its context-window and latent-dimension settings. Both parameters show a rise-then-fall pattern, while classification and imputation losses descend toward convergence under η = 0.3.

  • Experimental setup: At missing rate η = 0.3 on IEMOCAP(four-class), parameter tuning evaluates w from {1, 2, 3, 4} and h from {50, 100, 200}.The study visualizes how these two user-specified parameters influence classification performance.
  • Context window: Classification performance first improves and then degrades as context window size w increases.The tested values are w ∈ {1, 2, 3, 4}; larger windows add edges and potentially irrelevant context.
  • Latent dimension: Classification performance first improves and then degrades as latent representation dimension h increases.The tested values are h ∈ {50, 100, 200}; larger dimensions increase trainable parameters and over-fitting risk.
  • Convergence: Classification and imputation losses maintain a generally descending trend and finally converge on training and validation sets.Training and validation curves converge at close epochs on IEMOCAP(four-class) with η = 0.3.

5.7 Visualization of Embedding Space

t-SNE visualizations indicate that GCNet produces clearer class separation and increasingly reveals class structure during training. A representative incomplete-conversation example also reports more accurate predictions when speaker and temporal information are incorporated.

  • Evaluation scope: The qualitative analysis evaluates latent representations on the IEMOCAP(four-class) test set with missing rate η fixed at 0.3.It includes baseline comparisons, increasing-iteration visualizations, and prediction results on incomplete conversational data.
  • Embedding separation: GCNet’s latent representations show clearer margins between classes than the baseline representations.The IEMOCAP(four-class) test-set visualization uses red, blue, green, and purple for happiness, sadness, neutral, and anger.
  • Training dynamics: As training iterations increase, GCNet’s latent representation reveals the underlying class distribution more clearly.The visualizations compare the 10th, 20th, 50th, and 78th… iterations at η = 0.3.
  • Prediction example: In a representative incomplete-conversation example, GCNet generates more accurate predictions than other methods.The example concerns a speaker sharing happiness after being accepted to college, with modalities marked as available or missing.

6 CONCLUSIONS

GCNet is presented as an effective framework for incomplete multimodal learning in conversations, outperforming advanced approaches across varying missing rates. The authors identify broader applications and dynamic edge construction as future directions.

  • GCNet consistently outperforms currently advanced approaches under varying missing rates on three benchmark datasets.The reported evaluation covers both classification and imputation performance.
  • GCNet achieves the best classification and imputation performance in the reported experiments.
  • Quantitative and qualitative analyses examine incomplete-data contributions, component necessity, hyper-parameter effects, and convergence.
  • Future work will extend GCNet beyond conversational emotion recognition and explore dynamic edge construction instead of fixed context-window limits.The proposed edge strategies would be based on correlations between utterances.
Loading 2203.02177v2…