Source-linked AI summary
Modality to Modality Translation: An Adversarial Representation Learning and Graph Fusion Network for Multimodal Fusion
Sijie Mai, Haifeng Hu, Songlong Xing
TL;DR
Multimodal fusion is challenged by heterogeneous modality distributions and incomplete joint embedding spaces. ARGF uses adversarial distribution translation with reconstruction and classification constraints, followed by hierarchical graph fusion; it reports state-of-the-art performance across multiple datasets and discriminative embeddings.
Problem
Heterogeneous modality distributions make complementary information difficult to mine, while many fusion methods do not learn a joint embedding space for modalities.
Method
ARGF uses adversarial encoders to match source modalities to a target distribution, decoders and a classifier to constrain embeddings, and hierarchical graph fusion for unimodal, bimodal, and trimodal dynamics.
Results
ARGF achieves state-of-the-art performance on three multimodal learning datasets and produces discriminative learned embeddings.
Takeaways & Limitations
Matching modality distributions before fusion enables direct representation fusion, while graph fusion successively models unimodal, bimodal, and trimodal dynamics.
Takeaways & Limitations
The graph interaction weighting assumes that similar information vectors contain less complementary information because their information has already been explored.
Abstract
from arXiv · showhide
Learning joint embedding space for various modalities is of vital importance for multimodal fusion. Mainstream modality fusion approaches fail to achieve this goal, leaving a modality gap which heavily affects cross-modal fusion. In this paper, we propose a novel adversarial encoder-decoder-classifier framework to learn a modality-invariant embedding space. Since the distributions of various modalities vary in nature, to reduce the modality gap, we translate the distributions of source modalities into that of target modality via their respective encoders using adversarial training. Furthermore, we exert additional constraints on embedding space by introducing reconstruction loss and classification loss. Then we fuse the encoded representations using hierarchical graph neural network which explicitly explores unimodal, bimodal and trimodal interactions in multi-stage. Our method achieves state-of-the-art performance on multiple datasets. Visualization of the learned embeddings suggests that the joint embedding space learned by our method is discriminative. code is available at: \url{https://github.com/TmacMai/ARGF_multimodal_fusion}
Introduction
The paper targets multimodal fusion’s modality gap by learning a shared embedding space and hierarchically modeling interactions among acoustic, language, and visual modalities.
- Multimodal fusion must address heterogeneous modality distributions to mine complementary information for comprehensive interpretation.
- The proposed adversarial encoder framework matches transformed modality distributions to learn a modality-invariant embedding space.Source encoders transform unimodal features and attempt to make their representations resemble those from a target modality under adversarial training.
- Prior methods do not explicitly model interactions among every subset of modalities through a hierarchical fusion process.
- The graph fusion network represents unimodal, bimodal, and trimodal dynamics in successive layers, passing lower-layer information upward for fusion.
- ARGF adds decoders and a classifier to retain unimodal information and improve embedding discrimination while matching multimodal distributions.
- ARGF achieves state-of-the-art performance on three multimodal learning datasets and provides embedding and graph-fusion visualizations.
Related Work
Related multimodal fusion work includes early, late, local, tensor, and modality-translation strategies, while GAN-based methods motivate distribution matching for shared representations.
- Early fusion concatenates modality features at the input level, whereas late fusion combines modality-level decisions through weighted averaging.
- Local fusion methods model time-dependent cross-modal interactions, including memory-based approaches and fusion strategies that combine bimodal and unimodal dynamics.
- Tensor Fusion Network uses an outer product of multimodal embeddings, while later methods target efficiency, redundancy reduction, or combined local and global fusion.
- Multimodal Transformer translates source modalities to target modalities with directional pairwise cross-modal attention, while MCTN learns joint representations through encoder-decoder translation.
- GAN-based approaches motivate matching different distributions, including work that learns common representations for heterogeneous multimodal data.
Model Architecture
ARGF learns a shared, modality-invariant embedding space before fusing language, acoustic, and visual representations with a hierarchical graph network. Adversarial, reconstruction, and classification objectives constrain the embeddings, while GFN models unimodal, bimodal, and trimodal interactions.
- ARGF uses two stages: adversarial joint embedding-space learning followed by graph fusion of encoder representations.
- Joint Embedding Space Learning: Adversarial training translates source-modality distributions toward the target distribution, narrowing the modality gap for direct fusion.
- Joint Embedding Space Learning: Decoders reconstruct language, acoustic, and visual inputs to retain unimodal information after distribution transformation.
- Joint Embedding Space Learning: A classifier imposes label information on encoded representations, making the learned embedding space discriminative for the task.
- Joint Embedding Space Learning: The total objective combines fake adversarial, reconstruction, true adversarial, and classification losses with staged parameter updates.
- Graph Fusion Network: GFN hierarchically represents unimodal, bimodal, and trimodal dynamics, using attention, MLP fusion, and similarity-based edge weighting.
Experiments
ARGF is evaluated on CMU-MOSI, IEMOCAP, and CMU-MOSEI, with comparisons covering performance, architecture ablations, fusion strategies, complexity, target modality, and embedding interpretability.
- Datasets and metrics: ARGF uses CMU-MOSI, IEMOCAP, and CMU-MOSEI, with sentiment or emotion accuracy and F1 score reported according to dataset task.CMU-MOSI uses 1,141 training, 306 validation, and 752 test segments; IEMOCAP reports recognition accuracy for each emotion.
- Benchmark performance: ARGF outperforms prior methods across the evaluated datasets, including about 0.5% over LMFN on CMU-MOSI and about 2% over state-of-the-art methods in IEMOCAP average accuracy.On CMU-MOSEI, ARGF also obtains the highest reported average accuracy and F1 score among the compared methods.
- Complexity analysis: ARGF has 1,270,770 trainable parameters and requires 2,017,645 testing FLOPs, compared with 8,491,844 FLOPs for TFN and 1,322,044 for BC-LSTM.Its parameter count is approximately 29.93% of TFN’s and 91.83% of BC-LSTM’s.
- Target modality: Language is the best target modality across the reported datasets, but acoustic and visual modalities perform closely, so no hard-and-fast optimal choice is established.The target modality serves as a distribution provider for source modalities’ transformed representations.
- Ablation studies: Removing any ARGF component decreases performance, while removing decoders causes the largest decline at more than 1.5%, followed by removing adversarial training.Removing the classifier decreases performance by around 0.4% but weakens the discriminative embedding space.
- Fusion strategies: GFN outperforms DFG by over 2% on IEMOCAP and 1% on CMU-MOSI by explicitly modeling unimodal, bimodal, and trimodal dynamics.The framework’s Tensor Fusion implementation reaches 59.46% accuracy on IEMOCAP versus 54.28% for TFN.
- Embedding visualization: ARGF embeddings form mostly separated positive and negative sentiment clusters, whereas embeddings without the classifier cannot distinguish sentiments effectively.Some samples remain in incorrect clusters, motivating more advanced fusion strategies for cross-modal dynamics.
- Graph-fusion interpretability: Graph-fusion weights show language is usually most predictive for unimodal interactions, while two-bimodal-vertex combinations dominate trimodal information.The visualization supports modeling pairwise bimodal interactions within hierarchical fusion.
Conclusions
The framework first learns a discriminative joint embedding space through adversarial distribution matching, then performs graph fusion across unimodal, bimodal, and trimodal dynamics.
- ARGF learns a discriminative joint embedding space by matching modality distributions adversarially before graph fusion.