Source-linked AI summary
Improving Multimodal Fusion with Hierarchical Mutual Information Maximization for Multimodal Sentiment Analysis
Wei Han, Hui Chen, Soujanya Poria
TL;DR
MSA fusion methods may fail to preserve task-related information as unimodal inputs become a fused representation. MMIM hierarchically maximizes MI at the input and fusion levels with tractable parametric and non-parametric estimators, and experiments on two datasets show superior or comparable results to state-of-the-art models. A case study also identifies a setting where misleading text can drive an incorrect prediction.
Problem
Multimodal fusion must integrate complementary modalities while preserving task-related information, but prior approaches can neglect information flowing from inputs to fusion results.
Method
MMIM jointly trains the MSA task with hierarchical MI maximization between unimodal pairs and between fusion results and unimodal representations, using parametric and non-parametric estimators.
Results
MMIM achieves superior or comparable results to state-of-the-art models across two datasets, significantly outperforming them on all metrics on CMU-MOSI and several metrics on CMU-MOSEI.
Takeaways & Limitations
The experiments and ablations verify the efficacy and necessity of the hierarchical MI-maximization framework for the reported MSA setting.
Takeaways & Limitations
A case study shows that the mechanism may malfunction when a misleading textual remark is unsupported by visual and acoustic cues, producing an opposite-direction error.
Abstract
from arXiv · showhide
In multimodal sentiment analysis (MSA), the performance of a model highly depends on the quality of synthesized embeddings. These embeddings are generated from the upstream process called multimodal fusion, which aims to extract and combine the input unimodal raw data to produce a richer multimodal representation. Previous work either back-propagates the task loss or manipulates the geometric property of feature spaces to produce favorable fusion results, which neglects the preservation of critical task-related information that flows from input to the fusion results. In this work, we propose a framework named MultiModal InfoMax (MMIM), which hierarchically maximizes the Mutual Information (MI) in unimodal input pairs (inter-modality) and between multimodal fusion result and unimodal input in order to maintain task-related information through multimodal fusion. The framework is jointly trained with the main task (MSA) to improve the performance of the downstream MSA task. To address the intractable issue of MI bounds, we further formulate a set of computationally simple parametric and non-parametric methods to approximate their truth value. Experimental results on the two widely used datasets demonstrate the efficacy of our approach. The implementation of this work is publicly available at https://github.com/declare-lab/Multimodal-Infomax.
1 Introduction
Multimodal sentiment analysis depends on multimodal fusion to integrate complementary visual, acoustic, and textual information while preserving task-related content. MMIM addresses this goal by hierarchically maximizing mutual information at input and fusion levels, using parametric and non-parametric estimators, and achieves superior or comparable results on two datasets.
- Motivation: MSA extracts emotional information from visual, acoustic, and textual data, whose complementary cues can disambiguate semantic and emotional content.Multimodal fusion integrates these inputs into a representation for sentiment prediction.
- MMIM: MMIM hierarchically maximizes mutual information between unimodal representations and between fusion results and low-level unimodal representations.The framework targets preservation of task-related information through multimodal fusion.
- MI Estimation: MMIM combines parametric neural-network methods with non-parametric estimation to address intractable mutual-information-bound terms.The non-parametric component uses a Gaussian mixture model, while the parametric component uses neural-network-based methods.
- Experiments: MMIM achieves superior or comparable results to state-of-the-art models on two publicly available multimodal sentiment-analysis datasets.The reported experiments are described as comprehensive and support the efficacy of the approach.
2 Related Work
Prior multimodal sentiment-analysis work used early fusion, geometric objectives, multitask learning, and cross-modal translation, while mutual-information research established MI maximization and variational approximation strategies. MMIM connects these directions by applying MI maximization between unimodal representations and between fusion embeddings and unimodal representations.
- Multimodal Sentiment Analysis: Earlier MSA systems used recurrent architectures and select-additive learning to process low-level multimodal inputs.These approaches were among the early fusion methods described in the related work.
- Multimodal Sentiment Analysis: Other MSA methods optimized geometric relationships between modality representations, jointly learned sentiment polarity and intensity, or cyclically translated between modalities.These methods pursued fusion or robustness through objectives beyond direct task-loss optimization.
- Mutual Information: MMIM applies MI maximization both between unimodal representations and between fusion embeddings and unimodal representations.This hierarchical design connects MI-based learning with multimodal fusion for downstream MSA.
- Mutual Information: Mutual-information maximization has been applied in deep learning and representation learning, but direct MI estimation in high-dimensional spaces is nearly impossible.Prior work therefore approximated MI using variational bounds.
3 Method
MMIM encodes text, visual, and acoustic inputs, then jointly optimizes multimodal fusion with hierarchical mutual-information objectives and the main sentiment task. It estimates inter-modality MI with a variational lower bound and models entropy using a sentiment-polarity GMM, while fusion-level MI encourages the fused representation to predict each modality.
- Modality Encoding: MMIM encodes text, visual, and acoustic sequences into unit-length modality representations before producing a unified fusion result for sentiment prediction.BERT encodes text, while modality-specific unidirectional LSTMs capture visual and acoustic temporal features.
- Hierarchical MI Maximization: Hierarchical MI maximization operates at the input and fusion levels alongside the main task loss to preserve task-related information in fusion results.The two objectives are optimized concurrently through back-propagation.
- Inter-modality MI Maximization: Inter-modality MI is optimized for text–visual and text–acoustic representation pairs using a tractable Barber–Agakov variational lower bound.Text is used as the predictor input because its representations are higher-dimensional than visual and acoustic representations.
- Inter-modality MI Maximization: The entropy term in the inter-modality MI bound is approximated with a Gaussian Mixture Model whose components correspond to non-negative and negative sentiment polarity classes.The formulation assumes equal prior probabilities for the two polarity classes and uses a lower entropy bound.
- Fusion-Level MI Maximization: Fusion-level MI trains the fusion result Z to reversely predict each modality representation, encouraging modality-invariant information to pass into Z.A normalized score and contrastive negative samples are used to gauge correlation between predictions and modality representations.
- Training Procedure: Training alternates conditional-likelihood maximization for predictors with joint MI-maximization training that estimates GMM statistics, computes losses, and updates model parameters.The procedure maintains an embedding-history memory for the non-parametric estimation component.
4 Experiments
Experiments evaluate MMIM on CMU-MOSI and CMU-MOSEI using established sentiment metrics and comparisons with diverse fusion baselines. MMIM achieves superior or comparable performance, while ablations support its MI losses and estimation components.
- Datasets and metrics: Experiments use CMU-MOSI and CMU-MOSEI, with sentiment labels and established regression, correlation, and classification metrics.The metric set includes MAE, Pearson correlation, Acc-7, Acc-2, and F1.
- Baselines: MMIM is compared with tensor, low-rank, factorization, transformer, feature-space, BERT-based, and self-supervised multimodal fusion baselines.The baselines include TFN, LMF, MFM, MulT, ICCN, MISA, MAG-BERT, and Self-MM.
- Results: MMIM significantly outperforms the state of the art on all CMU-MOSI metrics and on selected non-0 classification metrics on CMU-MOSEI.On other CMU-MOSEI metrics, its performance is within 0.5% of the state of the art.
- Ablation study: Removing part of the MI loss degrades performance, while removing all terms from one loss produces larger degradation than removing a single term.This ablation supports the contribution of the MI maximization framework.
- Ablation study: History-based entropy estimation prevents the NaN-gradient breakdown observed without memory, and polarity-class GMM estimation outperforms a unified Gaussian.Replacing the GMM with a unified Gaussian causes a clear drop across all metrics.
5 Further Analysis
The analysis traces training losses and examines representative predictions with fusion-modality scores. It finds distinct roles for the two MI losses, while showing that modality weighting can fail in misleading cases.
- 5.1 Tracing the Losses: Ltask and LCPC decrease throughout training, while LBA generally decreases after the beginning of training.The best validation epoch is marked by the minimum task loss on the validation set.
- 5.1 Tracing the Losses: LBA and LCPC provide supplemental unsupervised gradient rectification and help correct over-fitting of the task loss.
- 5.1 Tracing the Losses: LBA has a more significant impact on performance than LCPC under the reported experiment settings.At the best epoch, LBA reaches its lowest value in synchronization with validation loss, whereas LCPC does not.
- 5.2 Case Study: All three fusion-modality scores exceed 0.35 in the representative cases, indicating that fusion results retain domain-invariant features from the modalities.
- 5.2 Case Study: The model adjusts modality information according to individual predictive contributions, but misleading textual cues can produce an opposite-direction error.In case (D), “I’m sorry” draws attention to text while visual and acoustic inputs provide no corrective reminder.
6 Conclusion
The conclusion presents MMIM as a hierarchical MI-maximization framework for multimodal fusion, supported by experiments, ablations, visualized losses, and representative examples. It reports that the designed estimation methods support training and improve test outcomes, while the study motivates further work in representation learning and MSA.
- MMIM hierarchically maximizes MI at the unimodal-input and fusion stages of a multimodal fusion pipeline.
- The framework uses precise, fast, and robust estimation methods to address intractable terms in its MI lower bounds.
- Experiments on two datasets and an ablation study verify the efficacy of MMIM and the necessity of its MI-maximization framework.
- Loss visualizations and representative examples provide additional insight into how the model functions.
- The authors suggest that the work can inspire future creativity in representation learning and multimodal sentiment analysis.
A.1 Implementation details of history memory
The history embedding memory supports GMM parameter estimation by combining stored historical embeddings with the current batch, then updates storage as a FIFO queue.
- The history memory workflow has an estimation stage and an update stage.
- During estimation, the GMM parameters use both historical embeddings read from memory and the current batch input.
- During updating, the oldest data batch is removed to make room for new data.
- The memory is implemented as a FIFO queue for GMM and entropy estimation.
A.2 Proof of eq. (7)
The proof derives lower and upper bounds for the entropy of a Gaussian mixture model. Maximizing the lower-bound KL term therefore provides a route to maximizing H(X).
- For a GMM, the marginal density of x is expressed using a class indicator z and mixture components Ci.
- The proof uses convexity of g(x) to derive a lower bound for H(X).
- The derivation assumes equal prior probabilities for the two classes.
- An upper bound for H(X) is included following Huber et al. (2008).
- Maximizing the lower-bound KL(X) term can maximize H(X).