Source-linked AI summary
Bi-Bimodal Modality Fusion for Correlation-Controlled Multimodal Sentiment Analysis
Wei Han, Hui Chen, Alexander Gelbukh, Amir Zadeh, Louis-philippe Morency, Soujanya Poria
TL;DR
Multimodal sentiment analysis lacks fusion schemes that jointly model relevance and independence across modalities. BBFN performs pairwise bimodal fusion with feature separation and gated Transformer control, and it outperforms state-of-the-art models across three datasets and almost all reported metrics.
Problem
Existing multimodal fusion methods do not properly model the competition between modality relevance and independence, risking feature-space collapse or extra noise.
Method
BBFN uses two Transformer-based bimodal modules for progressive pairwise fusion, a modality-specific feature separator, and gated control.
Results
BBFN outperforms state-of-the-art models on almost all metrics across CMU-MOSI, CMU-MOSEI, and UR-FUNNY.
Takeaways & Limitations
The experiments and analyses support BBFN as an effective fusion architecture for multimodal sentiment analysis.
Abstract
from arXiv · showhide
Multimodal sentiment analysis aims to extract and integrate semantic information collected from multiple modalities to recognize the expressed emotions and sentiment in multimodal data. This research area's major concern lies in developing an extraordinary fusion scheme that can extract and integrate key information from various modalities. However, one issue that may restrict previous work to achieve a higher level is the lack of proper modeling for the dynamics of the competition between the independence and relevance among modalities, which could deteriorate fusion outcomes by causing the collapse of modality-specific feature space or introducing extra noise. To mitigate this, we propose the Bi-Bimodal Fusion Network (BBFN), a novel end-to-end network that performs fusion (relevance increment) and separation (difference increment) on pairwise modality representations. The two parts are trained simultaneously such that the combat between them is simulated. The model takes two bimodal pairs as input due to the known information imbalance among modalities. In addition, we leverage a gated control mechanism in the Transformer architecture to further improve the final output. Experimental results on three datasets (CMU-MOSI, CMU-MOSEI, and UR-FUNNY) verifies that our model significantly outperforms the SOTA. The implementation of this work is available at https://github.com/declare-lab/multimodal-deep-learning.
1 INTRODUCTION
Multimodal sentiment analysis must combine complementary emotional cues across modalities while preserving their independence. BBFN addresses this tension through pairwise fusion, feature separation, and gated Transformer control, achieving strong results across three datasets.
- A good fusion scheme integrates meaningful cross-modal information while preserving mutual independence between modalities.
- Multimodal sentiment analysis combines visual, acoustic, and linguistic data to predict sentiment and emotion labels.Emotional cues may be distributed across modalities, making multimodal fusion important.
- BBFN uses two Transformer-based bimodal learning modules with progressive fusion over modality sequence pairs.The model uses text-visual and text-acoustic pairs because text is empirically the most significant modality.
- Feature space separation regularizes modality representations so heterogeneous features remain distinct and modality independence is maintained.
- BBFN outperforms state-of-the-art models on almost all metrics across CMU-MOSI, CMU-MOSEI, and UR-FUNNY.The evaluation covers sentiment intensity prediction and humor recognition.
- BBFN adds gated control to enhance its Transformer-based fusion process.
2 RELATED WORK
Prior multimodal sentiment work uses fusion, attention, alignment, correlation, and factorization to model multimodal representations. The paper distinguishes its approach by progressively fusing selected modality pairs while regularizing feature spaces and controlling interactions.
- Multimodal sentiment analysis integrates acoustic, visual, and textual information to comprehend varied human emotions.
- Previous methods commonly use early or late fusion, attention, tensor interactions, hierarchical attention, or multi-task learning.
- Several approaches model interactions across every modality pair, whereas this paper uses two pairs involving a common central modality.
- Correlation-based Approach: Correlation-based methods learn or select correlated multimodal representations but overlook modality independence and competition between these objectives.
- Alignment-based Approach: Attention-based soft alignment maps signals across modalities, and Transformer architectures support multimodal representation learning and feature fusion.
- Alignment-based Approach: The paper’s Transformer fusion architecture differs by combining progressive fusion, feature-space regularization, and fine-grained gate control.
3 METHODOLOGY
BBFN performs iterative bimodal fusion while preserving modality-specific independence through feature separation and gated information flow. It uses two text-related modality pairs and concatenates their head representations for prediction.
- Overall Architecture: BBFN uses two modality complementation modules for iterative bimodal fusion across text-visual and text-acoustic pairs.Each module processes a modality sequence pair through stacked complementation layers.
- Modality Sequence Encoder: The model encodes modality sequences before fusion, using BERT for text and BiGRU-based sequence encoders for temporal dependencies and common hidden dimensions.The resulting sequences serve as initial inputs to the modality complementation modules.
- Modality-Specific Feature Separator: The feature space separator uses discriminator loss to preserve modality-specific representations and maintain mutual independence during deep fusion.Representations are grouped and averaged before classification to reduce noise, with grouping variance decreasing as group size increases.
- Output: Four head representations from the two text-related modality pairs are concatenated to generate the final prediction.The two fusion pipelines therefore contribute jointly to the output.
- Gated Complementation Transformer: The Gated Complementation Transformer controls cross-modal information flow with retain and compound gates that regulate forwarding and injection of modality components.The gated mechanism is integrated into the Transformer fusion and separator pipelines.
- Output Layer and Training: The total training objective combines the task loss with layer-wise separator losses.The task loss is MSE for sentiment intensity prediction, while separator loss is accumulated across layers.
4 EXPERIMENTS
The experiments evaluate BBFN on sentiment intensity prediction and humor detection across three multimodal datasets, using processed visual, acoustic, and text inputs and comparing against advanced fusion frameworks.
- BBFN is evaluated on sentiment intensity prediction and humor detection using CMU-MOSI, CMU-MOSEI, and UR-FUNNY.
- CMU-MOSI and CMU-MOSEI contain sentiment-labeled video segments, while UR-FUNNY contains multimodal punchlines labeled for humor detection.
- Inputs are processed into machine-understandable tensors from text, visual, and acoustic modalities for fair comparison with baselines.
- Visual features come from Facet or Openface, acoustic features from COVAREP, and visual and acoustic signals are word-aligned to text using P2FA.
- A complementation layer uses two identical pipelines to propagate a main modality and fuse it with a complementary modality using regularization and gated control.
- The comparison includes bimodal, tensor-based, attention-based, correlation-based, Transformer, and modality-separation fusion frameworks.
5 RESULTS AND ANALYSIS
BBFN outperforms competing models on nearly all reported metrics, while analyses attribute gains to its bi-bimodal architecture, feature separator, and gated control. Further studies examine error distributions, ablations, and gate behavior.
- 5.1 Summary of Results: BBFN outperforms other models on almost all metrics across CMU-MOSI, CMU-MOSEI, and UR-FUNNY, except CMU-MOSI correlation coefficient.
- 5.1 Summary of Results: Around 1% improvement is achieved in binary classification accuracy, with more than 2.5% improvement in 7-class accuracy.
- 5.1 Summary of Results: More than 4% is the best performance boost, reported for CMU-MOSEI MAE.
- 5.1 Summary of Results: For CMU-MOSEI test predictions with error< 0.25, BBFN’s absolute-error distribution has more peaks than MISA’s.
- 5.2 Ablation Study: Using all three modalities improves MAE and Acc-7 over using one modality pair, whereas adding a visual-acoustic module degrades performance.
- 5.2 Ablation Study: Removing the feature separator, gates, or both degrades all metrics except correlation, while the overall bi-bimodal architecture provides the largest baseline improvement.
- 5.3 Further Analysis: Gate weights indicate balanced visual and acoustic contributions in one case, while text in the TV module is complemented by visual information in another.
- 5.3 Further Analysis: BBFN outputs are closer to ground truth than MISA’s in the two case studies, attributed to fine-grained gate control and reduced distraction from insignificant modalities.
6 CONCLUSION
BBFN progressively performs bimodal fusion while controlling feature-space collapse and fusion-time coordination. Experiments and analyses report performance above current state-of-the-art approaches.
- BBFN processes pairwise fusion progressively through stacked complementation layers in each learning module.
- A modality-specific feature-space separator alleviates feature-space collapse during fusion.
- A gated control mechanism provides control during the fusion process.
- Comprehensive experiments and analyses show that BBFN outperforms current state-of-the-art approaches.