Source-linked AI summary
Hybrid Contrastive Learning of Tri-Modal Representation for Multimodal Sentiment Analysis
Sijie Mai, Ying Zeng, Shuangjia Zheng, Haifeng Hu
TL;DR
Multimodal sentiment analysis still faces modality gaps, insufficient intra-/inter-modal learning, neglected inter-class relationships, and limited datasets. HyCon addresses these issues with hybrid contrastive learning and reports superior performance to existing methods on public datasets.
Problem
MSA lacks sufficiently aligned cross-modal representations and inter-class relationship modeling, while limited datasets constrain generalization.
Method
HyCon combines supervised intra-/inter-modal contrastive learning with semi-contrastive learning, refinement, modality-margin, and generated training pairs.
Results
HyCon outperforms existing works and achieves state-of-the-art performance on public datasets, including with simple fusion methods.
Takeaways & Limitations
The framework learns cross-modal dynamics and inter-class relationships while reducing modality-gap and limited-data concerns without adding loss-function parameters.
Takeaways & Limitations
COBRA's use of one same-modality positive per anchor leaves cross-modal dynamics between different samples of the same class underused.
Abstract
from arXiv · showhide
The wide application of smart devices enables the availability of multimodal data, which can be utilized in many tasks. In the field of multimodal sentiment analysis (MSA), most previous works focus on exploring intra- and inter-modal interactions. However, training a network with cross-modal information (language, visual, audio) is still challenging due to the modality gap, and existing methods still cannot ensure to sufficiently learn intra-/inter-modal dynamics. Besides, while learning dynamics within each sample draws great attention, the learning of inter-class relationships is neglected. Moreover, the size of datasets limits the generalization ability of existing methods. To address the afore-mentioned issues, we propose a novel framework HyCon for hybrid contrastive learning of tri-modal representation. Specifically, we simultaneously perform intra-/inter-modal contrastive learning and semi-contrastive learning (that is why we call it hybrid contrastive learning), with which the model can fully explore cross-modal interactions, preserve inter-class relationships and reduce the modality gap. Besides, a refinement term is devised to prevent the model falling into a sub-optimal solution. Moreover, HyCon can naturally generate a large amount of training pairs for better generalization and reduce the negative effect of limited datasets. Extensive experiments on public datasets demonstrate that our proposed method outperforms existing works.
1 INTRODUCTION
HyCon addresses modality gaps, neglected inter-class relationships, and limited-data overfitting in multimodal sentiment analysis through hybrid contrastive learning of tri-modal representations. It combines three contrastive objectives, refinement and modality-margin terms, and generated training pairs, achieving state-of-the-art performance on public datasets.
- MSA predicts sentiment scores from audio, visual, and language features, but learning meaningful multimodal representations remains challenging because of the modality gap.
- Existing methods mainly learn intra-sample dynamics while neglecting inter-sample and inter-class relationships, limiting discriminative joint embeddings.
- HyCon combines intra-modal, inter-modal, and semi-contrastive learning to model cross-modal dynamics, preserve inter-class relationships, and reduce modality gaps.IAMCL and IEMCL learn intra-/inter-modal and intra-/inter-class relationships, while SCL aggregates modalities within each sample.
- The refinement term and modality margin address shortcomings of contrastive learning and support more suitable representation learning for positive pairs.
- HyCon generates many positive and negative training pairs, reducing the negative impact of limited datasets and the possibility of overfitting.
- HyCon achieves state-of-the-art performance on public datasets with simple fusion methods and learns a more discriminative sentiment embedding space.
2 RELATED WORK
Prior multimodal sentiment methods use increasingly sophisticated fusion strategies but remain vulnerable to computational cost, overfitting, modality gaps, and neglected inter-class relationships. HyCon instead combines contrastive-learning approaches with direct fusion and reports new state-of-the-art results.
- Early fusion concatenates unimodal features, whereas late fusion averages unimodal decisions; both outperform unimodal methods but incompletely model modal dynamics.
- Sophisticated fusion strategies increase parameters and computational costs, worsening overfitting when training data are limited.
- Prior multimodal methods focus on intra-/inter-modal dynamics but neglect inter-class relationships that can support more discriminatory decision boundaries.
- HyCon combines multiple contrastive-learning methods with direct fusion, preserves inter-class relationships, and reports new state-of-the-art results despite simple fusion.
- Contrastive Learning: Traditional self-supervised contrastive learning uses one positive and many negative samples but cannot exploit available labels.
- Contrastive Learning: COBRA learns joint cross-modal embeddings with supervised contrastive learning but uses few positive samples and omits cross-modal positives across samples of the same class.
3 ALGORITHM
HyCon trains tri-modal sentiment representations with unimodal encoders, generated cross-modal pairs, and hybrid contrastive losses. Its training strategy targets modality alignment, intra-/inter-modal dynamics, inter-class relationships, and dataset-limited generalization while using simple fusion at inference.
- HyCon Motivation: The hybrid strategy addresses prior limitations involving modality alignment, neglected inter-sample and inter-class relationships, and overfitting on small datasets.These limitations motivate contrastive pair learning and the use of simple fusion rather than relying solely on elaborate fusion networks.
- Fusion and Prediction: During inference, the contrastive learning used in training is discarded and the learned unimodal embeddings are combined with a simple fusion method for prediction.The default fusion method is elementwise addition, while other fusion strategies are also evaluated.
- Unimodal Learning Network: HyCon extracts unimodal representations from audio, visual, and language inputs before generating contrastive pairs during training.Transformers process audio and visual sequences, while BERT processes language; the final timestep embedding is used for fusion and learning.
- Pair Generation: Pair generation randomly samples positive and negative pairs from K-sample mini-batches, producing several times more pairs than samples.The authors state that this increases dataset utilization and supports better generalization.
- Hybrid Contrastive Learning: SCL pulls representations from different modalities within the same utterance closer while retaining a modality margin α for modality-specific information.SCL uses only positive pairs and is designed to reduce the modality gap without fully eliminating modality discrepancies.
4.1 Datasets
The experiments use CMU-MOSI and CMU-MOSEI, two public multimodal sentiment-analysis datasets with sentiment-annotated video utterances.
- CMU-MOSI contains 2,199 opinion video utterances annotated with sentiment intensity from -3 to +3.The split includes 1,284 training, 229 validation, and 686 testing utterances.
- CMU-MOSEI contains over 20,000 video utterances from more than 1,000 YouTube speakers across 250 topics.
4.2 Evaluation Metrics
CMU-MOSEI and CMU-MOSI are evaluated with classification, error, and correlation metrics covering sentiment prediction quality.
- Evaluation uses Acc7, Acc2, F1, MAE, and Corr across both datasets.Acc7 measures seven-way accuracy, Acc2 binary accuracy, F1 classification performance, MAE prediction error, and Corr agreement with human predictions.
4.3 Baselines
The baseline set includes fusion, recurrent, attention, tensor, and transformer-based models for multimodal sentiment analysis.
- EF-LSTM concatenates modality features at the word level before LSTM-based prediction, whereas LF-LSTM combines modality-specific decisions by voting.
- RAVEN shifts language representations using audio and visual features to model multimodal interactions.
- MFN uses delta-attention and multi-view gated memory, while MULT translates source modalities into target modalities with a cross-modal Transformer.
- IMR adds multimodal routing to improve MULT’s interpretability, and TFN uses outer products to learn unimodal, bimodal, and trimodal interactions.
4.4 Experimental Details
Experiments tune models through repeated random grid search and repeated training, using specified datasets, features, software, and optimization settings.
- Each algorithm undergoes 50 random hyperparameter searches, followed by five training runs using the best setting; reported results are five-run means.
- The experiments compare HyCon with baselines on CMU-MOSI and CMU-MOSEI using Tables 1 and 2.Table 2 notes that IMR lacks MAE and Corr results because it cannot perform regression.
- Language, audio, and visual inputs have dimensions 768, 74, and 35 on CMU-MOSEI, versus 768, 74, and 47 on CMU-MOSI.
- HyCon uses PyTorch with Adam at learning rate 1e-5, modality margin α=0.8, λ1=λ2=λ3=1, and representation dimension d=50.Positive and negative pair generation uses positive versus negative sentiment classes rather than seven fine-grained classes.
4.5 Experimental Results
HyCon consistently improves multimodal sentiment performance, while ablations show that its contrastive losses, refinement term, modality margin, and simple fusion choices contribute to representation quality.
- 3.7% on Acc7, 1.7% on Acc2, and 1.6% on F1 score are HyCon’s improvements over MAG-BERT on CMU-MOSI.On CMU-MOSEI, HyCon improves Acc7 by 0.9%, Acc2 by 0.4%, and F1 score by 0.6%.
- Ablation Study: Removing all contrastive losses significantly degrades Acc7, Acc2, and F1 score on CMU-MOSI.This ablation supports the effectiveness of the proposed contrastive learning method.
- Ablation Study: The refinement term causes a slight performance drop when removed, while omitting modality margin α also degrades results.The refinement term is especially useful when negative pairs are rare, and α preserves modality-specific information while reducing modality gap.
- Ablation Study: Removing any one of the three contrastive losses lowers performance, with the largest drops occurring when IAMCL or IEMCL is removed.The authors identify inter-class relationship learning through IAMCL and IEMCL as fundamental to HyCon’s performance.
- Fusion Strategies: HyCon achieves state-of-the-art performance with simple concatenation or elementwise-addition fusion, and direct addition performs best among the tested strategies.Cross-modal contrastive learning is argued to reduce modality gap sufficiently for direct addition to exploit complementary information.
- Modality Margin: α = 0.8 gives the best performance; both lower and higher values degrade results by respectively retaining too much modality gap or losing modality-specific information.The experiments evaluate α values including 0.5, 0.7, 0.8, and 0.9 on CMU-MOSI.
- Embedding Visualization: Without contrastive learning, t-SNE embeddings are scattered and sentiment classes do not form distinguishable clusters.The figure compares HyCon’s embedding space with the space learned without contrastive learning; red x marks negative sentiment and blue dots mark positive sentiment.
- Comparison of Loss Functions: HyCon outperforms hard-triplet mining because it uses many positive and negative pairs and incorporates a modality margin for cross-modal mapping.The modality margin permits modality-specific information to remain available for multimodal fusion.
5 CONCLUSION
The conclusion presents HyCon as a parameter-free hybrid contrastive framework for multimodal sentiment analysis that learns cross-modal and inter-class structure. Experiments report state-of-the-art performance, alongside reduced overfitting risk and improved generalization ability.
- HyCon combines intra-modal, inter-modal, and semi-contrastive learning to model cross-modal dynamics and inter-class relationships.Its refinement term supports latent embedding learning for sentiment prediction.
- The designed loss function introduces no additional parameters, reducing the possibility of overfitting and improving generalization ability.
- Experiments demonstrate that HyCon outperforms state-of-the-art methods.