Source-linked AI summary
UniMSE: Towards Unified Multimodal Sentiment Analysis and Emotion Recognition
Guimin Hu, Ting-En Lin, Yi Zhao, Guangming Lu, Yuchuan Wu, Yongbin Li
TL;DR
Existing research generally treats multimodal sentiment analysis and conversational emotion recognition separately, despite their complementary relationship. UniMSE unifies the tasks through shared features, labels, and generative modeling, with multi-level modality fusion and contrastive learning. It reports consistent improvements over state-of-the-art methods across four benchmark datasets.
Problem
Most existing works treat MSA and ERC as separate tasks, overlooking similarities and complementarities between sentiments and emotions.
Method
UniMSE unifies MSA and ERC as a generative task through shared audio-video features, Universal Labels, multi-level textual fusion, and inter-modal contrastive learning.
Results
UniMSE achieves state-of-the-art performance across MOSI, MOSEI, MELD, and IEMOCAP, improving multiple reported metrics over previous state-of-the-art results.
Takeaways & Limitations
The results support jointly modeling sentiment and emotion and show relevance between them in a shared multimodal embedding space.
Takeaways & Limitations
Context is integrated only for MELD and IEMOCAP, while universal-label generation considers textual but not acoustic or visual modalities.
Abstract
from arXiv · showhide
Multimodal sentiment analysis (MSA) and emotion recognition in conversation (ERC) are key research topics for computers to understand human behaviors. From a psychological perspective, emotions are the expression of affect or feelings during a short period, while sentiments are formed and held for a longer period. However, most existing works study sentiment and emotion separately and do not fully exploit the complementary knowledge behind the two. In this paper, we propose a multimodal sentiment knowledge-sharing framework (UniMSE) that unifies MSA and ERC tasks from features, labels, and models. We perform modality fusion at the syntactic and semantic levels and introduce contrastive learning between modalities and samples to better capture the difference and consistency between sentiments and emotions. Experiments on four public benchmark datasets, MOSI, MOSEI, MELD, and IEMOCAP, demonstrate the effectiveness of the proposed method and achieve consistent improvements compared with state-of-the-art methods.
1 Introduction
MSA and ERC are related multimodal tasks, but existing work usually treats them separately despite complementary similarities and differences between sentiments and emotions. UniMSE unifies their features, labels, and task formulation while combining multi-level modality fusion with inter-modal contrastive learning.
- MSA predicts sentiment intensity or polarity, whereas ERC predicts predefined emotion categories.
- Most existing works study MSA and ERC separately, overlooking similarities and complementarities between sentiments and emotions.
- Emotions reflect perceived change over a short period, while sentiments form and persist over longer periods.The paper also reports that MSA videos are almost twice as long as ERC videos in its preliminary study.
- UniMSE reformulates MSA and ERC as a generative task, unifying their inputs, outputs, and task labels through Universal Labels.It extracts and unifies audio and video features across the two tasks.
- The framework fuses acoustic and visual signals with syntactic and semantic textual features and applies inter-modal contrastive learning.The contrastive objective minimizes intra-class variance and maximizes interclass variance across modalities.
- UniMSE reports state-of-the-art performance on MOSI, MOSEI, MELD, and IEMOCAP for both MSA and ERC.The paper identifies this as a contribution and states that it is the first to use unified audio and video features across both tasks.
2 Related Work
Prior work studies MSA and ERC through multimodal fusion, alignment, context modeling, external knowledge, and task-unification strategies. UniMSE is positioned within broader efforts to unify related tasks, including text-to-text and shared-label formulations.
- Multimodal Sentiment Analysis: MSA research includes multimodal fusion, alignment, context modeling, and external knowledge, with fusion methods ranging from geometric manipulation to information-based objectives.
- Emotion Recognition in Conversations: Multimodal ERC research commonly uses multimodal fusion, context-aware models, external knowledge, and graph-based modeling of utterance or speaker dependencies.
- Unified Framework: Unified frameworks cast related tasks into shared formats or label spaces to reduce different treatment across subtasks.Examples include T5's text-to-text formulation and a framework unifying entity detection with relation classification.
- Unified Framework: UniMSE is presented as a unified framework for MSA and ERC within this broader task-unification research direction.
3 Method
UniMSE unifies MSA and ERC through shared inputs, universal labels, and a generative architecture. It combines contextual multimodal encoding, multi-level fusion, and inter-modality contrastive learning to connect sentiment and emotion information.
- Task Formalization: UniMSE reformulates MSA and ERC as one generative task with unified input features, model architecture, and universal labels.Universal labels combine sentiment polarity, sentiment intensity, and emotion categories.
- Input Formalization: The model concatenates each utterance with two preceding and two following utterances to provide conversational context.Segment identifiers distinguish the current utterance from its textual context.
- Input Formalization: Audio is represented with Mel-spectrogram features and video with fixed-frame EfficientNet features, while separate LSTMs encode the non-text modalities.T5 encodes the textual modality and the unified feature extractors are used across datasets.
- Label Formalization: Universal-label completion aligns samples with the same sentiment polarity using textual semantic similarity, with SimCSE providing sentence embeddings.For an MSA sample with a sentiment score, the most semantically similar ERC sample supplies the missing emotion category, and vice versa.
- Label Formalization: The universal-label completion procedure achieved about 90% accuracy in a manual evaluation of 80 MOSI samples.The generated universal labels support decoding predictions for both MSA and ERC.
- Pre-trained Modality Fusion: Pre-trained modality fusion injects acoustic and visual representations into multiple T5 Transformer layers, combining shallow textual syntax with deeper textual semantics.The fusion layer is inserted as an adapter after selected Transformer layers, while the remaining layers receive non-verbal signals.
- Inter-modality Contrastive Learning: Inter-modality contrastive learning brings representations from the same sample closer and separates representations from different samples.The method applies this objective to acoustic, visual, and fused representations after temporal convolution.
4 Experiments
UniMSE is evaluated across four public MSA and ERC benchmarks using shared metrics and comparisons, with ablations and visualization examining modalities, fusion, contrastive learning, and cross-task representations.
- Baselines: The experiments compare UniMSE with early multimodal fusion methods, modality-interaction models, and conversational emotion-recognition baselines.The baselines include TFN, LMF, MFM, MulT, ICCN, MMGCN, MM-DFN, and COGMEN, among others.
- Evaluation setup: UniMSE is compared on MOSI, MOSEI, IEMOCAP, and MELD using task-appropriate sentiment and emotion metrics.MOSI and MOSEI use MAE, Corr, ACC-7, ACC-2, and F1; MELD and IEMOCAP use ACC and WF1.
- Results: UniMSE significantly outperforms previous state-of-the-art methods across all four datasets and improves several reported accuracy and F1 metrics.ACC-2 gains are 1.65% on MOSI and 1.16% on MOSEI; ACC gains are 2.6% on MELD and 2.35% on IEMOCAP.
- Ablation Study: Removing visual or acoustic inputs degrades performance, showing that non-verbal signals complement text and that acoustic information is more important than visual information for UniMSE.The ablation is conducted on MOSI, where removing either or both non-verbal modalities reduces performance.
- Ablation Study: Removing PMF or contrastive learning increases MAE and decreases Corr, supporting both modules’ effectiveness for multimodal representation learning.The ablation study evaluates the effects of pre-trained modality fusion and contrastive learning on MOSI.
- Visualization: Representations from positive sentiment overlap with joy-emotion samples, while original and universal-label-generated joy samples share a common embedding space.The visualizations use multimodal fusion representations from MOSI and MELD samples.
5 Conclusion
The paper concludes that jointly modeling sentiment and emotion is feasible through UniMSE, a unified multimodal knowledge-sharing framework. It reports state-of-the-art results across four benchmark datasets and positions the framework as a new perspective for MSA and ERC.
- UniMSE jointly models sentiment and emotion by aligning multimodal input features and output labels.The framework fuses acoustic and visual representations with multi-level textual features and applies inter-modality contrastive learning.
- Experiments on four benchmark datasets achieve state-of-the-art results in all metrics.
- The work presents a new experimental setting and perspective for the MSA and ERC research communities.
Limitations
The preliminary study has two stated limitations: context is integrated only for MELD and IEMOCAP, and universal-label generation uses textual modality alone. The authors identify both areas for future work.
- Context information is integrated only on MELD and IEMOCAP, while MOSI and MOSEI context remain future work.
- Universal-label generation considers textual modality but excludes acoustic and visual modalities.
- The authors plan to address missing context coverage and multimodal universal-label generation in future work.
Ethics Statement
The study uses open-source research data and discusses possible applications in emotional companion robots and intelligent customer service. However, the reported six-class MELD accuracy is 65%, described as far from usable.
- All data used in the study are open-source data intended for research purposes.
- The paper identifies emotional companion robots and intelligent customer service as possible application areas.
- 65% accuracy on six-class MELD emotion recognition is reported as far from usable.
A.1 Datasets
The dataset analysis compares video-segment duration across MSA and ERC and reports emotion-category distributions for MELD and IEMOCAP. Table 4 defines dataset- and task-level average video-length measures.
- The average video-segment duration for sentiment in MSA is longer than for emotion in ERC.The paper treats video-segment length as the duration of sentiment or emotion.
- The average video-segment length in MOSEI is 7.6 seconds.The paper notes that this may help explain why MOSEI is usually used to study sentiments rather than emotions.
- Emotion-category distributions for MELD and IEMOCAP are reported across training, validation, and test sets.
- Table 4 reports average video length using dataset-average and task-average measures, denoted D-A VL(s) and T-A VL(s).
A.2 Decoding Algorithm for MSA and ERC tasks
The decoding algorithm converts UniMSE’s predicted target sequence into a sentiment intensity for MSA or an emotion category for ERC.
- The algorithm decodes UniMSE’s predicted target sequence into task-specific outputs for MSA and ERC.It produces sentiment intensity for MSA and emotion category for ERC.
A.3 Experimental Environment
The experiments use NVIDIA A100 and V100 GPUs with a T5-base model, while the supplied fragments also show target-sequence decoding steps and IEMOCAP emotion-category distribution information.
- Experiments were conducted on NVIDIA RTX A100 and RTX V100 GPUs.
- T5-base has 220M parameters, 12 layers, 768 hidden dimensions, and 12 heads.
- Table 6 reports the distribution of emotion categories on the IEMOCAP dataset.
- The supplied implementation fragments include target-sequence elements and separate decoding branches for MSA and ERC.The fragments show task-specific conditional steps for MSA and ERC decoding.
- PML uses two projection layers, with a parameter count nearly a thousandth of the original parameter number.