Source-linked AI summary
Multimodal Sentiment Analysis: Addressing Key Issues and Setting up the Baselines
Soujanya Poria, Navonil Majumder, Devamanyu Hazarika, Erik Cambria, Alexander Gelbukh, Amir Hussain
TL;DR
Multimodal sentiment analysis lacks consistent evaluation of context, speaker independence, modality contributions, and generalization. The paper establishes deep-learning baselines, fixed dataset splits, and analyses across modalities and speaker settings. It concludes that these evaluations provide useful baselines while identifying contextual dependency learning as a limitation to address.
Problem
Multimodal sentiment analysis has limited treatment of context, speaker-exclusive evaluation, modality impact, and model generalization, hindering effective comparison of methods.
Method
The paper builds deep-learning multimodal baselines, extracts textual features with CNNs, evaluates fixed train/test splits, and performs speaker-exclusive and modality analyses.
Results
Multimodal models generally outperform unimodal models, audio outperforms visual features, text is usually strongest unimodally, and context-aware bc-LSTM outperforms SVM.
Takeaways & Limitations
The study provides useful baselines and highlights unknown-speaker and cross-dataset performance as important dimensions of multimodal evaluation.
Takeaways & Limitations
Future work must add contextual dependency learning to overcome limitations identified in the paper.
Abstract
from arXiv · showhide
We compile baselines, along with dataset split, for multimodal sentiment analysis. In this paper, we explore three different deep-learning based architectures for multimodal sentiment classification, each improving upon the previous. Further, we evaluate these architectures with multiple datasets with fixed train/test partition. We also discuss some major issues, frequently ignored in multimodal sentiment analysis research, e.g., role of speaker-exclusive models, importance of different modalities, and generalizability. This framework illustrates the different facets of analysis to be considered while performing multimodal sentiment analysis and, hence, serves as a new benchmark for future research in this emerging field.
I. INTRODUCTION
Video-based sentiment analysis offers vocal, visual, and textual cues, while multimodal research still lacks consistent treatment of context, speaker independence, modality contributions, and generalization. The paper addresses these issues through baseline methods and fixed dataset evaluation.
- Motivation: Videos provide vocal modulations, facial expressions, and text that offer more behavioral cues than text-only analysis.These complementary signals can help identify the opinion holder’s affective state.
- Research gap: Existing multimodal sentiment research often leaves context, speaker splitting, modality impact, and generalization insufficiently addressed.These omissions make comparisons among methods difficult.
- Approach: The paper outlines methods addressing these issues and establishes a baseline using state-of-the-art approaches.It uses a deep CNN to extract visual and textual features.
- Paper scope: The paper evaluates multimodal sentiment analysis through organized baseline discussion, experiments, and concluding analysis.Its structure covers related work, baseline methods, experimental results, and conclusions.
II. RELATED WORK
Prior work established benefits from audio-visual fusion, but comparatively few studies combined text with audio and visual modalities. This paper examines multimodal methods under speaker independence, generalizability, and individual-modality performance.
- Audio-visual fusion: Early audio-visual emotion-recognition studies reported higher accuracy for bimodal systems than for unimodal systems.These studies fused audio and visual information for emotion recognition.
- Multimodal sentiment analysis: Only a few studies addressed multimodal emotion or sentiment analysis using text together with visual and audio modalities.Prior examples include systems combining all three modalities or pairing audio with text.
- Paper focus: The paper studies speaker independence, model generalizability, and individual-modality performance in the method proposed in.These aspects were described as rarely addressed by other authors.
III. UNIMODAL FEATURE EXTRACTION
The paper extracts modality-specific representations using CNN-based text processing, 3D-CNN visual features, and openSMILE acoustic features.
- Unimodal Feature Extraction: The unimodal feature-extraction procedures follow bc-LSTM.This establishes the feature-processing basis for the subsequent multimodal models.
- Textual Feature Extraction: Textual utterances are represented with CNN ngram features from filters of sizes 3, 4, and 5, followed by pooling, ReLU, concatenation, and a 100-dimensional dense layer.The network is trained at utterance level using emotion labels.
- Visual and Acoustic Feature Extraction: Visual and acoustic features are extracted with 3D-CNN and openSMILE, respectively.The procedures follow the method used in.
C. Fusion
The fusion scheme concatenates modality-specific feature vectors before classification, while a bidirectional LSTM incorporates context from surrounding utterances into context-aware representations.
- C. Fusion: Feature-level fusion concatenates the feature vectors of the selected modalities and sends the combined vector to a classifier.Because the scheme uses concatenation without overlapping or merging features, scaling and normalization are avoided.
- C. Fusion: A bidirectional LSTM captures context from surrounding utterances to generate context-aware utterance representations.This contextual representation is part of the bc-LSTM method.
2) SVM:
The experiments use extracted features merged and classified with an RBF-kernel SVM, evaluated across multimodal sentiment datasets and settings.
- Extracted features are merged and sent to an SVM with an RBF kernel for final classification.
- The experiments analyze datasets and experimental settings while evaluating the methods' results.
2) Multimodal Emotion Recognition Dataset:
The study evaluates person-independent multimodal emotion recognition using speaker-disjoint splits across IEMOCAP, MOUD, and MOSI, with dataset-specific fold designs.
- Speaker-Exclusive Experiment: Speaker-exclusive evaluation uses completely speaker-disjoint train and test splits, requiring classification of utterances from unseen speakers.
- Dataset Splits: Table I specifies approximately 70/30% person-independent train/test splits, with validation extracted from shuffled training data at an 80/20% ratio.
- IEMOCAP: IEMOCAP uses a 10-fold speaker-exclusive test with one of its 10 speakers held out in each round.
- MOUD: MOUD uses five speaker groups and averages accuracy across a 5-fold person-exclusive experiment.
- MOSI: MOSI uses five speaker groups for 5-fold person-exclusive testing, with approximately 75 people in each training set on average.
1) Speaker-Inclusive vs. Speaker-Exclusive:
The paper contrasts speaker-inclusive and speaker-exclusive evaluation, reporting lower performance when test utterances come from speakers absent during training.
- Speaker-exclusive evaluation yields inferior results compared with speaker-inclusive evaluation.The authors attribute this to absent speaker knowledge during testing.
- In speaker-inclusive experiments, audio consistently outperforms visual modality on MOSI and IEMOCAP.
- Text is the most important modality for both emotion recognition and sentiment analysis in the reported speaker-inclusive results.
- Figure 1 reports RMSE and TP-rate for models using different modalities on IEMOCAP and MOSI.
C. Contributions of the Modalities
The modality analysis finds consistent gains from multimodal fusion, modality-specific performance differences, and poor cross-dataset transfer from MOSI to MOUD.
- Modality Fusion: Bimodal and trimodal models outperform unimodal models in all experiments.
- Individual Modalities: Audio performs better than visual across all datasets, while text is substantially stronger than the other unimodal modalities except on MOUD.
- Cross-Dataset Generalization: A model trained speaker-exclusively on MOSI performs poorly when tested on MOUD.
- Cross-Dataset Generalization: The MOSI-to-MOUD transfer result is mainly attributed to MOUD reviews being in Spanish and MOSI reviews being in English, causing audio and text recognition failures.
- Scope Boundary: The study could not test same-language cross-dataset generalization because benchmark datasets were lacking, and cross-dataset emotion detection lacked multiple datasets.
E. Comparison among the Baseline Methods
The comparison evaluates SVM and bc-LSTM fusion across MOSI, MOUD, and IEMOCAP, finding that bc-LSTM consistently outperforms SVM. The results indicate that incorporating context substantially improves classification performance.
- bc-LSTM performs better than SVM across all evaluated experiments.The comparison covers MOSI, MOUD, and IEMOCAP datasets.
- Context-aware classification substantially boosts performance relative to the compared baseline.The paper attributes the improvement to considering context in the classification process.
- Multimodal representations reduce overlap more effectively than unimodal representations in MOSI and IEMOCAP visualizations.MOSI shows more structured declustering in video and all-modalities settings, while IEMOCAP's multimodal distribution has the least overlap.
V. CONCLUSION
The paper presents baselines for multimodal sentiment analysis and emotion recognition while examining speaker-exclusive and cross-dataset performance. It identifies limitations involving visual semantics, cross-modal relatedness and fusion, and contextual dependency, motivating future work.
- The study presents baselines for multimodal sentiment analysis and multimodal emotion recognition, including evaluations in unknown-speaker and cross-dataset settings.
- Future work targets extracting semantics from visual features, modeling cross-modal relatedness and fusion, and learning contextual dependencies.These directions are presented as ways to address limitations discussed earlier in the paper.
- Speaker-exclusive accuracy is reported using dataset-specific cross-validation averages for IEMOCAP, MOUD and MOSI.IEMOCAP uses 10-fold averaging, while MOUD and MOSI use 5-fold averaging; the modality legend covers audio, video and text.
- T-SNE visualizations compare MOSI and IEMOCAP representations formed from unimodal versus multimodal features.