Source-linked AI summary
Select-Additive Learning: Improving Generalization in Multimodal Sentiment Analysis
Haohan Wang, Aaksha Meghawat, Louis-Philippe Morency, Eric P. Xing
TL;DR
Multimodal sentiment models face limited annotated datasets and can learn speaker-specific confounds that hinder generalization. The paper proposes Select-Additive Learning, which identifies identity-related representation dimensions and suppresses them with Gaussian noise. SAL improves sentiment prediction across verbal, acoustic, visual, and fused modalities and generalizes across datasets.
Problem
Limited annotated datasets can cause multimodal sentiment models to learn speaker-specific characteristics as confounding factors, restricting generalization.
Method
SAL identifies identity-related dimensions in neural representations and adds Gaussian noise to reduce their influence on sentiment classification.
Results
SAL increased sentiment-category clustering ratios by 44%, 15%, and 72% for verbal, acoustic, and visual modalities, respectively.
Takeaways & Limitations
SAL improved prediction accuracy significantly across verbal, acoustic, visual, and fused modalities and achieved good accuracy when tested across datasets.
Abstract
from arXiv · showhide
Multimodal sentiment analysis is drawing an increasing amount of attention these days. It enables mining of opinions in video reviews which are now available aplenty on online platforms. However, multimodal sentiment analysis has only a few high-quality data sets annotated for training machine learning algorithms. These limited resources restrict the generalizability of models, where, for example, the unique characteristics of a few speakers (e.g., wearing glasses) may become a confounding factor for the sentiment classification task. In this paper, we propose a Select-Additive Learning (SAL) procedure that improves the generalizability of trained neural networks for multimodal sentiment analysis. In our experiments, we show that our SAL approach improves prediction accuracy significantly in all three modalities (verbal, acoustic, visual), as well as in their fusion. Our results show that SAL, even when trained on one dataset, achieves good generalization across two new test datasets.
1. INTRODUCTION
Multimodal sentiment analysis uses verbal, acoustic, and visual cues from online videos, but limited annotated data can make models learn speaker-specific confounds instead of sentiment. SAL addresses this problem through a two-phase procedure designed to improve generalization across people and datasets.
- Multimodal sentiment analysis combines spoken words, speaking style, and accompanying gestures or facial expressions to analyze sentiment in online videos.
- Only a few high-quality datasets exist, with annotations still on the order of thousands of samples, making cross-dataset generalization challenging.
- Limited data can make a model associate a speaker characteristic such as wearing glasses with negative sentiment when that association is specific to the training dataset.
- A robust classifier should exclude confounding features so sentiment predictions do not depend on a speaker’s unique identity-related characteristics.
- In MOSI, a chi-square test produced p-value 1.202 × 10−19, strongly suggesting dependence between individual identities and expressed sentiment.
- SAL identifies confounding factors in latent representations, then adds Gaussian noise to those elements so the original model gives them less importance.
2. RELATED WORK
Prior work spans multimodal behavior analysis and sentiment modeling across textual, acoustic, and visual signals. Against this background, the paper introduces SAL as an extension intended to improve neural-network generalizability across modalities and their fusion.
- Multimodal data has been used to analyze behaviors including person detection, action recognition, face recognition, and sentiment.
- Sentiment analysis has expanded from textual units to audio, emotional state estimation, facial expressions, and visual affective regions.
- Fusion of textual, acoustic, and visual modalities has attracted increasing attention, with convolutional neural networks achieving state-of-the-art performance.
- SAL is presented as a procedure that improves neural-network generalizability and prediction accuracy across verbal, acoustic, visual, and fused inputs.
3. SELECT-ADDITIVE LEARNING
SAL enhances a pretrained discriminative neural network by identifying identity-related dimensions in its learned representation and reducing their influence with Gaussian noise. Its selection and addition phases target confounding features while preserving dimensions useful for sentiment prediction.
- Architecture: SAL aims to emphasize sentiment-associated features over identity-related features such as wearing glasses.
- Problem formulation: The input matrix X stores p features for n utterances, the sentiment vector y stores their labels, and Z encodes speaker identities in an n × m one-hot matrix.
- Architecture: SAL extends a pretrained discriminative neural network composed of a representation learner g(·; θ) and classification component f(·; φ).
- Selection Phase: A separate network h(·; δ) predicts identity-related dimensions from speaker identities Z by matching h(Z; δ) to the learned representation g(X; θ).
- Addition Phase: During the addition phase, Gaussian noise is applied to selected confounding dimensions while the classifier is trained to minimize prediction error.
- Selection Phase: The selection phase tunes δ, using L1 regularization to identify a sparse set of identity-related confounding dimensions while reducing overfitting.
- Illustration: Figure 3 presents SAL’s network structure and training objective, with circles denoting neurons and squares denoting representation dimensions.
- Addition Phase: Noise makes identity-related dimensions less informative, allowing the classifier to ignore them and focus its weights on the remaining dimensions.
4. EXPERIMENTS
The experiments evaluate whether SAL improves generalizability using three datasets, across-dataset testing, and person-independent splits. Comparisons include a state-of-the-art CNN and its SAL-enhanced version.
- Evaluation setup: Experiments use three datasets and reserve two datasets exclusively for testing in across-dataset evaluations.
- Evaluation setup: All experiments use a person-independent methodology in which test subjects do not appear in the training data.
- Compared models: The comparison includes a state-of-the-art seven-layer CNN previously used for multimodal sentiment analysis.
- Compared models: SAL-CNN applies SAL after the CNN is fully trained to increase its generalizability and predict sentiment.
4.2. Datasets
The experiments use three multimodal sentiment datasets—MOSI, YouTube, and MOUD—with differences in size, language, recording quality, and curation supporting across-dataset evaluation.
- Dataset composition: The study evaluates three datasets: MOSI, YouTube, and MOUD.The datasets contain 93 videos and 2199 utterances, 47 videos and 280 utterances, and 498 Spanish utterances from 55 individuals, respectively.
- MOSI: MOSI contains 2199 manually segmented and sentiment-annotated utterances from 93 YouTube videos, each featuring one individual.
- YouTube: YouTube contains 280 manually sentiment-annotated utterances from 47 opinion videos, each featuring one individual.
- MOUD: MOUD contains 498 Spanish opinion utterances from 55 unique individuals.
- Cross-dataset evaluation: The datasets differ in recording quality and post-curation processing, and MOUD requires Spanish-to-English transcript translation.These differences make the three datasets suitable for evaluating across-dataset generalization.
4.3. Feature Extraction
The study represents utterances using padded word embeddings for text and low-level descriptors for audio, with transcripts obtained or translated as needed.
- Text features: Each word is mapped to a word2vec embedding pretrained on Google News, and utterance embeddings are concatenated and zero-padded to length 60.Words beyond the maximum length are discarded.
- Transcript processing: YouTube transcripts are extracted with IBM Bluemix speech-to-text, while MOUD Spanish transcripts are translated into English.
- Acoustic features: Low-level audio descriptors are extracted with openSMILE.
4.4. Experiment Setup
MOSI supplies training and validation data, while held-out MOSI individuals plus YouTube and MOUD provide three test sets after neutral utterances are removed.
- Preprocessing: Neutral utterances are removed before training, validation, and testing.
- Training and validation: The first 62 MOSI individuals provide approximately 1250 utterances, shuffled into 80% training and 20% validation splits.
- Testing: The test sets contain 546 MOSI utterances from 31 remaining individuals, 195 YouTube utterances from 47 individuals, and 450 MOUD utterances from 55 individuals.
- Dataset selection: MOSI is selected for training because it is the largest and most recent of the three datasets.
4.5. Experiment Results
Results compare CNN with SAL-CNN on within-dataset and across-dataset tests, showing improved generalizability and significant overall gains with two reported exceptions.
- Within-dataset results: SAL-CNN improves generalizability over CNN on the remaining 31 individuals’ MOSI data.
- Across-dataset results: Across-dataset tests include YouTube and MOUD, where CNN performance is sometimes worse than mere chance.This inferior performance is reported as evidence of the targeted non-generalization problem.
- Overall results: Select-Additive Learning consistently increases robustness and performance except for video on MOUD and acoustic-visual fusion on MOSI.
- Statistical significance: Permutation tests reject the no-improvement null hypothesis with p-values 0.037, 0.0003, and 0.0023 for MOSI, YouTube, and MOUD, respectively.
4.6. Discussion
SAL changes learned representations so sentiment structure improves substantially while identity clustering is largely preserved. The analysis also shows that utterance representations cluster by person, with some dimensions generalizing across individuals.
- Selection-phase visualization: Figure 4 plots h(Z, δ) during the Selection phase across the first 50 utterances and 100 representation values.Blue denotes the lowest values, red the highest, and intermediate colors are linearly interpolated.
- Reproducibility: The implementation of Select-additive Learning is publicly available.The paper provides a GitHub repository for the implementation.
- Representation analysis: Utterance representations formed clear clusters associated with individual speakers, while some dimensions generalized across individuals.After noise was introduced, the model assigned greater weight to dimensions that generalized across individuals.
- Representation analysis: SAL increased sentiment-cluster separation by 44%, 15%, and 72% for verbal, acoustic, and visual modalities.For identity-clustered representations, the corresponding increases were 9%, 3%, and 13%.
- Representation analysis: SAL nearly maintained identity clustering while greatly improving clustering by sentiment category.The higher inter-cluster-distance-to-intra-cluster-distance ratio indicates clearer clustering.
5. CONCLUSION
The paper addresses limited generalizability in multimodal sentiment analysis by proposing SAL. Experiments report improved accuracy across individual modalities and their fusion, including good cross-dataset accuracy.
- Motivation: High-quality multimodal sentiment datasets contain only a few thousand samples, restricting model generalizability and contributing to confounding-factor problems.The paper identifies limited resources as the motivation for SAL.
- Contribution: SAL is proposed to mitigate confounding factors and improve the generalizability of state-of-the-art multimodal sentiment models.The conclusion summarizes the procedure as addressing the generalization problem caused by limited resources.
- Results: SAL significantly improved prediction accuracy in verbal, acoustic, and visual modalities, as well as their fusion.The reported gains cover each individual modality and the combined multimodal setting.
- Results: SAL achieved good prediction accuracy when tested across datasets.The conclusion reports cross-dataset performance in addition to within-dataset modality and fusion results.