Source-linked AI summary
MOSI: Multimodal Corpus of Sentiment Intensity and Subjectivity Analysis in Online Opinion Videos
Amir Zadeh, Rowan Zellers, Eli Pincus, Louis-Philippe Morency
TL;DR
MOSI addresses the lack of datasets and methods for studying sentiment and subjectivity in online opinion videos, including the challenge of segmenting rapidly changing opinions. It introduces an opinion-level multimodal corpus and a multimodal dictionary, whose joint modeling of words and gestures outperforms simple feature concatenation for sentiment prediction.
Problem
Online opinion videos remain understudied because suitable datasets, methodologies, baselines, and analyses of cross-modal information are limited.
Method
The paper constructs MOSI with opinion-level subjectivity segmentation, sentiment-intensity annotations, aligned language, visual, and audio features, and a multimodal dictionary jointly modeling words and gestures.
Results
The multimodal dictionary achieves the best performance among the sentiment-analysis baselines, outperforming simple concatenation of features.
Takeaways & Limitations
MOSI provides a foundation for multimodal sentiment and subjectivity studies, and the results show that information beyond text can improve predictions.
Abstract
from arXiv · showhide
People are sharing their opinions, stories and reviews through online video sharing websites every day. Studying sentiment and subjectivity in these opinion videos is experiencing a growing attention from academia and industry. While sentiment analysis has been successful for text, it is an understudied research question for videos and multimedia content. The biggest setbacks for studies in this direction are lack of a proper dataset, methodology, baselines and statistical analysis of how information from different modality sources relate to each other. This paper introduces to the scientific community the first opinion-level annotated corpus of sentiment and subjectivity analysis in online videos called Multimodal Opinion-level Sentiment Intensity dataset (MOSI). The dataset is rigorously annotated with labels for subjectivity, sentiment intensity, per-frame and per-opinion annotated visual features, and per-milliseconds annotated audio features. Furthermore, we present baselines for future studies in this direction as well as a new multimodal fusion approach that jointly models spoken words and visual gestures.
1 Introduction
Online opinion videos pose challenges for fine-grained sentiment and subjectivity analysis because speakers shift topics, sentiment varies in intensity, and visual cues can clarify spoken opinions. MOSI addresses these challenges with opinion-level annotations, synchronized multimodal features, and baselines including a word–gesture fusion approach.
- Challenges: Rapid topic and opinion shifts make it difficult to identify and segment the distinct opinions expressed in a video.A single spoken utterance may contain multiple opinions.
- Challenges: Sentiment analysis must estimate intensity, not merely classify a segment as positive or negative.The dataset therefore represents sentiment along a spectrum from highly positive to highly negative.
- Challenges: Visual information can resolve sentiment ambiguity in spoken words alone, such as interpreting “this movie was different” alongside a strong frown.MOSI studies language and gesture jointly as an enabling step toward multimodal sentiment analysis.
- MOSI contribution: MOSI identifies 3702 video segments, including 2199 opinion segments, using a fine-grained subjectivity annotation scheme.The corpus also provides sentiment-intensity labels and transcriptions synchronized with acoustic and visual features at word and phoneme levels.
- MOSI contribution: MOSI introduces baselines and a multimodal dictionary that jointly models spoken words and visual gestures.The paper presents this representation as an enabling approach for studying interactions between modalities.
2 Related Work
Prior multimodal sentiment datasets generally annotate polarity at video or utterance level and do not provide sentiment-intensity labels. Earlier approaches also commonly concatenated modalities without studying their relations and often used speaker-dependent experiments.
- Multimodal sentiment datasets: The YouTube Opinion Dataset contains 47 videos annotated for video-level sentiment polarity, with manually transcribed text and automatically extracted audio and visual features.The MMMO dataset extends this collection from 47 to 370 videos.
- Multimodal sentiment datasets: Existing datasets focus on polarity rather than sentiment intensity and commonly analyze videos or utterances instead of fine-grained opinion segments.The Spanish Multimodal Opinion Dataset contains 105 videos annotated for sentiment polarity at utterance level.
- Multimodal analysis: Earlier multimodal approaches directly concatenated modalities without studying relations between them and used speaker-dependent experiments.Related work also focused on sentiment polarity rather than intensity and on utterances rather than opinion segments.
- Audio-visual emotion recognition: Audio-visual emotion-recognition research provides related evidence from facial expressions, head gestures, and other visual cues.This work connects multimodal sentiment analysis with affective computing and computer vision.
3 MOSI: Multimodal Opinion-level Sentiment Intensity Corpus
MOSI combines multimodal observations with fine-grained opinion segmentation, sentiment-intensity annotation, and aligned speech, visual, and acoustic information. Its analyses examine how facial gestures interact with spoken words across neutral, emphasizer, and polarized-word patterns.
- Dataset overview: MOSI integrates transcribed speech, visual gestures, automatic audio and visual features, opinion-level subjectivity segmentation, sentiment intensity, and cross-modal alignment.The dataset reports high coder agreement for sentiment-intensity annotations and aligns words with visual and acoustic features.
- Opinion-level subjectivity segmentation: The annotation scheme segments subjective content by distinct private states and by modifications or corollaries that preserve the subject.This supports separating multiple opinions within one utterance while retaining related additions when they modify the same private state.
- Opinion-level subjectivity segmentation: The corpus contains 2199 subjective segments and 1503 objective segments, with sentiment annotations restricted to subjective segments.The segmentation scheme distinguishes private-state expressions from objective content and supports fine-grained sentiment analysis.
- Sentiment intensity annotation: Sentiment intensity is obtained by averaging ratings from five workers, with inter-annotator agreement of 0.77 Krippendorf’s Alpha.Workers rated the speaker’s sentiment state rather than whether they agreed with the speaker’s content.
- Multimodal analysis of words and visual gestures: MOSI identifies neutral, emphasizer, positive, and negative interaction patterns between spoken words and facial gestures.The analyses use sentiment-intensity distributions across opinion segments to characterize these patterns.
- Multimodal analysis of words and visual gestures: Neutral words such as “the” largely follow common gesture effects, whereas “really” shifts toward more extreme sentiment with smiles, nods, frowns, or head shakes.For polarized words such as “love” and “don’t,” gestures produce different, more limited distribution changes than for neutral or emphasizer words.
4 Sentiment and Subjectivity Analysis Baselines
The paper evaluates speaker-independent baselines for sentiment intensity and subjectivity on MOSI, comparing verbal, visual, concatenated, and jointly modeled multimodal representations.
- Sentiment Analysis Baselines: All prediction models use nu-SVR with 5-fold cross-validation, while hyperparameters are selected through 4-fold validation on training sets.Regressors are evaluated using mean absolute error and correlation.
- Sentiment Analysis Baselines: The sentiment experiments compare random, verbal, visual, verbal-plus-visual, and multimodal-dictionary models.The verbal model uses unigram and bigram bag-of-words features, while the visual model uses binary smile, frown, head-nod, and head-shake features.
- Sentiment Analysis Baselines: The multimodal dictionary jointly represents words and gestures through co-occurrence and non-co-occurrence features.For each word and gesture pair, the dictionary records whether the word occurs with or without that gesture.
- Sentiment Analysis Baselines: Human predictions provide a sentiment-intensity baseline and a target for future machine-learning methods.The human baseline asks people to predict the sentiment score of each opinion segment.
- Results: The multimodal dictionary outperforms simple feature concatenation, and combining verbal and visual cues improves sentiment prediction.The discussion reports that the multimodal dictionary has the best performance among the sentiment baselines.
- Subjectivity Analysis: Subjectivity baselines use linear C-SVM and deep neural networks with automatically extracted audio-visual features.The neural network varies the number of layers from 1 to 5 and neurons per layer from 10 to 50; Table 4 reports accuracy.
5 Discussion
The experiments show that combining verbal and visual cues improves sentiment and subjectivity prediction, with the multimodal dictionary performing best among sentiment baselines. Visual gestures can correct or sharpen predictions based on words alone, although human performance leaves room for further research.
- Discussion: Combining verbal and visual cues improves sentiment prediction, while the multimodal dictionary performs best among the sentiment baselines.The same discussion reports that multimodal information helps both sentiment and subjectivity analysis.
- Discussion: Visual gestures refine verbal predictions by strengthening neutral interpretations or moderating highly positive ones.Strong positive visual cues can raise a neutral verbal prediction, while a negative headshake can reduce an overly positive interpretation.
- Discussion: Human performance indicates that sentiment and subjectivity analysis on MOSI still has room for future research.This conclusion is stated alongside the reported benefits of multimodal information.
6 Conclusion
The paper introduces MOSI, an opinion-level multimodal dataset for sentiment intensity and subjectivity analysis. It combines aligned text, visual, and audio annotations with a word-gesture representation, and reports benefits from using information beyond text.
- Conclusion: MOSI is presented as the first multimodal sentiment-analysis dataset with opinion-level sentiment-intensity and subjectivity annotations.The dataset contains manual and automatic annotations for text, visual, and audio features.
- Conclusion: The dataset aligns text, visual features, and audio features for multimodal studies of language.Its annotations include per-frame visual features, per-opinion annotations, and per-millisecond audio features in the paper context.
- Conclusion: The paper presents a representation that captures the joint distribution of words and gestures using statistical observations from MOSI.This representation supports modeling information beyond text.
- Conclusion: Using information beyond text helps models make better sentiment-intensity and subjectivity predictions.The conclusion states this outcome for both tasks without specifying a single metric.
- Conclusion: MOSI is publicly available with extracted features to support further studies of sentiment and subjectivity in multimedia content.The paper frames the dataset as an opening for more detailed future research.