Source-linked AI summary
Tensor Fusion Network for Multimodal Sentiment Analysis
Amir Zadeh, Minghai Chen, Soujanya Poria, Erik Cambria, Louis-Philippe Morency
TL;DR
Multimodal sentiment analysis must account for interactions within and across language, visual, and acoustic modalities, especially in volatile spoken opinions. The paper introduces Tensor Fusion Network, which learns these dynamics end-to-end, and reports state-of-the-art performance for multimodal and unimodal sentiment analysis on CMU-MOSI.
Problem
Multimodal sentiment analysis needs to model inter-modality and intra-modality dynamics in spoken-language videos involving language, visual, and acoustic behaviors.
Method
TFN uses modality embedding subnetworks, a Tensor Fusion Layer for unimodal, bimodal, and trimodal interactions, and a sentiment inference subnetwork.
Results
TFN achieves state-of-the-art performance against multimodal approaches and for language-only, visual-only, and acoustic-only sentiment analysis on CMU-MOSI.
Takeaways & Limitations
The approach provides an end-to-end fusion method that explicitly represents unimodal, bimodal, and trimodal interactions for sentiment analysis.
Takeaways & Limitations
Language subnetworks remain constrained because spoken-language sentiment cues may be diluted by unrelated words or lack spatial proximity.
Abstract
from arXiv · showhide
Multimodal sentiment analysis is an increasingly popular research area, which extends the conventional language-based definition of sentiment analysis to a multimodal setup where other relevant modalities accompany language. In this paper, we pose the problem of multimodal sentiment analysis as modeling intra-modality and inter-modality dynamics. We introduce a novel model, termed Tensor Fusion Network, which learns both such dynamics end-to-end. The proposed approach is tailored for the volatile nature of spoken language in online videos as well as accompanying gestures and voice. In the experiments, our model outperforms state-of-the-art approaches for both multimodal and unimodal sentiment analysis.
1 Introduction
Multimodal sentiment analysis extends text-based sentiment analysis to videos containing language, visual gestures, and acoustic voice. The paper frames its challenges as modeling volatile spoken language plus intra- and inter-modality dynamics, and introduces TFN to learn both end-to-end.
- Multimodal sentiment analysis combines spoken language, visual gestures, and acoustic voice to analyze sentiment in opinionated videos.
- Inter-modality dynamics arise when language, visual behavior, and acoustic behavior jointly change perceived sentiment.
- Intra-modality dynamics are also challenging because relevant behavior unfolds within each modality across space and time.
- Spoken opinions often ignore proper language structure, making their volatile form difficult to analyze.
- Earlier approaches used early or late fusion, which did not directly model both intra-modality and inter-modality dynamics.
- TFN learns both dynamics end-to-end through modality embedding subnetworks and Tensor Fusion, which explicitly aggregates unimodal, bimodal, and trimodal interactions.
- Experiments report that TFN outperforms prior multimodal approaches, while its modality subnetworks also outperform unimodal state-of-the-art approaches.
2 Related Work
Related work situates multimodal sentiment analysis within NLP, audio-visual emotion recognition, and broader multimodal machine learning. Existing efforts integrate verbal and nonverbal behaviors using diverse datasets and modeling approaches.
- Sentiment analysis research includes opinion-word methods, n-grams and language models, compositionality and dependency analysis, and distributional sentiment representations.
- Multimodal sentiment analysis integrates verbal and nonverbal behaviors to detect user sentiment.
- CMU-MOSI is identified as the only English dataset among those discussed with utterance-level sentiment labels.
- Related multimodal approaches include CNNs with multiple-kernel learning, SAL-CNN, and SVMs with a multimodal dictionary.
- Audio-visual emotion recognition uses audio and visual features jointly, while multimodal machine learning applies multiple modalities across diverse tasks.
3 CMU-MOSI Dataset
CMU-MOSI is an annotated YouTube movie-review dataset designed for spoken opinion analysis. It uses utterance-level sentiment labels on a seven-step scale and includes 2,199 utterances from 93 speakers.
- CMU-MOSI contains annotated video opinions from YouTube movie reviews.
- Figure 2 depicts sentiment distributions across opinions and opinion sizes in CMU-MOSI.
- Sentiment annotations follow a seven-step Likert scale ranging from very negative to very positive.
- Unlike sentence-segmented text datasets, CMU-MOSI is segmented by opinion utterances because spoken-language sentence boundaries are less clear.
- CMU-MOSI includes 2,199 opinion utterances from 93 distinct speakers.
4 Tensor Fusion Network
The Tensor Fusion Network combines modality-specific embeddings with explicit unimodal, bimodal, and trimodal interactions before sentiment inference. Its subnetworks model language, visual, and acoustic inputs, while Tensor Fusion represents their combinations through an outer product.
- Architecture: TFN uses Modality Embedding Subnetworks, a Tensor Fusion Layer, and a Sentiment Inference Subnetwork.The subnetworks produce modality embeddings, Tensor Fusion models their interactions, and the inference network predicts sentiment for classification or regression tasks.
- Modality Embedding Subnetworks: The spoken-language subnetwork uses time-dependent word representations and a fully connected network to produce a language embedding.An LSTM processes sequences of 300-dimensional GloVe word vectors, and the resulting representations are passed to a fully connected network.
- Modality Embedding Subnetworks: Visual embeddings are generated from pooled frame-level facial, head-position, head-rotation, and landmark features.FACET supplies emotion and facial-action features, while OpenFace supplies head and landmark estimates before visual embedding.
- Modality Embedding Subnetworks: Acoustic embeddings are generated from mean-pooled audio features extracted with COVAREP and processed by a three-layer network of 32 ReLU units.The extracted features include MFCCs, pitch, voiced/unvoiced indicators, glottal-source parameters, and peak-slope parameters.
- Tensor Fusion Layer: Tensor Fusion explicitly models unimodal, bimodal, and trimodal interactions through a three-fold Cartesian product of modality embeddings.The outer product forms separate regions for individual modalities, modality pairs, and the joint language-visual-acoustic interaction; early fusion is a special case containing only unimodal interactions.
5 Experiments
The experiments evaluate TFN against multimodal and unimodal state-of-the-art approaches, while ablations examine its subtensors, modalities, and comparison with early fusion. TFN outperforms prior multimodal approaches, and its modality-specific networks also outperform unimodal state-of-the-art systems.
- Experiment 1: The experiments compare TFN with previous multimodal sentiment-analysis approaches, including neural and non-neural baselines.The evaluation includes comparisons with state-of-the-art models and reports results in Table 1.
- Experiment 2: Ablations evaluate TFN subtensors, individual modalities, and the commonly used early-fusion approach.The experiments test the importance of tensor components and compare concatenated modality embeddings with Tensor Fusion.
- Experiment 3: The modality-specific networks outperform state-of-the-art unimodal approaches for language, visual, and acoustic sentiment analysis.The experiments compare language, visual, and acoustic TFN variants with corresponding unimodal baselines; language is reported as the most predictive modality in the ablation.
- Experiment 1: TFN outperforms previously proposed neural and non-neural approaches, with the difference especially visible for 5-class classification.The comparison is reported using the ΔSOTA measure in Table 1.
- Experiment 2: Removing the trimodal subtensor produces a large improvement gap relative to full TFN, supporting the importance of trimodal dynamics and all tensor components.Bimodal subtensors are also reported as more informative when used without other subtensors.
- Experiment 2: TFN outperforms early fusion, which concatenates the three modality embeddings before passing them to the sentiment inference network.The early-fusion comparison is described as TFNearly.
6 Qualitative Analysis
Qualitative examples show that TFN integrates complementary language, visual, and acoustic evidence, whereas early fusion largely follows the language modality.
- Table 6 compares unimodal models, early fusion, and TFN on CMU-MOSI examples scored from -3 to +3.
- TFNearly mostly replicates language predictions, while TFN integrates intermodality dynamics to estimate sentiment levels.
- TFN captures a bimodal interaction when a negative facial expression correctly shifts a weakly negative utterance toward the observed sentiment.
- TFN identifies trimodal interactions when ambiguous words gain complementary acoustic and visual evidence supporting a positive prediction.
- Strong negative visual behavior can counter positive language and move TFN’s prediction toward neutrality, while acoustic behavior also influences predictions.
7 Conclusion
The paper introduces an end-to-end fusion method that explicitly represents unimodal, bimodal, and trimodal interactions. On CMU-MOSI, it reports state-of-the-art performance for multimodal and modality-specific sentiment analysis.
- TFN explicitly represents unimodal, bimodal, and trimodal interactions between behaviors through an end-to-end fusion method.
- TFN achieved state-of-the-art performance on CMU-MOSI against multimodal approaches and for language-only, visual-only, and acoustic-only analysis.