Source-linked AI summary
Multimodal Sentiment Analysis with Word-Level Fusion and Reinforcement Learning
Minghai Chen, Sen Wang, Paul Pu Liang, Tadas Baltrušaitis, Amir Zadeh, Louis-Philippe Morency
TL;DR
Multimodal sentiment analysis needs to combine language, acoustic, and visual information beyond holistic video-level representations. The paper introduces GME-LSTM(A), which performs gated word-level fusion and temporal attention, and reports state-of-the-art classification and regression results on CMU-MOSI.
Problem
Earlier multimodal sentiment methods focused on holistic speech-segment features and abstract video-level fusion, limiting finer-grained modeling of multimodal speech structure.
Method
GME-LSTM(A) combines a gated multimodal embedding for selective word-level fusion with an LSTM Temporal Attention model for important speech moments.
Results
4.08% in binary classification accuracy and 13.2% in MAE improvement over the state of the art on CMU-MOSI.
Takeaways & Limitations
The analysis identifies Temporal Attention and Gated Multimodal Embedding as crucial components for multimodal fusion and sentiment prediction.
Abstract
from arXiv · showhide
With the increasing popularity of video sharing websites such as YouTube and Facebook, multimodal sentiment analysis has received increasing attention from the scientific community. Contrary to previous works in multimodal sentiment analysis which focus on holistic information in speech segments such as bag of words representations and average facial expression intensity, we develop a novel deep architecture for multimodal sentiment analysis that performs modality fusion at the word level. In this paper, we propose the Gated Multimodal Embedding LSTM with Temporal Attention (GME-LSTM(A)) model that is composed of 2 modules. The Gated Multimodal Embedding alleviates the difficulties of fusion when there are noisy modalities. The LSTM with Temporal Attention performs word level fusion at a finer fusion resolution between input modalities and attends to the most important time steps. As a result, the GME-LSTM(A) is able to better model the multimodal structure of speech through time and perform better sentiment comprehension. We demonstrate the effectiveness of this approach on the publicly-available Multimodal Corpus of Sentiment Intensity and Subjectivity Analysis (CMU-MOSI) dataset by achieving state-of-the-art sentiment classification and regression results. Qualitative analysis on our model emphasizes the importance of the Temporal Attention Layer in sentiment prediction because the additional acoustic and visual modalities are noisy. We also demonstrate the effectiveness of the Gated Multimodal Embedding in selectively filtering these noisy modalities out. Our results and analysis open new areas in the study of sentiment analysis in human communication and provide new models for multimodal fusion.
1 INTRODUCTION
Multimodal sentiment analysis combines language, acoustic, and visual cues, but earlier video-level fusion largely ignored speech structure. The paper addresses this with word-level gated fusion and temporal attention.
- Earlier approaches fused bag-of-words, average smile intensity, and other whole-video statistics, overlooking speech structure.These methods combined modalities at an abstract video level.
- Speech-level multimodal fusion can clarify ambiguity, sarcasm, and sentiment intensity by connecting words with acoustic and visual behavior.Examples include “crazy” with a smile or frown and “great” accompanied by a frown.
- The model must select useful modalities at each timestep and focus attention on important moments in communication.The introduction frames modality gating and temporal attention as the two central design questions.
- GME-LSTM(A) performs word-level multimodal fusion with a gated mechanism and uses Temporal Attention for sentiment prediction.Its recurrent state captures global interactions while word-level fusion captures local interactions.
- The paper evaluates the proposed approach on CMU-MOSI and organizes the study around related work, methods, experiments, results, analysis, and conclusion.The section roadmap identifies the dataset, methodology, results, component analysis, and conclusion sections.
2 RELATED WORK
Related work spans multimodal learning, written-text sentiment analysis, and multimodal sentiment datasets and models. The paper positions its method as introducing word-level fusion and reinforcement-learning-based control for noisy modalities.
- Multimodal machine learning has been applied to sentiment analysis, medical applications, emotion recognition, image captioning, question answering, and translation.The related-work overview places sentiment analysis within a broader set of multimodal tasks.
- Written-text sentiment analysis has progressed from bag-of-words and n-grams to opinion-word methods, trees, graphs, and dependency-based semantic analysis.These approaches derive sentence sentiment from words and their compositions using increasingly structured representations.
- CMU-MOSI is described as the first dataset with opinion-level sentiment labels among multimodal sentiment datasets.It integrates verbal and nonverbal behavior for sentiment prediction.
- Recent multimodal models use deep neural architectures and explore early or late fusion, while some prior work concatenates modalities at the video level.The paper distinguishes its finer-grained fusion from video-level concatenation.
- Related approaches also model multimodal data through conditional independence assumptions, correlation-sensitive kernels, or recurrent controllers for CNN structure.These methods provide alternative treatments of multimodal dependence and model control.
- The proposed method claims two advances: word-level modality alignment and attention plus a reinforcement-learning-trained input gate for noisy modalities.Words are aligned with corresponding video frames and audio segments.
3 PROPOSED APPROACHES
GME-LSTM(A) combines word-level gated multimodal fusion with an LSTM and temporal attention to model local and global interactions while filtering noisy modalities.
- Gated Multimodal Embedding: The Gated Multimodal Embedding performs selective fusion of aligned language, acoustic, and visual features at each word-level timestep.Words are aligned with corresponding video and audio segments using P2FA before feature fusion.
- Gated Multimodal Embedding: Input gate controllers independently decide whether acoustic and visual features contribute at each timestep, while textual features always enter the model.Rejected modality inputs are replaced with 0; accepted inputs remain unchanged.
- LSTM with Temporal Attention: The word-level multimodal inputs are concatenated and passed through an LSTM that captures global temporal interactions for sentiment prediction.The LSTM uses input, forget, and output gates, memory cell c, and hidden output h.
- LSTM with Temporal Attention: Temporal attention assigns higher weights to important communication moments and combines LSTM hidden states for final sentiment prediction.The attention weights are computed with a shared vector and softmax, then used to weight each timestep’s hidden representation.
- Training Details: The gate controllers are trained with reinforcement learning because controller outputs change validation MAE discretely and make the objective non-convex and non-differentiable.The reward is e^-L, where L is validation MAE; REINFORCE with sampled controller outputs is used for updates.
4 EXPERIMENTAL METHODOLOGY
The experiments evaluate GME-LSTM(A) on CMU-MOSI using aligned text, visual, and audio inputs, speaker-independent splits, selected features, and specified training settings.
- CMU-MOSI Dataset: Experiments use CMU-MOSI, an online-video dataset of movie opinions with continuous clip-level sentiment labels ranging from −3 to +3.Each clip contains one opinion expressed by one or more sentences.
- CMU-MOSI Dataset: CMU-MOSI contains 93 videos and 2,199 labeled clips, split into 52 training videos, 10 validation videos, and 31 test videos.The splits contain 1,284, 229, and 686 clips respectively, with distinct speakers across videos.
- Input Features: The model uses text, video, and audio modalities, with pretrained 300-dimensional GloVe embeddings for transcript words and COVAREP acoustic features.Visual features are extracted with Facet; the selected Facet and COVAREP features are normalized using training-set maximum absolute values.
- Experimental Details: LSTM(A) uses 64 hidden units, maximum sequence length T=115, a 50-unit ReLU layer, ADAM learning rate 0.0005, and MAE loss.GME-LSTM(A) controllers use one 32-unit hidden layer with sigmoid activation and sample n=5 inputs per training step.
5 EXPERIMENTAL RESULTS
The experiments compare the proposed models with multimodal and language-based baselines, showing that GME-LSTM(A) achieves the strongest reported performance.
- Compared models: The baselines include neural, kernel-learning, support-vector-machine, random-forest, random, and human-performance comparisons.The compared language-based models include DAN, D-CNN, and RNTN, while multimodal baselines include SAL-CNN, SVM-MD, and C-MKL.
- Multimodal results: GME-LSTM(A) improves upon the state of the art by 4.08% in binary classification accuracy and 13.2% in MAE.It gives the best result across all compared models.
- Multimodal results: Both GME-LSTM(A) and LSTM(A) outperform multimodal and single-modality sentiment analysis models.The comparison includes the proposed gated and ungated approaches.
- Language-based comparison: GME-LSTM(A) immediately outperforms all language-based and multimodal baseline models after adding audio and video with multimodal fusion.The language-only LSTM(A) performs slightly worse than some language-based methods using more complex dependency-based models.
6 DISCUSSION
The discussion finds that temporal attention and gated multimodal embedding are both crucial for fusing modalities and predicting sentiment. Temporal attention helps exploit aligned visual and acoustic cues while reducing the impact of noisy frames, and gating further selects useful modality inputs.
- Temporal Attention Layer and Gated Multimodal Embedding are both crucial for multimodal fusion and sentiment prediction.
- LSTM with Temporal Attention Analysis: Text alone is stronger than unimodal audio or visual inputs, while ungated multimodal fusion can slightly worsen F-score and MAE.The audio and video features are described as noisy when fusion is not carefully performed.
- LSTM with Temporal Attention Analysis: Temporal attention improves sentiment prediction across modality combinations by focusing on important moments and abandoning noisy frames.The LSTM(A) model consistently outperforms the LSTM model across all modality combinations.
- LSTM with Temporal Attention Analysis: Word-level fusion lets LSTM(A) use aligned audio and visual indicators to resolve ambiguous words and improve video-sentiment predictions.Examples include using disappointed or sad facial expressions when transcript words such as “want” or “lines” are ambiguous.
- Gated Multimodal Embedding Analysis: GME-LSTM(A) outperforms LSTM(A) on all metrics by selectively accepting expressive visual frames and rejecting frames with blank or noisy expressions.In one example, the gated model predicts 1.57 versus -0.94 for LSTM(A), with ground truth sentiment 3.0.
7 CONCLUSION
The paper proposes Gated Multimodal Embedding LSTM with Temporal Attention for word-level multimodal fusion, using reinforcement-learning-trained gating and attention to model speech dynamics.
- The Gated Multimodal Embedding LSTM with Temporal Attention performs multimodal fusion at the word level.The model uses selective word-level fusion between modalities.
- Reinforcement-learning-trained gating selectively controls fusion between modalities at each word.The gating mechanism is designed for the complex structure of speech.
- Temporal attention focuses the model on important moments in speech.The stateful model also captures long interactions between modalities.
- The approach achieves state-of-the-art performance on the CMU-MOSI dataset and includes qualitative analysis of communication dynamics.