Source-linked AI summary
Words Can Shift: Dynamically Adjusting Word Representations Using Nonverbal Behaviors
Yansen Wang, Ying Shen, Zhun Liu, Paul Pu Liang, Amir Zadeh, Louis-Philippe Morency
TL;DR
Words can change meaning with facial and vocal context, but multimodal language models often miss these fine-grained, dynamic nonverbal cues. RAVEN models nonverbal subword structure and shifts word representations accordingly, achieving competitive performance on sentiment analysis and emotion recognition while producing interpretable variation patterns.
Problem
Multimodal language modeling lacks fine-grained treatment of visual and acoustic subword sequences that can alter a word’s conveyed intent.
Method
RAVEN models nonverbal subword structure and dynamically shifts word representations conditioned on accompanying visual and acoustic behaviors.
Results
RAVEN achieves competitive performance on multimodal sentiment analysis and emotion recognition, with ablations showing both subword analysis and multimodal shifting are critical for good performance.
Takeaways & Limitations
Shifted word representations form interpretable context-dependent patterns, including stronger variation for polarity-incongruent contexts and minimal variation for sentiment-irrelevant words.
Abstract
from arXiv · showhide
Humans convey their intentions through the usage of both verbal and nonverbal behaviors during face-to-face communication. Speaker intentions often vary dynamically depending on different nonverbal contexts, such as vocal patterns and facial expressions. As a result, when modeling human language, it is essential to not only consider the literal meaning of the words but also the nonverbal contexts in which these words appear. To better model human language, we first model expressive nonverbal representations by analyzing the fine-grained visual and acoustic patterns that occur during word segments. In addition, we seek to capture the dynamic nature of nonverbal intents by shifting word representations based on the accompanying nonverbal behaviors. To this end, we propose the Recurrent Attended Variation Embedding Network (RAVEN) that models the fine-grained structure of nonverbal subword sequences and dynamically shifts word representations based on nonverbal cues. Our proposed model achieves competitive performance on two publicly available datasets for multimodal sentiment analysis and emotion recognition. We also visualize the shifted word representations in different nonverbal contexts and summarize common patterns regarding multimodal variations of word representations.
Recurrent Attended Variation Embedding Network (RAVEN) · Introduction
RAVEN addresses the limits of text-only word embeddings by modeling fine-grained visual and acoustic subword sequences and dynamically shifting word representations according to nonverbal cues. The proposed approach is evaluated on multimodal sentiment analysis and emotion recognition.
- Introduction: Text-pretrained word embeddings are insufficient for highly dynamic multimodal language because identical words can convey different sentiments with opposing facial expressions and intonations.The paper illustrates this contrast using the adjective “sick” when describing movies.
- Introduction: The same underlying word can vary in sentiment when paired with different nonverbal behaviors.Figure 1 depicts variation in the word representation of “sick” under differing facial expressions and intonations.
- Introduction: Modeling nonverbal context requires fine-grained analysis because visual and acoustic behaviors occur at higher temporal frequency than words.Each uttered word is accompanied by a sequence of visual and acoustic “subword” units.
- Introduction: The structure of accompanying visual and acoustic subword sequences is important for representing nonverbal dynamics.The paper motivates subword analysis as necessary for capturing the fine-grained structure of behaviors concurrent with uttered words.
- Recurrent Attended Variation Embedding Network (RAVEN): RAVEN models fine-grained nonverbal subword structure and dynamically shifts word representations based on accompanying nonverbal cues.Its name is Recurrent Attended Variation Embedding Network.
- Recurrent Attended Variation Embedding Network (RAVEN): The authors conduct experiments on multimodal sentiment analysis and emotion recognition to test the importance of subword analysis and dynamic word meanings.These experiments are designed to verify the paper’s hypotheses about subword information and dynamic behaviors of word meanings.
Related Works
Prior work spans subword word representations, context-dependent word representations, and multimodal sentiment and emotion recognition. However, limited research has modeled variations in word representations arising from nonverbal behaviors.
- Research scope: Prior multimodal machine-learning research has modeled multiple modalities, but relatively little work addresses word-representation variation caused by nonverbal behaviors.The paper organizes prior research into subword representations, representation variation, and multimodal sentiment and emotion recognition.
- Subword word representations: Subword-to-word compositional representations help models infer representations for words absent from the training vocabulary.This approach has been applied especially to machine translation, language modeling, and word representation learning.
- Subword word representations: Character-convolution neural word representations contain transferable language information for question answering, textual entailment, sentiment analysis, and natural language inference.The cited work describes deep word representations learned through neural models with character convolutions.
- Modeling representation variations: Context-dependent word-representation research learns different representations for a word’s senses using Bayesian Nonparametric models, Gaussian Mixture Models, or extended FastText representations.The passage identifies methods by Li and Jurafsky, Nguyen et al., and Athiwaratkun, Wilson, and Anandkumar.
- Multimodal sentiment and emotion recognition: Early fusion concatenates multimodal inputs and can outperform unimodal models, but it has limited modality-specific interaction learning and tends to overfit.The passage also states that early-fusion methods can learn robust representations.
Recurrent Attended Variation Embedding Network (RAVEN)
RAVEN models fine-grained visual and acoustic subword behaviors and uses them to dynamically shift word representations conditioned on nonverbal context. Its gated, recurrent architecture produces multimodal representations for downstream sentiment and emotion prediction.
- Architecture: RAVEN combines Nonverbal Sub-networks, a Gated Modality-mixing Network, and Multimodal Shifting to condition word representations on accompanying visual and acoustic behaviors.The model is designed to capture both subword nonverbal structure and multimodal shifts in word meaning.
- Nonverbal Sub-networks: Separate modality-specific LSTMs encode visual and acoustic frame sequences aligned with each word to produce nonverbal embeddings.The inputs are temporal visual and acoustic subword units carried alongside each word.
- Gated Modality-mixing Network: Attention gating learns a nonlinear combination of visual and acoustic embeddings, while modality-specific influence gates control each modality’s contribution to the nonverbal shift vector.The gates account for context-dependent relative importance and intensity of visual and acoustic cues.
- Multimodal Shifting: Multimodal Shifting integrates the nonverbal shift vector into each original word embedding to create a shifted word-representation sequence.A scaling factor constrains shift magnitude while preserving its direction, and threshold β is selected by validation-set cross-validation.
- Downstream Prediction: A word-level LSTM encodes the shifted sequence into an utterance-level multimodal representation for sentiment or emotion prediction, with all components trained end-to-end.The utterance representation is passed to a fully connected layer for task-specific outputs.
Experiments
Experiments evaluate RAVEN on multimodal sentiment analysis and emotion recognition using CMU-MOSI and IEMOCAP, with standardized language, visual, and acoustic features. The evaluation compares RAVEN with conventional and multimodal neural baselines using task-appropriate metrics.
- Tasks and Datasets: RAVEN is evaluated on CMU-MOSI for multimodal sentiment analysis and IEMOCAP for multimodal emotion recognition.Both datasets involve tri-modal human communications.
- Tasks and Datasets: CMU-MOSI contains 2,199 video segments from 93 YouTube movie reviews with sentiment-intensity annotations ranging from [−3,+3].Negative values indicate negative sentiment, and positive values indicate positive sentiment.
- Tasks and Datasets: IEMOCAP contains 151 dyadic-interaction videos featuring scripted emotional scenes and annotations for nine emotions.The emotions include angry, excited, fear, sad, surprised, frustrated, happy, disappointed, and neutral.
- Feature Extraction: The experiments use GloVe language vectors, FACET facial-expression features, and COVAREP acoustic features.GloVe vectors are 300-dimensional and trained on 840B tokens; FACET extracts visual features at 30Hz, while COVAREP extracts 74 acoustic features at 100Hz.
- Baselines: RAVEN is compared against SVM, Deep Fusion, BC-LSTM, MV-LSTM, MARN, MFN, RMFN, and LMF multimodal baselines.These models represent concatenated-feature, late-fusion, context-dependent, interaction-focused, memory-based, multistage, and low-rank fusion approaches.
Results and Discussion
RAVEN performs competitively across multimodal sentiment and emotion-recognition tasks while learning interpretable, context-dependent shifts in word representations. Visualization and ablation results indicate that these shifts capture consistent multimodal patterns and that both subword modeling and multimodal shifting are important for performance.
- Task Results: RAVEN achieves competitive performance across multimodal sentiment prediction and emotion recognition tasks.It uses a simple LSTM for prediction, which could be enhanced with more advanced modules such as temporal attention.
- Representation Analysis: RAVEN learns consistent shifts for instances of the same word by integrating each word with its accompanying nonverbal contexts.The model visualizes shifted representations on CMU-MOSI after projecting them into 2-dimensional space with PCA and plotting Gaussian contours.
- Representation Analysis: Words with inherent polarity vary strongly in opposite-sentiment contexts but minimally in their default contexts.For example, “great” has an overall centroid close to its positive centroid, while its negative centroid is far from both.
- Representation Analysis: Nouns used in both positive and negative contexts are polarized into distinct regions based on accompanying multimodal context.Their positive and negative centroids are far from the overall centroid and typically occupy different half-planes.
- Representation Analysis: Words that are not critical for sentiment exhibit minimal variation across positive and negative contexts, keeping their centroids close together.The examples “that” and “the” show this pattern.
- Ablation Studies: Ablation results show that both Nonverbal Sub-networks and Multimodal Shifting are necessary for achieving state-of-the-art performance.Removing these components disables subword modeling, dynamic adjustment in the original word-embedding space, or both; crude averaging of higher-frequency visual and acoustic sequences can hurt performance.
Conclusion
The paper presents RAVEN, a model that captures fine-grained nonverbal behaviors at the subword level and dynamically shifts word representations across nonverbal contexts. It achieves competitive results on multimodal sentiment analysis and emotion recognition.
- RAVEN models the fine-grained structure of nonverbal behaviors at the subword level.
- RAVEN builds multimodal-shifted word representations that dynamically capture variations across different nonverbal contexts.
- RAVEN achieves competitive results on established multimodal sentiment analysis and emotion recognition tasks.