Source-linked AI summary
Multilogue-Net: A Context Aware RNN for Multi-modal Emotion Detection and Sentiment Analysis in Conversation
Aman Shenoy, Ashish Sardana
TL;DR
Multimodal conversation analysis needs to capture modality-specific information, conversational context, and speaker-listener emotional dynamics. Multilogue-Net combines modality-specific state and context modeling with pairwise attention, and the paper reports consistent performance across multimodal benchmark settings, with results varying by dataset and modality subset.
Problem
Existing multimodal systems inadequately model modality-specific information, conversation context, modality relationships, and the dynamics between speakers and listeners.
Source: §1 · p. 1Method
Multilogue-Net models state, emotion, and context with modality-specific recurrent representations, then uses pairwise attention to fuse modality information.
Source: §1 · pp. 1–2, §3.2 · p. 3Results
Multilogue-Net consistently performs well on CMU-MOSI and CMU-MOSEI across multimodal settings; on CMU-MOSI it outperforms the current state of the art on only one modality subset.
Source: §4.3 · p. 8, §6 · p. 9Takeaways & Limitations
The paper concludes that its model treats modalities independently while capturing conversation context and performs consistently on the evaluated multimodal benchmarks.
Source: §6 · p. 9Further extensions to feature extractors, modalities, and participants are future work because labeled datasets with those extensions are unavailable.
Source: §6 · p. 9
Abstract
from arXiv · showhide
Sentiment Analysis and Emotion Detection in conversation is key in several real-world applications, with an increase in modalities available aiding a better understanding of the underlying emotions. Multi-modal Emotion Detection and Sentiment Analysis can be particularly useful, as applications will be able to use specific subsets of available modalities, as per the available data. Current systems dealing with Multi-modal functionality fail to leverage and capture - the context of the conversation through all modalities, the dependency between the listener(s) and speaker emotional states, and the relevance and relationship between the available modalities. In this paper, we propose an end to end RNN architecture that attempts to take into account all the mentioned drawbacks. Our proposed model, at the time of writing, out-performs the state of the art on a benchmark dataset on a variety of accuracy and regression metrics.
1 Introduction
Multimodal conversation analysis must account for the distinct information held by each modality while modeling conversational context and speaker-listener dynamics. Multilogue-Net uses modality-specific representations and pairwise attention to address these gaps.
1 Introduction
- Existing systems inadequately account for modality-specific information, multimodal fusion, conversational context, speaker-listener state separation, and the emotional effect of an utterance on listeners.Source: §1 · p. 1
- Multilogue-Net models interlocutor state, context, and emotion as interrelated factors, using multiple modality-specific state vectors and pairwise attention across modalities.The paper sets aside interlocutor intent because it depends on prior knowledge about the speaker.Source: §1 · p. 1
- The model uses two GRUs per modality for interlocutor state and emotion, plus an interconnected context network with one GRU per available modality.Current utterance representations and historical GRU outputs support predictions at each timestamp.Source: §1 · pp. 1–2
- The model produces one emotional-state representation per modality at each timestamp and fuses them for the final prediction; text-only context-GRU input was observed to be key to results.Source: §1 · p. 2
2 Related Work
Prior work explored multimodal cues, recurrent models, and speaker interaction for emotion recognition. DialogueRNN and pairwise attention advanced conversation modeling, while motivating stronger multimodal representations and fusion.
2 Related Work
- Earlier work fused acoustic and visual cues for emotion recognition, while other work used contextual information in multimodal settings.Source: §2 · p. 2
- Deep recurrent networks built on learned modality representations, with Poria et al. (2017) followed by works including Chen et al. (2017) and Zadeh et al. (2018a; 2018c).Source: §2 · p. 2
- DialogueRNN (Majumder et al., 2018) tracks interlocutor states and conversational context, but its multimodal performance was held back by limited multimodal representation and fusion.It extends a text-focused model to multimodal inputs by concatenating modality representations.Source: §2 · p. 2
- Ghosal et al. (2018) used pairwise attention to emphasize similar attributes and diminish differences between modality representations, achieving state-of-the-art multimodal results.Source: §2 · p. 2
3.1 Problem Formulation
The task assigns sentiment and emotion predictions to each utterance and participant. Predictions use utterance representations together with the participant’s previous emotional state and the conversation context.
3.1 Problem Formulation
- Each utterance receives a sentiment score and one of six emotion labels, and is associated with the participant who spoke it.Source: §3.1 · p. 3
- Utterance-level predictions allow participant-average sentiment to be captured and avoid fixed-interval classification during long silences.Source: §3.1 · p. 3
- The prediction function maps three independent utterance representations, the participant’s previous emotional state, and current conversation context to a sentiment score and emotion label.Source: §3.1 · p. 3
3.2 Model Details
Multilogue-Net models interlocutor state, emotion, and conversational context across modalities, then fuses modality-specific emotion representations for prediction. Its design explicitly models three of four proposed determinants of utterance sentiment or emotion, treating interlocutor intent as implicit.
3.2 Model Details
- The model explicitly represents interlocutor state, emotion, and conversational context, while assuming interlocutor intent will be learned implicitly.Its design assumes sentiment or emotion depends on interlocutor state and intent, preceding and future emotions, and conversational context.Source: §3.2 · p. 3
- Each modality has context, state, and emotion GRUs, while context GRUs are shared across participants in the conversation.The model has 2mp + m GRUs for p participants and m modalities.Source: §3.2 · p. 3
3.2.1 Context GRU (cGRU)
- The context GRU combines the current modality-specific utterance with the previous speaker-state output to update the context representation.The stated update uses context vectors of size Dc and concatenation of the modality utterance and prior speaker states.Source: §3.2.1 · p. 4
3.2.2 State GRU (sGRU)
- State GRUs update each participant’s modality-specific state using the current utterance and attention pooled over previous context representations.Attention scores weight previous context vectors, which are pooled to form the context input to the state update.Source: §3.2.2 · p. 4
3.2.3 Emotion GRU (eGRU)
- Emotion GRUs decode state-GRU encodings together with their previous emotion outputs into representations used for prediction.The resulting emotion representation is also passed to the pairwise attention mechanism.Source: §3.2.3 · p. 5
3.2.4 Pairwise Attention Mechanism
- Pairwise attention forms pairwise fusion representations from modality-specific emotion vectors and concatenates them for prediction.For text, audio, and video, the model forms all three modality pairs.Source: §3.2.4 · pp. 5–6
3.2.5 Final Predictions
- Sentiment predictions concatenate the three pairwise-attention outputs with emotion-GRU outputs before a fully connected layer and softmax or tanh.Emotion prediction instead uses a fully connected layer and softmax to produce probabilities for six classes from Lt.Source: §3.2.5 · p. 6
3.2.6 Training
- Emotion training uses categorical cross-entropy with L2 regularization, while sentiment regression uses MSE with L2 regularization.For sentiment, the authors report using small-weight initialization and Adam with mini-batch gradient descent to avoid saturation-related very small gradients.Source: §3.2.6 · p. 6
4 Experiments, Datasets, and Results
The experiments evaluate Multilogue-Net on CMU-MOSI and CMU-MOSEI, with results varying by dataset and modality subset. On CMU-MOSEI, the authors report gains over prior work on most sentiment metrics and emotion labels.
4.1 Datasets
- The evaluation uses CMU-MOSI and CMU-MOSEI, reporting performance on sentiment and, for CMU-MOSEI, emotion tasks.Source: §4.1 · p. 6, §4.1.2 · p. 7
4.1.1 CMU-MOSI
- CMU-MOSI contains 93 videos and 2,199 utterances, split into 1,151 training, 296 validation, and 752 test utterances; labels are positive or negative.Source: §4.1.1 · pp. 6–7
4.1.2 CMU-MOSEI
- CMU-MOSEI sentiment labels range from -3 to +3; the study also evaluates six emotion labels and a two-class A2 sentiment setup.Source: §4.1.2 · p. 7
4.2.1 CMU-MOSEI
- For CMU-MOSEI, utterance features average sentiment-label-level features: 300-dimensional GloVe text, 35-dimensional Facet visual, and 384-dimensional OpenSMILE acoustic features.Source: §4.2.1 · p. 7
4.2.2 CMU-MOSI
- For CMU-MOSI, the study uses provided utterance-level text, visual, and acoustic features with dimensions 100, 100, and 73, respectively.Source: §4.2.2 · p. 7
4.3 Experiments
- On CMU-MOSEI, Multilogue-Net outperforms the state of the art on all but one tri-modal sentiment metric and by a considerable margin on emotion labels.Source: Table 2 · p. 7, Table 3 · p. 8, §4.3 · p. 8
- On CMU-MOSI, Multilogue-Net consistently outperforms the previous state of the art but beats the current state of the art on only one modality subset.Source: §4.3 · pp. 7–8
- Across both datasets, tri-modal results are best and audio-plus-video results worst; the authors attribute improved results to model capacity, conversation-inspired representations, and more training data.Source: §4.3 · p. 8
5 Ablation Studies and Analysis
The ablations examine the fusion mechanism and emotion GRU (eGRU). Fusion is beneficial in reported cases that include text, while the eGRU analysis finds state and context representations relatively similar across tasks and emotion representations more task-dependent.
5.1 Fusion Mechanism
- Fusion improves the reported cases that include text, but not video plus audio, and the results further support text's role in tri-modal performance.The paper also reports that the added computational cost is negligible relative to the increased performance.Source: Table 4 · p. 9, §5.1 · p. 9
- The added computational cost is considered negligible relative to the increased performance, which the authors largely attribute to non-parametric fusion and few additional prediction-layer parameters.The prediction layer adds 6De parameters for sentiment regression and 36De for emotion recognition.Source: §5.1 · p. 9
5.2 Emotion GRU (eGRU)
- At timestamp 4, emotion-representation distances between the two tasks are much larger than state- and context-representation distances.The comparison uses intermediate text representations from test sample 6, with separate models trained for emotion detection and sentiment regression.Source: Table 5 · p. 9, §5.2 · p. 9
- State and context representations are relatively similar across tasks, whereas emotion representations are more varied and task-dependent.This motivates reusing cGRU and sGRU weights while using a different eGRU for each task; parallel multi-task training is left for future work.Source: §5.2 · p. 9
6 Conclusion
The conclusion presents Multilogue-Net as an RNN for multimodal sentiment analysis and emotion detection in conversation that treats modalities independently while modeling conversational context. It reports strong performance on CMU-MOSI and CMU-MOSEI multimodal settings, while noting that extensions await suitable labeled datasets.
6 Conclusion
- Multilogue-Net is an RNN architecture for multimodal sentiment analysis and emotion detection in conversation.It focuses on conversational context and treats each modality independently according to the information it can hold.Source: §6 · p. 9
- The model consistently performs well on CMU-MOSI and CMU-MOSEI in multimodal settings.Source: §6 · p. 9
- Extensions to better feature extractors, more modalities, and more conversation participants are left to future work because labeled datasets with these extensions are unavailable.Source: §6 · p. 9