Source-linked AI summary
Multi-task Learning for Multi-modal Emotion Recognition and Sentiment Analysis
Md Shad Akhtar, Dushyant Singh Chauhan, Deepanway Ghosal, Soujanya Poria, Asif Ekbal, Pushpak Bhattacharyya
TL;DR
The paper addresses the challenge of fusing unequal textual, acoustic, and visual information for sentiment and emotion analysis, while leveraging relationships between the tasks and contextual utterances. It proposes a deep multi-task framework with contextual inter-modal attention and reports higher performance than single-task and existing systems on CMU-MOSEI. The analysis also identifies weaker performance for fear and surprise, possibly reflecting dataset under-representation.
Problem
Fusing diverse multimodal information is challenging, and sentiment and emotion are related tasks that can provide mutually useful information.
Method
A deep multi-task framework uses GRU-based contextual inter-modal attention to jointly predict sentiment and emotion from text, acoustic, visual, and contextual utterance information.
Results
The evaluation on CMU-MOSEI suggests that multitask learning improves over single-task learning and attains higher performance than existing systems for sentiment and emotion analysis.
Takeaways & Limitations
Jointly learning sentiment and emotion can exploit their inter-dependence through a shared multimodal representation.
Takeaways & Limitations
MTL performs poorly on fear and surprise, possibly because these instances are under-represented in the dataset.
Abstract
from arXiv · showhide
Related tasks often have inter-dependence on each other and perform better when solved in a joint framework. In this paper, we present a deep multi-task learning framework that jointly performs sentiment and emotion analysis both. The multi-modal inputs (i.e., text, acoustic and visual frames) of a video convey diverse and distinctive information, and usually do not have equal contribution in the decision making. We propose a context-level inter-modal attention framework for simultaneously predicting the sentiment and expressed emotions of an utterance. We evaluate our proposed approach on CMU-MOSEI dataset for multi-modal sentiment and emotion analysis. Evaluation results suggest that multi-task learning framework offers improvement over the single-task framework. The proposed approach reports new state-of-the-art performance for both sentiment analysis and emotion analysis.
1 Introduction
Multimodal videos provide complementary textual, acoustic, and visual information, but fusing these modalities is challenging. The paper motivates jointly analyzing sentiment and emotion while accounting for contextual utterances and unequal modality contributions.
- Multimodal affect analysis: Multimodal affect analysis combines text, acoustic, and visual information that provide distinct clues about spoken content, emphasis, and speaker gestures.Text conveys semantics, acoustic signals reveal emphasis, and visual frames capture gesture and posture.
- Multimodal affect analysis: Effectively fusing diverse modalities remains a major challenge for sentiment and emotion analysis.
- Joint analysis: The proposed multi-task model jointly extracts sentiment and emotion from a speaker’s video.Sentiment is positive or negative; emotions include anger, disgust, fear, happiness, sadness, and surprise.
- Context: The model hypothesizes that contextual utterances can assist classification because neighboring utterances may share inter-dependent sentiment or emotion.
- Unequal modality contributions: Modalities contribute unequally across instances, with acoustic or visual cues sometimes clarifying affect that text does not reveal.The paper uses sarcasm and a superficially neutral “I’m fine” example to motivate modality-sensitive analysis.
2 Problem Definition
The paper frames sentiment and emotion as related tasks whose inter-dependence can support joint prediction. Its framework combines contextual multimodal attention with multi-task learning to emphasize relevant utterances and modalities.
- Motivation: Multi-task learning uses related tasks’ inter-relatedness and shared representations to improve individual task performance while handling multiple problems in one system.
- Motivation: Sentiment and emotion are associated because many emotional states correspond to positive or negative situations.
- Framework: The framework applies attention to contextual utterances and inter-modal information simultaneously to identify contributing features.
- Framework: A recurrent neural network contextual inter-modal attention framework computes correlations across modalities of target and contextual utterances.This selects important contextual modalities before combining them with target-utterance modalities.
- Contributions: The contributions include leveraging sentiment–emotion inter-dependence, weighting contextual utterances and modalities, and reporting state-of-the-art predictions for both tasks.
3 Related Work
Prior work explored multimodal fusion, modality interactions, and contextual information separately. The paper distinguishes its approach by combining contextual inter-modal attention with joint multi-task sentiment and emotion analysis.
- Prior multimodal analysis: Multimodal sentiment prediction is newer than text-based sentiment prediction, although reviews cover the progression from unimodal to multimodal analysis.
- Prior multimodal analysis: Earlier systems modeled facial–word interactions, tensor-based modality dynamics, or convolutional fusion for multimodal sentiment and emotion prediction.
- Research gap: Existing methods variously omitted context, used context without attention, or used attention without contextual information.
- Research gap: The proposed method applies attention to multimodal information from contextual utterances in a single step.This is presented as a way to reveal contributing features across modalities and utterances simultaneously.
- Contributions: The paper also presents comparative single-task versus multi-task analysis and state-of-the-art sentiment and emotion results.
4 Multi-task Multi-modal Emotion Recognition and Sentiment Analysis
The framework combines contextual information from three modalities with pair-wise inter-modal attention, then jointly predicts sentiment and emotion. Its MTMM-ES pipeline uses attention-derived representations for both task outputs.
- Architecture: Three bi-GRU networks capture contextual information separately from text, visual, and acoustic modalities.The resulting modality representations are combined with pair-wise contextual inter-modal attention outputs.
- Contextual Inter-modal Attention: Pair-wise attention computes cross-modality matching matrices over all utterances in a video.The matrices M1 and M2 have dimensions R^u×u, where u is the number of utterances.
- Contextual Inter-modal Attention: Softmax converts cross-modality matrices into contextual attention weights, which produce modality-wise attentive representations before multiplicative gating.The gated representations A1 and A2 are concatenated as the attention output.
- Architecture: CIM-Attention is applied to text–visual, acoustic–visual, and text–acoustic modality pairs.The attention outputs are combined with the original text, visual, and acoustic representations.
- Multi-task Prediction: The combined representation feeds separate sentiment and emotion predictors in the MTMM-ES procedure.The procedure returns polarity and emotion outputs jointly.
5 Datasets, Experiments, and Analysis
Experiments on CMU-MOSEI compare single-task and multi-task learning across multimodal configurations, benchmark metrics, and contextual attention analyses. The multi-task framework improves sentiment and emotion performance, with statistically significant gains and stronger handling of selected examples, while fear and surprise remain difficult.
- Datasets: CMU-MOSEI contains 3,229 videos, over 23,000 utterances, and six emotion values per utterance, with 16,216, 1,835, and 4,625 utterances in training, validation, and test sets.The dataset includes more than 1,000 online YouTube speakers.
- Experiments: The experiments evaluate sentiment with F1-score and accuracy, and emotion with F1-score and weighted accuracy to account for unbalanced emotion classes.Emotion uses seven sigmoid outputs for six emotions plus no-emotion, with binary cross-entropy loss.
- Experiments: The proposed model tests text-visual, text-acoustic, acoustic-visual, and tri-modal inputs, alongside a self-attention variant for separately modeling each modality.The proposed attention mechanism requires at least two modalities.
- Results: 78.86% sentiment F1-score and 80.47% accuracy are obtained by MTL, versus 77.67% and 79.8% for STL with tri-modal inputs.These results report the proposed frameworks' sentiment performance under tri-modal inputs.
- Results: 78.6% versus 77.7% emotion F-score and 62.8 versus 60.8 weighted accuracy favor MTL over STL.The reported comparison is for emotion classification.
- Comparative Analysis: The MTL improvements over STL and state-of-the-art results are statistically significant with p-values < 0.05.The significance comparison includes both the state-of-the-art and proposed single-task results.
- Comparative Analysis: Compared with state-of-the-art systems, MTL reaches 78.6% emotion F1-score and 62.8% weighted accuracy, and 78.8% sentiment F1-score and 80.4% accuracy.The sentiment gains are reported as 1.8% and 3.5% over the state-of-the-art systems.
- STL v/s MTL framework: MTL improves example-level predictions by combining task information, including using positive sentiment to assist prediction of the happy emotion.For one example, MTL improves emotion precision from 1/3 to 2/4 and recall from 1/2 to 2/2.
6 Conclusion
The paper proposes a deep multitask framework that jointly learns multimodal sentiment and emotion analysis through a shared representation and contextual inter-modal attention. On CMU-MOSEI, the multitask framework achieves higher performance than existing systems, while future work extends the framework to intensity prediction.
- 6 Conclusion: The proposed framework jointly addresses multimodal sentiment and emotion analysis by leveraging their inter-dependence.It learns a joint representation for both tasks using a GRU-based inter-modal attention framework.
- 6 Conclusion: The approach was evaluated on the CMU-MOSEI benchmark for multimodal sentiment and emotion analysis.The dataset is described as a recently released benchmark for both tasks.
- 6 Conclusion: Multitask learning enables sentiment and emotion to assist each other during joint learning.The conclusion reports that the framework achieves higher performance than various existing systems across all compared cases.
- 6 Conclusion: Future work will examine sentiment and emotion intensity prediction, including configurations combining classification and intensity tasks.The authors also propose exploring all four tasks together.