Source-linked AI summary
A Transformer-based joint-encoding for Emotion Recognition and Sentiment Analysis
Jean-Benoit Delbrouck, Noé Tits, Mathilde Brousmiche, Stéphane Dupont
TL;DR
Multimodal sentiment and emotion prediction requires integrating information across affective signals. The paper proposes a Transformer-based joint-encoding model with modular co-attention and glimpse layers, achieving results that compare with or sometimes surpass the state of the art, especially for sentiment on CMU-MOSEI. Its visual-input assumption and remaining F1-score weaknesses define important scope boundaries.
Problem
Predicting affective states from multimedia is challenging because emotion recognition and sentiment analysis involve multiple signal types.
Method
The model uses dedicated modality Transformers enhanced with joint encoding, modular co-attention, and glimpse layers to combine modalities.
Results
The model sometimes surpasses current state-of-the-art results, with the strongest reported scores on CMU-MOSEI for sentiment and the L+A combination performing best among tested modalities.
Takeaways & Limitations
Transformer-based joint encoding is a computationally efficient and viable approach for sentiment analysis and emotion recognition on CMU-MOSEI.
Takeaways & Limitations
The visual input was not cropped to the face, and adding visual input did not improve results; emotion F1-scores, especially for less frequent classes, still have room for improvement.
Abstract
from arXiv · showhide
Understanding expressed sentiment and emotions are two crucial factors in human multimodal language. This paper describes a Transformer-based joint-encoding (TBJE) for the task of Emotion Recognition and Sentiment Analysis. In addition to use the Transformer architecture, our approach relies on a modular co-attention and a glimpse layer to jointly encode one or more modalities. The proposed solution has also been submitted to the ACL20: Second Grand-Challenge on Multimodal Language to be evaluated on the CMU-MOSEI dataset. The code to replicate the presented experiments is open-source: https://github.com/jbdel/MOSEI_UMONS.
1 Introduction
The paper addresses multimodal affect prediction by combining Transformer and modular co-attention ideas into a computationally efficient solution for sentiment and emotion tasks. It evaluates this approach on CMU-MOSEI alongside the paper’s experimental setup and results.
- The proposed solution combines Transformer and modular co-attention ideas for multimodal sentiment analysis and emotion recognition.
- The paper evaluates the approach on the CMU-MOSEI dataset and reports comparisons with existing results for both tasks.
- The paper proceeds from related work to model description, modality-feature extraction, dataset presentation, and experimental results.
2 Related work
Prior CMU-MOSEI work includes memory-, graph-, tensor-, and context-aware RNN approaches, while the authors identify Transformer-based solutions as absent from this set.
- Prior CMU-MOSEI models include memory-based, graph-based, tensor-fusion, and context-aware RNN approaches.The surveyed methods address multimodal sequence synchronization, cross-modal dynamics, modality fusion, and conversational emotion detection or sentiment analysis.
- The authors state that, to their knowledge, the reviewed CMU-MOSEI solutions did not use a Transformer-based architecture.
- MFN synchronizes multimodal sequences with a multi-view gated memory storing intraview and cross-view interactions over time.
- Graph-MFN adds a Dynamic Fusion Graph that learns n-modal interactions and can alter its fusion structure during inference.
- Tensor Fusion Network methods use modality outer products, with frame-by-frame fusion preferred because whole-sequence fusion expands the feature space exponentially.
- Multilogue-Net applies a context-aware RNN to multimodal emotion detection and sentiment analysis in conversation.
3 Model
The model uses Transformer encoders for individual modalities and extends them with joint multimodal encoding, modular co-attention, and glimpse layers.
- The model evaluates monomodal encoders for linguistic, visual, or acoustic inputs and multimodal encoders for modality combinations.
- 3.1 Monomodal Transformer Encoding: Each monomodal encoder stacks identical Transformer blocks containing residual connections, layer normalization, multi-head self-attention, and an MLP.The block output follows LayerNorm(x + Sublayer(x)).
- 3.1 Monomodal Transformer Encoding: Self-attention computes affinities between input rows, while multi-head attention learns attention weights across different feature subspaces and positions.In self-attention, keys, queries, and contexts are the same input.
- 3.2 Multimodal Transformer Encoding: The multimodal design adds a dedicated Transformer per modality, joint encoding, modular co-attention, and a glimpse layer at each block.Joint encoding processes modalities simultaneously rather than completing one modality before another.
- 3.2 Multimodal Transformer Encoding: Modular co-attention lets a primary modality x modulate modality y by using x as the key and context while retaining y through a residual connection.The feature sizes of x and y must match for the attention operation and residual addition, with transformations in multi-head attention providing adjustment.
- 3.2 Multimodal Transformer Encoding: The glimpse layer projects each modality into a new representation by stacking soft-attention outputs, then combines final modality vectors by element-wise summation.The architecture can be duplicated for any number of modalities, and linguistic input always modulates the others in this contribution.
4 Feature extractions
The paper independently pre-computes linguistic, acoustic, and visual features as inputs to the Transformer blocks, using modality-specific representations from utterances and videos.
- Each modality is pre-computed independently for every dataset example before entering the Transformer blocks.
- Linguistic features: Linguistic features use lowercase tokenization, punctuation removal, a 14,176-word training vocabulary, and 300-dimensional GloVe embeddings.
- Acoustic features: Acoustic features use mel-spectrograms because they retain speech, nonverbal, and prosodic information while balancing dimensionality and representation capacity.
- Acoustic features: Mel-spectrograms are extracted with 80 filter banks and temporally reduced by selecting one frame every 16 frames.
- Visual features: Visual features come from a pretrained R(2+1)D-152 model using 32-frame RGB clips, sliding with an 8-frame stride, without face cropping.
5 Dataset
The experiments use CMU-MOSEI, a multimodal dataset containing sentiment scores and multi-label binary emotion annotations across thousands of sentences and speakers.
- CMU-MOSEI contains 23,453 annotated sentences from 1,000 speakers, with sentiment scored from -3 to +3 and six annotated emotion classes.
- Table 1 reports test-set results for modality combinations, while emotion accuracies are not compared because prior work uses weighted accuracy and this paper uses standard accuracy.
- Emotion labels are binary per class but allow multiple emotions in one sentence, making emotion recognition a multi-label problem.
- The dataset distribution is skewed toward frequent emotions, with happiness exceeding 12,000 positive samples and fear having almost 1,900.
6 Experiments
The experiments evaluate modality combinations and model variants on CMU-MOSEI, finding linguistic-plus-acoustic input strongest while visual integration remains difficult.
- Experimental settings: Training uses Adam with learning rate 1e-4, batch size 32, validation-based decay, early stopping, and averaged predictions from 5 models.
- Experimental settings: The model generally uses 6 Transformer blocks of hidden size 512, 4 attention heads, a 1024-unit MLP layer, and dropout rates of 0.1 and 0.5.
- Experimental settings: Acoustic and visual features are truncated above 40 spatial dimensions, with 40 also used as the number of glimpses.
- Results: L+A is the best model, and adding visual input does not improve results, indicating that visual integration remains the most difficult modality challenge.
- Results: The L+A model improves most clearly on 7-class sentiment and surpasses the previous state of the art for that task.
- Results: Multilogue-Net predicts some emotion classes better than this model, possibly because it uses conversational context that the presented model does not.
7 Discussions
The authors report a computationally efficient, robust model with strong accuracy on CMU-MOSEI, while recognizing weaker F1-scores for less prevalent emotion classes.
- Strong accuracy on CMU-MOSEI led the authors to describe TBJE as robust, with the strongest sentiment-task scores reported for the dataset.
- Weaker F1-scores remain, particularly for emotion classes that are less prevalent in the dataset.
- Alternative inputs included OpenFace 2.0 facial-behavior features and DenseNet frame-level visual features.
- The authors also varied mel filter-bank counts, temporal reduction, spectrogram usage, and whether to use GloVe embeddings.