PaperlayerView exact revision on arXiv

Source-linked AI summary

A Transformer-based joint-encoding for Emotion Recognition and Sentiment Analysis

Jean-Benoit Delbrouck, Noé Tits, Mathilde Brousmiche, Stéphane Dupont

arXiv:2006.15955v1cs.CLcs.HCcs.LG

TL;DR

Multimodal sentiment and emotion prediction requires integrating information across affective signals. The paper proposes a Transformer-based joint-encoding model with modular co-attention and glimpse layers, achieving results that compare with or sometimes surpass the state of the art, especially for sentiment on CMU-MOSEI. Its visual-input assumption and remaining F1-score weaknesses define important scope boundaries.

  • Problem

    Predicting affective states from multimedia is challenging because emotion recognition and sentiment analysis involve multiple signal types.

    Source: §1 · p. 1, §4.2 · p. 4

  • Method

    The model uses dedicated modality Transformers enhanced with joint encoding, modular co-attention, and glimpse layers to combine modalities.

    Source: §3.2 · pp. 2–3, §3.1 · p. 2

  • Results

    The model sometimes surpasses current state-of-the-art results, with the strongest reported scores on CMU-MOSEI for sentiment and the L+A combination performing best among tested modalities.

    Source: §1 · p. 1, §6.2 · p. 5, §7 · p. 6

  • Takeaways & Limitations

    Transformer-based joint encoding is a computationally efficient and viable approach for sentiment analysis and emotion recognition on CMU-MOSEI.

    Source: §1 · p. 1, §7 · p. 6

    The visual input was not cropped to the face, and adding visual input did not improve results; emotion F1-scores, especially for less frequent classes, still have room for improvement.

    Source: §4.3 · p. 4, §6.2 · p. 5, §7 · p. 6

Abstract

from arXiv · show

Understanding expressed sentiment and emotions are two crucial factors in human multimodal language. This paper describes a Transformer-based joint-encoding (TBJE) for the task of Emotion Recognition and Sentiment Analysis. In addition to use the Transformer architecture, our approach relies on a modular co-attention and a glimpse layer to jointly encode one or more modalities. The proposed solution has also been submitted to the ACL20: Second Grand-Challenge on Multimodal Language to be evaluated on the CMU-MOSEI dataset. The code to replicate the presented experiments is open-source: https://github.com/jbdel/MOSEI_UMONS.

1 Introduction

The paper addresses multimodal affect prediction by combining Transformer and modular co-attention ideas into a computationally efficient solution for sentiment and emotion tasks. It evaluates this approach on CMU-MOSEI alongside the paper’s experimental setup and results.

  • The proposed solution combines Transformer and modular co-attention ideas for multimodal sentiment analysis and emotion recognition.Source: §1 · p. 1
  • The paper evaluates the approach on the CMU-MOSEI dataset and reports comparisons with existing results for both tasks.Source: §1 · p. 1
  • The paper proceeds from related work to model description, modality-feature extraction, dataset presentation, and experimental results.Source: §1 · p. 1

2 Related work

Prior CMU-MOSEI work includes memory-, graph-, tensor-, and context-aware RNN approaches, while the authors identify Transformer-based solutions as absent from this set.

  • Prior CMU-MOSEI models include memory-based, graph-based, tensor-fusion, and context-aware RNN approaches.The surveyed methods address multimodal sequence synchronization, cross-modal dynamics, modality fusion, and conversational emotion detection or sentiment analysis.Source: §2 · pp. 1–2
  • The authors state that, to their knowledge, the reviewed CMU-MOSEI solutions did not use a Transformer-based architecture.Source: §2 · p. 1
  • MFN synchronizes multimodal sequences with a multi-view gated memory storing intraview and cross-view interactions over time.Source: §2 · p. 1
  • Graph-MFN adds a Dynamic Fusion Graph that learns n-modal interactions and can alter its fusion structure during inference.Source: §2 · p. 1
  • Tensor Fusion Network methods use modality outer products, with frame-by-frame fusion preferred because whole-sequence fusion expands the feature space exponentially.Source: §2 · p. 1
  • Multilogue-Net applies a context-aware RNN to multimodal emotion detection and sentiment analysis in conversation.Source: §2 · p. 2

3 Model

The model uses Transformer encoders for individual modalities and extends them with joint multimodal encoding, modular co-attention, and glimpse layers.

  • The model evaluates monomodal encoders for linguistic, visual, or acoustic inputs and multimodal encoders for modality combinations.Source: §3 · p. 2

3.1 Monomodal Transformer Encoding

  • Each monomodal encoder stacks identical Transformer blocks containing residual connections, layer normalization, multi-head self-attention, and an MLP.The block output follows LayerNorm(x + Sublayer(x)).Source: §3.1 · p. 2
  • Self-attention computes affinities between input rows, while multi-head attention learns attention weights across different feature subspaces and positions.In self-attention, keys, queries, and contexts are the same input.Source: §3.1 · p. 2

3.2 Multimodal Transformer Encoding

  • The multimodal design adds a dedicated Transformer per modality, joint encoding, modular co-attention, and a glimpse layer at each block.Joint encoding processes modalities simultaneously rather than completing one modality before another.Source: §3.2 · pp. 2–3
  • Modular co-attention lets a primary modality x modulate modality y by using x as the key and context while retaining y through a residual connection.The feature sizes of x and y must match for the attention operation and residual addition, with transformations in multi-head attention providing adjustment.Source: §3.2 · pp. 2–3
  • The glimpse layer projects each modality into a new representation by stacking soft-attention outputs, then combines final modality vectors by element-wise summation.The architecture can be duplicated for any number of modalities, and linguistic input always modulates the others in this contribution.Source: §3.2 · p. 3, §3.3 · p. 3

4 Feature extractions

The paper independently pre-computes linguistic, acoustic, and visual features as inputs to the Transformer blocks, using modality-specific representations from utterances and videos.

  • Each modality is pre-computed independently for every dataset example before entering the Transformer blocks.Source: §4 · p. 3

Linguistic features

  • Linguistic features use lowercase tokenization, punctuation removal, a 14,176-word training vocabulary, and 300-dimensional GloVe embeddings.Source: §4.1 · pp. 3–4

Acoustic features

  • Acoustic features use mel-spectrograms because they retain speech, nonverbal, and prosodic information while balancing dimensionality and representation capacity.Source: §4.2 · p. 4
  • Mel-spectrograms are extracted with 80 filter banks and temporally reduced by selecting one frame every 16 frames.Source: §4.2 · p. 4

Visual features

  • Visual features come from a pretrained R(2+1)D-152 model using 32-frame RGB clips, sliding with an 8-frame stride, without face cropping.Source: §4.3 · p. 4

5 Dataset

The experiments use CMU-MOSEI, a multimodal dataset containing sentiment scores and multi-label binary emotion annotations across thousands of sentences and speakers.

  • CMU-MOSEI contains 23,453 annotated sentences from 1,000 speakers, with sentiment scored from -3 to +3 and six annotated emotion classes.Source: §5 · p. 4
  • Table 1 reports test-set results for modality combinations, while emotion accuracies are not compared because prior work uses weighted accuracy and this paper uses standard accuracy.Source: Table 1 · p. 5
  • Emotion labels are binary per class but allow multiple emotions in one sentence, making emotion recognition a multi-label problem.Source: §5 · p. 5
  • The dataset distribution is skewed toward frequent emotions, with happiness exceeding 12,000 positive samples and fear having almost 1,900.Source: §5 · p. 5, Figure 3 · p. 5

6 Experiments

The experiments evaluate modality combinations and model variants on CMU-MOSEI, finding linguistic-plus-acoustic input strongest while visual integration remains difficult.

Experimental settings

  • Training uses Adam with learning rate 1e-4, batch size 32, validation-based decay, early stopping, and averaged predictions from 5 models.Source: §6.1 · p. 5
  • The model generally uses 6 Transformer blocks of hidden size 512, 4 attention heads, a 1024-unit MLP layer, and dropout rates of 0.1 and 0.5.Source: §6.1 · p. 5
  • Acoustic and visual features are truncated above 40 spatial dimensions, with 40 also used as the number of glimpses.Source: §6.1 · p. 5, Figure 4 · p. 5

Results

  • L+A is the best model, and adding visual input does not improve results, indicating that visual integration remains the most difficult modality challenge.Source: §6.2 · p. 5
  • The L+A model improves most clearly on 7-class sentiment and surpasses the previous state of the art for that task.Source: §6.2 · p. 6
  • Multilogue-Net predicts some emotion classes better than this model, possibly because it uses conversational context that the presented model does not.Source: §6.2 · p. 6

7 Discussions

The authors report a computationally efficient, robust model with strong accuracy on CMU-MOSEI, while recognizing weaker F1-scores for less prevalent emotion classes.

  • Strong accuracy on CMU-MOSEI led the authors to describe TBJE as robust, with the strongest sentiment-task scores reported for the dataset.Source: §7 · p. 6
  • Weaker F1-scores remain, particularly for emotion classes that are less prevalent in the dataset.Source: §7 · p. 6
  • Alternative inputs included OpenFace 2.0 facial-behavior features and DenseNet frame-level visual features.Source: §7 · p. 6
  • The authors also varied mel filter-bank counts, temporal reduction, spectrogram usage, and whether to use GloVe embeddings.Source: §7 · p. 6

Related papers

Loading 2006.15955v1…