PaperlayerView exact revision on arXiv

Source-linked AI summary

Video Sentiment Analysis with Bimodal Information-augmented Multi-Head Attention

Ting Wu, Junjie Peng, Wenqiang Zhang, Huiran Zhang, Chuanshuai Ma, Yansong Huang

arXiv:2103.02362v3cs.AI

TL;DR

Video sentiment analysis must fuse textual, visual, and acoustic signals because sentiment varies across contexts and pairwise modality interactions contribute differently. The paper proposes bimodal information-augmented multi-head attention and reports better performance than existing models on four datasets, with visualized explanations of bimodal contributions.

  • Problem

    Video sentiment analysis must integrate textual, visual, and acoustic signals, while existing attention approaches often focus on single-modality features and omit differences in inter-modal interactions.

    Source: §1 · p. 1

  • Method

    The paper proposes BMHA, using multi-head attention to model intra-modal, inter-modal, and inter-bimodal interactions for video sentiment analysis.

    Source: §1 · p. 2

  • Results

    The proposed model outperforms existing methods on CH-SIMS, CMU-MOSI, MOSEI, and IEMOCAP.

    Source: §1 · p. 2, §6 · p. 10

  • Takeaways & Limitations

    Visualized attention indicates that pairwise modality contributions vary across samples, and the authors report that bimodal information can enhance BMHA performance to some extent.

    Source: §1 · p. 2, §5.4 · p. 10

    The authors plan further fusion-method comparisons on CH-SIMS to provide more benchmark results for Chinese multimodal sentiment analysis.

    Source: §6 · p. 10

Abstract

from arXiv · show

Humans express feelings or emotions via different channels. Take language as an example, it entails different sentiments under different visual-acoustic contexts. To precisely understand human intentions as well as reduce the misunderstandings caused by ambiguity and sarcasm, we should consider multimodal signals including textual, visual and acoustic signals. The crucial challenge is to fuse different modalities of features for sentiment analysis. To effectively fuse the information carried by different modalities and better predict the sentiments, we design a novel multi-head attention based fusion network, which is inspired by the observations that the interactions between any two pair-wise modalities are different and they do not equally contribute to the final sentiment prediction. By assigning the acoustic-visual, acoustic-textual and visual-textual features with reasonable attention and exploiting a residual structure, we attend to attain the significant features. We conduct extensive experiments on four public multimodal datasets including one in Chinese and three in English. The results show that our approach outperforms the existing methods and can explain the contributions of bimodal interaction in multiple modalities.

1. Introduction

The paper proposes bimodal information-augmented multi-head attention for video sentiment analysis, where pairwise modality interactions can contribute differently. It motivates this design by the challenge of understanding human intentions and the limits of approaches that focus on single-modality features.

1. Introduction

  • The authors identify a gap in attention methods that extract informative single-modality features without considering differences in information generated by inter-modal interactions.They note that pairwise fusion features can contribute differently to sentiment decisions.Source: §1 · pp. 1–2
  • The paper studies the contribution of pairwise modalities and proposes attention over acoustic-visual, acoustic-textual, and visual-textual interactions.It describes extracting independent and consistent information and fusing weighted bimodal features with original inter-modal features.Source: §1 · p. 2
  • The proposed BMHA uses three multi-head attention units to model intra-modal, inter-modal, and inter-bimodal interactions, with multimodal features as source and bimodal features as target.The paper describes the model as following hierarchical fusion.Source: §1 · p. 2
  • The authors report that bimodal information can enhance BMHA performance to some extent and provide visualized explanations of pairwise interaction differences.They also describe bimodal attention as efficient for sentiment prediction.Source: §1 · p. 2
  • The evaluation covers four public datasets, including one Chinese dataset and three English datasets.The introduction frames intention understanding as important for reasonable responses and behavior in service settings.Source: §1 · pp. 1–2

2. Related Work

The related work surveys multimodal fusion strategies, deep architectures, sequence-translation approaches, and attention methods. It presents these as ways to model unimodal and cross-modal interactions in video data.

2. Related Work

  • Fusion strategies include early fusion after feature extraction, late fusion of modality-specific decisions, and hybrid combinations of early-fusion outputs with unimodal predictions.Late fusion may combine decisions by averaging, weighted sum, majority voting, or neural networks.Source: §2 · p. 2
  • Deep-learning approaches include tensor and low-rank fusion, temporal CNN, RNN, and LSTM models, as well as staged and hierarchical fusion architectures.The review also describes memory-based models whose state evolves with unimodal LSTM hidden states.Source: §2 · pp. 2–3
  • Sequence-to-sequence work translates between source and target modalities, while other frameworks separate or dynamically model unimodal, bimodal, and trimodal interactions.The cited examples include parallel translation and joint and modality-specific subspaces.Source: §2 · p. 3
  • Attention approaches use bimodal attention matrices, multi-head self-attention, hierarchical attention, or mutual attention to extract features and model modality relationships.The review describes attention across representation subspaces and positions.Source: §2 · p. 3

3. Method

BIMHA processes text, audio, and visual inputs through unimodal representation learning, pairwise interaction modeling, inter-bimodal attention, and sentiment prediction. Its design uses outer-product pairwise features and attention to weight their contributions.

3. Method

  • BIMHA routes text, audio, and visual inputs through unimodal representation learning, pairwise interaction, inter-bimodal attention, and a sentiment prediction network.The architecture comprises URL, IMI, IBI, and prediction components.Source: §3 · p. 3

3.1. Unimodal Representation Learning

  • BERT sentence embeddings are padded or truncated and encoded with an LSTM; acoustic and visual features are extracted and averaged across each utterance’s frames.Acoustic features include ZCR, MFCCs, and CQT; aligned faces provide visual features.Source: §3.1 · pp. 3–4

3.2. Inter-Modal Interaction Information

  • Pairwise AV, AT, and VT features come from outer products, then private projections unify dimensions and a shared layer extracts deep features while reducing storage.The shared layer uses common parameters for the three pairwise features.Source: §3.2 · p. 4

3.3. Inter-Bimodal Interaction Information

  • BMHA uses each AV, AT, or VT feature as a query against multimodal D as key and value, assigning attention to pairwise features according to their significance.The model uses shared query-projection parameters across bimodal features to reduce parameter count and memory consumption.Source: §3.3 · p. 5

3.4. Prediction

  • The attended inter-bimodal features are concatenated and added residually to the original features before a three-layer DNN generates the sentiment output.The residual addition is used to avoid the vanishing gradient problem.Source: §3.4 · p. 5

4. Experimental Settings

The experiments evaluate the model on four public datasets, compare it with a range of multimodal baselines, and specify the training and evaluation settings.

4.1. Datasets

  • Four datasets span Chinese and English video sentiment tasks, with different segment counts, sentiment annotations, and evaluation scopes.CH-SIMS is Chinese; CMU-MOSI, MOSEI, and IEMOCAP are English. IEMOCAP evaluation uses only happy, sad, angry, and neutral.Source: §4.1 · p. 6

4.2. Baselines

  • The baselines include recurrent, tensor-fusion, memory, cross-modal attention, and modality-representation approaches.The list includes EF-LSTM, LF-DNN, TFN, LMF, MFN, DFG, MulT, MISA, and multi-task or self-supervised variants.Source: §4.2 · p. 6

4.3. Setup

  • Models are run five times per dataset, with average test performance reported and hyperparameters tuned across modality-specific and fusion components.Training uses Adam and early stopping at 20 epochs; metrics vary by dataset, including classification and regression measures for CH-SIMS and MOSI/MOSEI.Source: §4.3 · pp. 6–7
  • For MOSI and MOSEI, Table 2 lists separate hyperparameter values for aligned and unaligned datasets.In each left/right pair, the left value is for the aligned dataset and the right value for the unaligned dataset.Source: Table 2 · p. 7

5. Results and Analysis

On CH-SIMS, BIMHA outperforms other models on most metrics, while ablations favor combining modalities and attending to pairwise interactions. Results on English datasets vary by alignment and dataset, and visualizations show that pairwise attention weights differ across samples.

5.1. Comparative Analysis

  • BIMHA outperforms other CH-SIMS models on most metrics, especially Acc-5, with 2.5 M parameters—fewer than TFN, DFG, MISA, MTFN, and Self-MM.The authors note that model rankings differ from prior studies on CH-SIMS and may depend on dataset type.Source: Table 3 · p. 7, §5.1 · p. 7

5.2. Ablation Study

  • Text is the strongest single modality and audio performs poorly; all three modalities perform best, while visual-text combinations outperform other two-modality combinations.The authors attribute limitations in audio-containing combinations to word-level misalignment and background-music noise.Source: §5.2 · p. 8, Table 4 · p. 8
  • AT+VT achieves the best result among pairwise combinations, but three-way simple concatenation is weaker; BMHA with three MHA performs better than its two-MHA version.The experiments use outer-product bimodal features; the authors report that the pairwise interactions contribute differently.Source: §5.2 · p. 8, Table 5 · p. 9
  • The full model with IBI and IMI performs best, and IBI is more important than IMI; combining bimodal and unimodal targets is not satisfying overall.VT combined with A performs better than the other tested hybrid combinations.Source: §5.2 · p. 8, Table 6 · p. 9

5.3. The Generalization of Model

  • On MOSI and MOSEI, BIMHA improves on most aligned-model metrics except against CIA, but is less competitive than MulT, MISA, and Self-MM on some unaligned metrics.BIMHA performs better on unaligned than aligned data in the authors’ comparison; they also report that the gap is more pronounced on the smaller MOSI dataset.Source: §5.3 · p. 9, Table 7 · p. 10

5.4. Data Visualization

  • Unimodal text features appear compact while audio and video are scattered; with bimodal attention, the feature distributions appear more consistent.The authors suggest that network approximation and feature-balancing weights may explain the pattern.Source: Figure 3 · p. 9, §5.4 · pp. 9–10

5.3. The Generalization of Model

  • On IEMOCAP, BIMHA achieves the best F1 for all emotions, while its Happy and Angry Acc-2 results are slightly below ICCN.The authors report that all models perform best on Happy and worst on Neutral, possibly reflecting the dataset’s unbalanced distribution.Source: §5.3 · p. 9, Table 8 · p. 10

5.4. Data Visualization

  • Across 30 test samples, AV, AT, and VT attention varies by sample, although VT has greater influence overall; examples show AV or VT receiving higher weight.The visualization presents attention scores for the three modality pairs and two prediction examples.Source: §5.4 · p. 10

6. Conclusion

The paper proposes BIMHA to derive independent and consistent information from bimodal features for fusion, and reports evaluation on four datasets with performance better than existing models. It identifies further CH-SIMS fusion comparisons as future work.

6. Conclusion

  • The proposed method uses BMHA to obtain independent and consistent information from bimodal features for subsequent fusion.The paper evaluates the approach on CH-SIMS, CMU-MOSI, MOSEI, and IEMOCAP.Source: §6 · p. 10
  • The authors report that their model is effective and performs better than existing models across the evaluated datasets.The conclusion names CH-SIMS, CMU-MOSI, MOSEI, and IEMOCAP as the evaluation datasets.Source: §6 · p. 10
  • The authors plan to test more relevant fusion methods on CH-SIMS to provide additional Chinese multimodal sentiment-analysis benchmarks.They also plan to explore adding gender, emotional duration, and other dimensional characteristics.Source: §6 · p. 10

CRediT authorship contribution statement

The statement assigns conceptualization to multiple authors, with methodology and software assigned to Ting Wu and supervision to Junjie Peng.

CRediT authorship contribution statement

Loading 2103.02362v3…