Source-linked AI summary
TransModality: An End2End Fusion Method with Transformer for Multimodal Sentiment Analysis
Zilong Wang, Zhaohong Wan, Xiaojun Wan
TL;DR
Multimodal sentiment analysis needs fusion methods that capture relationships among textual, visual, and acoustic features without relying mainly on direct combination. TransModality uses end-to-end Transformer translation, including Forward and Backward Translation, and achieves strong results across CMU-MOSI, MELD, and IEMOCAP. Its scope assumes text is the main modality, with visual and acoustic features serving as auxiliaries.
Problem
Effective fusion remains difficult because multimodal features have distinct properties and prior methods rarely use end-to-end translation to model their correlations.
Method
TransModality translates between modalities with Transformer and uses parallel Forward and Backward Translation to blend their information.
Results
TransModality achieves state-of-the-art performance across CMU-MOSI, MELD, and IEMOCAP, significantly outperforming all baselines on CMU-MOSI and IEMOCAP and most on MELD.
Takeaways & Limitations
The validation results support Transformer-based translation and Forward and Backward Translation as effective components for multimodal fusion.
Takeaways & Limitations
The method assumes text is the main modality and uses visual and acoustic modalities as auxiliary features.
Abstract
from arXiv · showhide
Multimodal sentiment analysis is an important research area that predicts speaker's sentiment tendency through features extracted from textual, visual and acoustic modalities. The central challenge is the fusion method of the multimodal information. A variety of fusion methods have been proposed, but few of them adopt end-to-end translation models to mine the subtle correlation between modalities. Enlightened by recent success of Transformer in the area of machine translation, we propose a new fusion method, TransModality, to address the task of multimodal sentiment analysis. We assume that translation between modalities contributes to a better joint representation of speaker's utterance. With Transformer, the learned features embody the information both from the source modality and the target modality. We validate our model on multiple multimodal datasets: CMU-MOSI, MELD, IEMOCAP. The experiments show that our proposed method achieves the state-of-the-art performance.
1 INTRODUCTION
Multimodal sentiment analysis combines textual, visual, and acoustic information to address ambiguity, but effectively blending modalities remains difficult. TransModality uses Transformer-based translation between modalities and is evaluated on three benchmark datasets.
- Motivation: Visual and acoustic cues can resolve sentiment ambiguity that textual information alone may misclassify.The paper illustrates this with a smiling face and happy tone changing the interpretation of an apparently negative utterance.
- Challenge: Different modalities have unequal predictive abilities, making feature blending a major challenge.Text is treated as the key modality, while visual and acoustic features are used as auxiliary information.
- Proposed method: TransModality fuses multimodal features through an end-to-end Transformer translation model.Features from different modalities serve as source and target representations, allowing encoded features to combine both modalities.
- Evaluation: Experiments evaluate TransModality on CMU-MOSI, MELD, and IEMOCAP.These datasets are described as widely used benchmarks for multimodal sentiment analysis.
- Proposed method: Forward and Backward Translation are adopted to improve multimodal feature fusion.The paper presents translation in both directions as part of the proposed fusion strategy.
2 RELATED WORK
Prior multimodal fusion methods range from direct feature combination and interaction modeling to attention, gating, and sequence-to-sequence translation. TransModality instead uses parallel Transformer-based translation to reduce modality interference and distinguish itself from sequential translation approaches.
- Existing fusion methods: Earlier methods combine unimodal features directly, model multimodal interactions, or use recurrent attention and gated filtering.Examples include concatenation with LSTM, Cartesian-product interaction features, attention mechanisms, and reinforcement-learning gates.
- Indirect fusion: Some prior work uses sequence-to-sequence models for indirect multimodal fusion instead of direct mathematical combination.The paper positions its approach as following this indirect-fusion direction while changing the translation architecture and organization.
- TransModality: TransModality applies Transformer as an end-to-end translation model for multimodal fusion.Its framework is presented as a departure from simpler direct fusion methods and as an extension of prior indirect-fusion work.
- TransModality: Unlike sequential translation from textual to acoustic to visual modality, TransModality translates in parallel through two independent fusion cells.The cells blend textual features separately with acoustic and visual features to reduce interference between modalities.
3 PROPOSED MODEL
TransModality fuses textual, visual, and acoustic features by translating between modality pairs with Transformer-based Modality Fusion Cells. Forward and backward translation produce joint features used for sentiment classification, while translation losses further train the model.
- Model motivation: TransModality treats relationships between modalities as important for sentiment prediction and uses Transformer translation to learn joint utterance representations.Features from one modality are encoded and another modality is decoded from the encoded representation, so the encoded feature contains information from both modalities.
- Modality Fusion Cell: Each Modality Fusion Cell applies bidirectional GRUs and dense layers before translating between modalities with forward and backward Transformer models.Forward translation maps modality α toward β, while backward translation maps β toward α; the resulting features are generated for each utterance.
- Model architecture: The model uses parallel Modality Fusion Cells for text-visual and text-acoustic fusion, reducing interference between these auxiliary modalities.Each cell blends textual features with one additional modality, producing encoded and decoded features for later classification and translation evaluation.
- Training objective: Translation outputs are projected to target-feature dimensions and trained with mean absolute error alongside the classification loss.The joint loss combines translation losses with classification loss, and the whole-model loss averages the joint loss across utterances.
- Classification: The final joint feature concatenates encoded features from all modality-pair cells and is passed through a dense layer to predict sentiment labels.The model uses textual features as the main modality and visual and acoustic features as auxiliary modalities.
- Two-modality variant: Bi-TransModality adapts the framework to datasets with two modalities by using one Modality Fusion Cell and a weighted joint loss.It takes only two modality features as input and calculates the final joint features from that single cell.
4 EXPERIMENT SETTING
The experiments evaluate TransModality on three established multimodal datasets covering sentiment and emotion prediction, using utterance-level annotations and modality-specific features. Comparisons include recurrent, hierarchical, attention-based, gated, concatenation, and sequence-to-sequence fusion baselines.
- Datasets: TransModality is evaluated on CMU-MOSI, MELD, and IEMOCAP, where videos are segmented into utterances with sentiment or emotion labels.CMU-MOSI contains positive or negative utterance labels; MELD and IEMOCAP provide broader sentiment or emotion categories.
- CMU-MOSI: CMU-MOSI contains opinion videos from online sharing websites and uses utterance-level features for comparison with MMMU-BA.Each utterance is annotated as either positive or negative.
- MELD: MELD provides textual and acoustic features from multiparty conversations, with separate sentiment and seven-category emotion annotations.Sentiment labels are positive, neutral, or negative; emotion labels include anger, disgust, fear, joy, neutral, sadness, and surprise.
- IEMOCAP: IEMOCAP contains conversation videos segmented into utterances annotated with six emotion labels.The labels are happy, sad, neutral, angry, excited, and frustrated.
- Baselines: The comparison set spans concatenation, hierarchical fusion, complex attention, gated embeddings, temporal attention, and sequence-to-sequence fusion methods.Examples include bc-LSTM, CHFusion, MMMU-BA, GME-LSTM(A), Seq2Seq2Sent, MCTN, and MELD-base.
5 RESULTS AND DISCUSSION
TransModality generally outperforms strong baselines across multimodal datasets and modality settings. Additional analyses attribute gains to Transformer-based translation, parallel fusion, and Backward Translation.
- Comparison with Baselines: TransModality surpasses existing state-of-the-art methods on three datasets in most bi-modality and tri-modality settings.The comparisons cover CMU-MOSI, MELD, and IEMOCAP.
- Comparison with Baselines: Adding more modalities improves prediction, with three-modality TransModality outperforming its two-modality version.The authors report a large accuracy improvement when all three modalities are considered.
- Comparison with Baselines: Sign tests show significant gains over all baselines on CMU-MOSI and IEMOCAP, and over most baselines on MELD.Asterisks indicate sign-test p-values below 0.05.
- Analysis of Translation: The method combines end-to-end Transformer translation, parallel translation, and Forward and Backward Translation for multimodal fusion.The encoded features serve as joint features, while parallel fusion is intended to reduce modality interference.
- Analysis of Translation: Transformer-based translation improves performance over the seq2seq alternative across all three datasets.Trans-MCTN improves over MCTN, while TransModality further improves over Trans-MCTN.
- Analysis of Translation: Backward Translation adds approximately 1.5 accuracy points across datasets and settings.Comparisons with and without Backward Translation show improvements for both TransModality and Bi-TransModality.
6 CONCLUSION
TransModality is an end-to-end Transformer-based fusion method that translates between modalities and combines their information in encoded features. Validation on CMU-MOSI, MELD, and IEMOCAP supports its reliability, while future work will test other multimodal classification tasks.
- TransModality uses Transformer to translate between modalities and blend multimodal information into encoded features.
- Forward and Backward Translations model correlations between multimodal features and improve translation performance.
- Validation on CMU-MOSI, MELD, and IEMOCAP demonstrates the reliability of Transformer-based multimodality fusion and the effectiveness of Forward and Backward Translations.
- Future work will apply TransModality to multimodal sarcasm detection and stance detection to further test model robustness.