Source-linked AI summary
Multimodal Sentiment Analysis using Hierarchical Fusion with Context Modeling
N. Majumder, D. Hazarika, A. Gelbukh, E. Cambria, S. Poria
TL;DR
Multimodal sentiment analysis needs improved fusion of audio, video, and text features because concatenation does not account for modality interactions. The paper introduces hierarchical bimodal-to-trimodal fusion with recurrent context modeling. It outperforms early fusion and the state of the art across the reported multimodal tasks.
Problem
Feature concatenation does not account for differing or conflicting information across modalities, motivating improved multimodal fusion.
Method
The method hierarchically learns bimodal and trimodal correlations with deep neural networks and adds context modeling with GRU-based recurrent networks.
Results
The method outperforms widely used early fusion and the state of the art in multimodal sentiment analysis and emotion detection by significant margins.
Takeaways & Limitations
Hierarchical fusion with context modeling provides more useful multimodal feature vectors for sentiment classification than simple early fusion.
Takeaways & Limitations
The approach is bounded by modality informativeness: visual and audio modalities can be significantly less informative than textual modality.
Abstract
from arXiv · showhide
Multimodal sentiment analysis is a very actively growing field of research. A promising area of opportunity in this field is to improve the multimodal fusion mechanism. We present a novel feature fusion strategy that proceeds in a hierarchical fashion, first fusing the modalities two in two and only then fusing all three modalities. On multimodal sentiment analysis of individual utterances, our strategy outperforms conventional concatenation of features by 1%, which amounts to 5% reduction in error rate. On utterance-level multimodal sentiment analysis of multi-utterance video clips, for which current state-of-the-art techniques incorporate contextual information from other utterances of the same clip, our hierarchical fusion gives up to 2.4% (almost 10% error rate reduction) over currently used concatenation. The implementation of our method is publicly available in the form of open-source code.
1. Introduction
Multimodal sentiment analysis targets opinions expressed through audio, video, and text, but feature fusion remains a central challenge. The paper proposes hierarchical fusion with contextual modeling and reports improvements over concatenation and prior state-of-the-art methods.
- Multimodal sentiment analysis mines opinions from videos containing audio, video, and speech-derived text.
- Recent approaches commonly concatenate modality features, although this ignores potentially conflicting information across modalities.
- The proposed method hierarchically fuses unimodal, bimodal, and trimodal features using fully connected layers and recurrent context modeling.
- 1–2%: contextual recurrent modeling outperforms the state of the art across modality combinations with textual components.
- The method’s feature vectors are empirically more useful for sentiment classification, and its implementation is publicly available as open-source code.
2. Related Work
Prior multimodal research spans feature- and decision-level fusion, while textual integration and learned hierarchical fusion remain less explored. This work learns bimodal and trimodal correlations with deep neural networks instead of relying on concatenation or non-trainable tensor fusion.
- Audio-visual fusion has historically improved emotion-recognition accuracy over unimodal systems, with fusion studied at feature and decision levels.
- Adding text to audio and visual modalities has been comparatively little explored in multimodal emotion detection.
- Existing multimodal methods include feature-level, decision-level, convolutional, multiple-kernel, recurrent, deep-learning, and tensor-based approaches.
- The proposed hierarchical fusion learns bimodal and trimodal correlations using deep neural networks, unlike simple concatenation and non-trainable tensor fusion.
- The method is end-to-end and can be plugged into deep-neural-network-based multimodal sentiment-analysis frameworks.
3. Our Method
The method section introduces the paper’s methodology and presents it step by step. The approach addresses multimodal sentiment classification through a structured fusion pipeline.
- The methodology section first provides an overview of the proposed approach before detailing its individual steps.
- The method is organized around solving the sentiment-classification problem through a novel methodology.
- The section’s detailed procedure follows the initial overview rather than presenting the method as a single fusion operation.
3.1. Overview
The overview extracts modality-specific utterance features, adds recurrent context, and hierarchically combines modalities from bimodal to trimodal representations before softmax classification.
- 3.1. Overview: The pipeline extracts utterance-level features for all three modalities and models surrounding-utterance context with a GRU-based RNN.
- 3.1.2. Multimodal Fusion: Early fusion cannot filter conflicting or redundant cross-modal information, motivating hierarchical fusion from unimodal to bimodal and then trimodal vectors.
- 3.1.2. Multimodal Fusion: Bimodal fusion combines T+V, T+A, and A+V utterance vectors before producing trimodal features from the three bimodal representations.
- 3.1. Overview: The resulting feature vectors are classified with a softmax layer.
3.2. Unimodal Feature Extraction
The paper extracts textual, audio, and visual features separately from each utterance. Text uses CNNs, audio uses openSMILE descriptors, and video uses 3D-CNNs to capture spatial and temporal information.
- Textual Feature Extraction: Textual utterances are represented with 300-dimensional word2vec vectors and padded or truncated to 50 words.
- Textual Feature Extraction: Two convolutional layers with max-pooling transform the word arrays into textual features.The second convolutional layer uses 100 feature maps, and the penultimate fully connected layer supplies the textual feature.
- Audio Feature Extraction: 6,392 features are extracted from each input audio segment.The feature set includes descriptors such as pitch and voice intensity together with statistical functionals.
- Visual Feature Extraction: 3D-CNNs process video frames to model both per-frame visual features and temporal features across frames.Max-pooling selects relevant features before a dense layer and softmax; the dense-layer activations become the video features for each utterance.
- Visual Feature Extraction: The best visual-extraction configuration uses 32 filters, filter dimensions 5, a 3×3×3 pooling window, and a 300-neuron dense layer.
3.3. Context Modeling
Because utterances within a video are semantically dependent, the method models preceding utterances as context using modality-specific GRUs.
- Context Definition: The context of an utterance is defined through preceding utterances whose consideration can determine its complete meaning.
- Context Modeling: RNNs, specifically GRUs, model semantic dependencies among utterances in each video.
- Context Modeling: For each modality, unimodal utterance features are fed into a modality-specific GRU to produce context-aware unimodal features.Shorter videos are padded with dummy utterances represented by null vectors.
- Context Modeling: The resulting context-aware unimodal features are then used to define multimodal features.
3.4. Multimodal Fusion
The fusion strategy compares and combines modalities hierarchically: it first forms bimodal representations, then fuses those representations into a trimodal feature, with GRUs adding utterance context.
- Fusion Setup: Unimodal features are mapped into a common dimension D before fusion, accommodating potentially different original dimensions.
- Fusion Motivation: Abstract feature values may disagree across modalities, so the network compares modality-specific values before combining them.The paper motivates this comparison using modality-specific evidence for abstract concepts such as speaker anger.
- Bimodal Fusion: Bimodal fusion separately combines the audio–video, audio–text, and video–text feature pairs.
- Bimodal Fusion: GRUs incorporate contextual information into each bimodal representation across the utterances of a video.
- Trimodal Fusion: Trimodal fusion combines the three bimodal features using fully connected layers.The resulting trimodal features are subsequently passed through GRUAV T for contextual modeling.
- Trimodal Fusion: The context-aware trimodal feature vector has dimension D3 = 550.
3.5. Classification
The classifier applies a softmax layer to fused unimodal, bimodal, or trimodal features for binary sentiment classification.
- Classification: Fused features from unimodal, bimodal, or trimodal pathways are fed into the classifier for each utterance.
- Classification: The softmax classifier has C = 2 outputs corresponding to the two class values.
- Classification: The classifier computes class probabilities using learned weights and biases, then returns the estimated class value.
3.6. Training
Training uses categorical cross-entropy, Adam optimization, and early stopping over a structured parameter set.
- Training objective: Categorical cross-entropy is used as the training loss function.The loss defines N as the number of samples, i as a sample index, and j as a class value.
- Optimization: Adam optimizes parameters by adapting the learning rate for each parameter individually.
- Training schedule: The network is trained for 200 epochs with early stopping.
- Parameterization: The optimized parameter set covers unimodal, bimodal, and trimodal modality combinations.The combinations include A, V, T, VA, VT, TA, and AVT.
- Procedure: Algorithm 1 summarizes the training method.
4. Experiments
Experiments evaluate hierarchical and context-aware hierarchical fusion on CMU-MOSI and IEMOCAP using speaker-independent splits and comparisons with established fusion baselines. Across modality settings, the method generally improves over early fusion and competing approaches, including gains of 1–2.4% over the state of the art and 1–2% from context modeling.
- Dataset and evaluation setup: CMU-MOSI contains opinion-video utterances annotated on a sentiment scale from −3 to +3.The dataset includes sentimental expressions and videos segmented into utterances.
- Model variants: The context-aware model extracts unimodal features, maps modalities into a common space, fuses each bimodal pair, and then fuses the three bimodal representations.GRUs generate context-aware features before and after the bimodal fusion stages; trimodal fusion produces the final representation for softmax classification.
- Dataset and evaluation setup: Speaker-independent experiments use completely disjoint train and test speakers to emulate unseen conditions.For CMU-MOSI, training and test sets contain 1447 and 752 utterances from 62 and 31 speakers, respectively.
- Dataset and evaluation setup: IEMOCAP provides conversational utterances labeled with eight emotion categories, with experiments focusing on anger, happiness, sadness, and neutral.Only videos from the first eight speakers are used for training in the described setup.
- Baselines: The evaluation compares hierarchical fusion with early fusion, Poria et al.'s method, tensor fusion, and other strong baselines.Early fusion concatenates unimodal features before SVM classification, while Poria et al.'s approach applies MKL to concatenated features.
- Results: The model outperforms Poria et al.'s MKL approach by 1–1.8% across all bimodal and trimodal scenarios.The comparison uses features extracted by Poria et al. and reports the margin across modality combinations.
- Results: 1–2% improvement over the state of the art is reported for CMU-MOSI combinations containing text.Combining all modalities yields about 3.5% higher accuracy than text alone, while A+V is better but similar to the state of the art.
5. Conclusion
The paper presents a hierarchical fusion strategy for multimodal sentiment analysis, addressing the limited prior work on fusion and outperforming early fusion across evaluated datasets and modalities. Context modeling with GRU further improves performance against the state of the art, while future work targets better unimodal features.
- Conclusion: The paper presents a novel and comprehensive hierarchical fusion strategy for multimodal sentiment analysis.The strategy is introduced as the paper’s central contribution to multimodal fusion.
- Conclusion: The method outperforms widely used early fusion on both datasets used to evaluate multimodal sentiment analysis methods.This establishes the reported advantage across the paper’s two evaluation datasets.
- Conclusion: Adding GRU-based context modeling enables the method to outperform the state of the art in multimodal sentiment analysis and emotion detection.The conclusion attributes this stronger result to adding context modeling with GRU.
- Conclusion: Future work will improve unimodal feature quality, especially textual features, to further improve classification accuracy.The authors identify unimodal and particularly textual feature quality as an area for future improvement.