Source-linked AI summary
Multimodal Affective Analysis Using Hierarchical Attention Strategy with Word-Level Alignment
Yue Gu, Kangning Yang, Shiyu Fu, Shuhong Chen, Xinyu Li, Ivan Marsic
TL;DR
Affective analysis must represent informative cues from heterogeneous inputs and account for time-dependent interactions between modalities. This paper uses hierarchical attention and word-level text-audio fusion for utterance-level sentiment and emotion classification. It reports outperforming state-of-the-art approaches and visualizes attention across modalities.
Problem
Affective analysis must extract informative features from heterogeneous inputs and capture time-dependent interactions that abstract-level fusion overlooks.
Source: §1 · pp. 1–2Method
Results
Takeaways & Limitations
The model provides visualizations of modality-specific features and fused attention to support interpretation.
Source: §1 · p. 2, §7 · p. 9The approach requires both audio and text, limiting its scope relative to affect prediction from speech without textual information.
Source: §6 · p. 9
Abstract
from arXiv · showhide
Multimodal affective computing, learning to recognize and interpret human affects and subjective information from multiple data sources, is still challenging because: (i) it is hard to extract informative features to represent human affects from heterogeneous inputs; (ii) current fusion strategies only fuse different modalities at abstract level, ignoring time-dependent interactions between modalities. Addressing such issues, we introduce a hierarchical multimodal architecture with attention and word-level fusion to classify utter-ance-level sentiment and emotion from text and audio data. Our introduced model outperforms the state-of-the-art approaches on published datasets and we demonstrated that our model is able to visualize and interpret the synchronized attention over modalities.
1 Introduction
The paper addresses gaps in affective feature representation and multimodal fusion by aligning text and audio at the word level. It proposes hierarchical attention and word-level fusion, and reports stronger results than state-of-the-art approaches alongside visualizations of attention.
1 Introduction
- A hierarchical multimodal architecture uses bidirectional GRUs and multi-level attention to select informative textual and acoustic features for utterance-level classification.The model targets both sentiment and emotion recognition.Source: §1 · p. 1
- A central challenge is connecting extracted features to affective states, since handcrafted low-level representations lack high-level feature associations.The passage also notes that words and vocal delivery contribute unequally to predictions.Source: §1 · p. 1
- Prior fusion often combines modality predictions or utterance-level features, leaving time-dependent interactions between word-level text and audio insufficiently modeled.Text may be processed word by word while audio is processed at frame or utterance level.Source: §1 · pp. 1–2
- Word-level forced alignment supports three fusion strategies that combine textual and acoustic representations and attention on a common time scale.The strategies are horizontal, vertical, and fine-tuning attention fusion.Source: §1 · p. 2
- The paper reports that its architecture outperforms state-of-the-art approaches on four published sentiment and emotion datasets and enables attention visualization.The visualizations cover attention distributions in single- and multimodal systems.Source: §1 · p. 2
2 Related Work
Earlier affective-analysis work progressed from separately extracted low-level features to deep multimodal feature fusion. The paper distinguishes its approach from prior word-level fusion through fine-tunable hierarchical attention and visualizations of attention within modalities and at fusion.
2 Related Work
- Earlier systems combined lexical and handcrafted acoustic features or used BoW and POS with acoustic features.These approaches extracted low-level features separately for each modality.Source: §2 · p. 2
- Deep approaches used recurrent networks, CNNs, hierarchical LSTMs, and tensor fusion to learn higher-level multimodal representations [Eyben et al., 2010; W¨ollmer et al., 2013; Poria et al., 2015; Poria et al., 2016; Poria et al., 2017b; Gu et al., 2018; Zadeh et al., 2017].Chen et al. (2017) is identified as prior work using word-level fusion.Source: §2 · p. 2
- The proposed approach differs from Chen et al. (2017) by using fine-tunable hierarchical attention, fusing both word-level representations and attention weights, and visualizing attention within modalities and at fusion.The architecture draws on hierarchical attention for document classification and frame-level attention for speech emotion recognition.Source: §2 · p. 2
3 Method
The method aligns text and audio at the word level, extracts attentive representations from each modality, and combines them through three word-level fusion strategies for classification.
3.1 Forced Alignment and Preprocessing
- Forced alignment maps words to audio intervals; text uses 300-dimensional word2vec vectors, while audio uses zero-padded 64-filter-bank MFSC maps.Each word’s acoustic interval is represented with static MFSC coefficients and padded to the dataset’s maximum word-frame length.Source: §3.1 · p. 3
3.2 Text Attention Module
- Bidirectional GRUs encode contextual word states, and textual attention provides word-level weights and states for subsequent fusion.The model uses GRUs because LSTMs gave similar accuracy with around 25% more trainable parameters.Source: §3.2 · pp. 3–4
3.3 Audio Attention Module
- Frame-level attention weights acoustic frames to form word vectors, then word-level attention produces acoustic attention distributions and contextual states.The frame-level weighted sum uses bidirectional contextual states for the frames within each word.Source: §3.3 · p. 4
3.4 Word-level Fusion Module
- Horizontal fusion concatenates separately weighted text and audio representations, whereas vertical fusion averages their attentions over a shared contextual state.Horizontal fusion does not model attention interaction between modalities; vertical fusion forms joint attention from the averaged unimodal attentions.Source: §3.4 · p. 5
- Fine-tuning attention fusion adds trainable attention adjustment to the shared distribution before weighting the shared contextual state into a word representation.The method retains the original unimodal attentions while tuning the shared attention for the final prediction.Source: §3.4 · pp. 5–6
3.5 Decision Making
- A CNN with convolution and max-pooling extracts features from shared word-level vectors, concatenates feature maps, and uses softmax for classification.Convolution filters use multiple widths.Source: §3.5 · p. 6
4 Experiments
Experiments use published sentiment and emotion datasets and compare text-and-audio models, including proposed fusion baselines, under a speaker-independent training and testing setup.
4.1 Datasets
- MOSI and YouTube are used for sentiment, while IEMOCAP and EmotiW are used for emotion recognition.The experiments use binary sentiment labels for MOSI and positive/negative labels for YouTube; IEMOCAP and EmotiW use four emotion categories, with IEMOCAP also evaluated on five categories.Source: §4.1 · p. 6
4.2 Baselines
- Published model comparisons use only audio and text branches because the proposed model targets speech sentiment and emotion analysis..Source: §4.2 · p. 6
4.2.1 Sentiment Analysis Baselines
- Sentiment baselines include bag-of-words and acoustic-feature SVMs, decision-level fusion, CNN and multiple-kernel models, tensor fusion, and word-level LSTM attention.Source: §4.2.1 · pp. 6–7
4.2.2 Emotion Recognition Baselines
- Emotion baselines range from SVM trees and Gaussian Supervector/eVector features to CNN-based multiple-kernel learning and hybrid deep multimodal fusion.Source: §4.2.2 · p. 7
4.2.3 Fusion Baselines
- Fusion baselines either concatenate utterance-level text and audio features before softmax classification or combine separate modality predictions with weighted decision fusion.The decision-level baseline weights text at 1.2 and audio at 0.8.Source: §4.2.3 · p. 7
4.3 Model Training
- Training uses speaker-independent 80/20 train-test splits, with 20% of training data reserved for validation and five-fold cross-validation.Training balances class sample counts per iteration and uses Adam with a 0.001 learning rate.Source: §4.3 · p. 7
5 Result Analysis
The proposed FAF and word-level fusion strategies outperform baselines in sentiment and emotion tests, while modality contributions vary by task. Cross-dataset results and attention visualizations further examine generalization and how text and audio cues combine at the word level.
5.1 Comparison with Baselines
- 76.4% weighted accuracy on MOSI and 72.7% accuracy on emotion recognition; FAF outperformed all five sentiment baselines and all emotion baselines.For five-category emotion recognition, the model improved performance over H-DMS by 4.2% and achieved 0.644 weighted-F1, compared with H-DMS's 0.594 F1.Source: §5.1 · p. 8
- All three proposed fusion strategies outperform UL-Fusion and DL-Fusion on MOSI and IEMOCAP, and FAF outperforms HF and VF on both datasets.Source: §5.1 · p. 8
5.2 Modality and Generalization Analysis
- Text dominates MOSI sentiment prediction, with only a 1.4% accuracy improvement from text-audio fusion; on IEMOCAP, audio-only outperforms text-only.Combining text and audio on IEMOCAP brings a significant performance improvement.Source: §5.2 · p. 8, Table 2 · p. 8
- Trained on MOSI, the model achieves 66.2% accuracy and 0.665 F1 on YouTube; trained on IEMOCAP, it achieves 61.4% accuracy on EmotiW.The authors identify dataset labeling differences, incomplete YouTube utterances, and insufficient speech information in EmotiW as potential influences on generalization.Source: §5.2 · p. 8, Table 3 · p. 8
5.3 Visualize Attentions
- Attention visualizations show shared and fine-tuning attention combining textual and acoustic cues to focus jointly on relevant words.In the happy example, acoustic attention shifts emphasis away from “don’t” and “like” toward “you’re” and “west-sider”; the figure’s color gradation represents word-level source importance.Source: Figure 3 · p. 9, §5.3 · p. 9
6 Discussion
The architecture requires both audio and text, making it more suited to spoken-language analysis than affect prediction from speech alone. Limited multimodal data also constrains evaluation: EmotiW and MOUD were used only for generalization analysis, and MOSI has 2,199 sentence-level samples.
6 Discussion
- The architecture uses both audio and text, making it more related to spoken-language analysis than predicting affect from speech alone.Automatic speech recognition is a potential way to generate text from vocal signals.Source: §6 · p. 9
- MOSI has 2,199 sentence-level samples, while EmotiW and MOUD were used only for generalization analysis because of their small size.The authors call for larger, more general multimodal sentiment and emotion datasets.Source: §6 · p. 9
- Word alignment is readily applied to human speech, but aligning visual information with text is difficult, especially when text only describes video or audio.The authors identify incorporating visual information into an aligning model as a future research topic.Source: §6 · p. 9
7 Conclusion
The paper concludes that its hierarchical-attention architecture aligns text and audio at the word level and uses three fusion strategies for emotion classification. It reports outperforming state-of-the-art methods and providing visualizations for interpreting modality-specific and fusion features.
7 Conclusion
- The model outperforms state-of-the-art methods and provides effective visualization of modality-specific features and fusion-feature interpretation.The conclusion reports these outcomes for the proposed model.Source: §7 · p. 9
- The architecture aligns text and audio at the word level and applies attention to textual word, acoustic frame, and acoustic word vectors.These are the representations receiving attention in the model.Source: §7 · p. 9
- Three fusion strategies use a CNN structure to combine word-level features for emotion classification.The conclusion identifies the fusion strategies and their role in classification.Source: §7 · p. 9