Source-linked AI summary
Multimodal Language Analysis with Recurrent Multistage Fusion
Paul Pu Liang, Ziyin Liu, Amir Zadeh, Louis-Philippe Morency
TL;DR
Multimodal language analysis requires modeling interactions within and across language, visual, and acoustic modalities. RMFN decomposes fusion into recursive stages that specialize in subsets of signals while recurrent networks model temporal and intra-modal interactions. It achieves state-of-the-art performance across three tasks and public datasets, with visualizations showing coordinated modeling of synchronous and asynchronous interactions.
Problem
Multimodal language analysis needs to model both intra-modal structure and cross-modal interactions across language, visual, and acoustic signals.
Method
RMFN uses recursive multistage fusion, highlighting and fusing signal subsets with prior representations, and integrates recurrent neural networks for temporal and intra-modal interactions.
Results
RMFN achieves state-of-the-art performance across sentiment analysis, emotion recognition, and speaker-trait recognition on three public multimodal datasets.
Takeaways & Limitations
Visualizations show that fusion stages coordinate to capture synchronous and asynchronous multimodal interactions while learning increasingly discriminative representations.
Abstract
from arXiv · showhide
Computational modeling of human multimodal language is an emerging research area in natural language processing spanning the language, visual and acoustic modalities. Comprehending multimodal language requires modeling not only the interactions within each modality (intra-modal interactions) but more importantly the interactions between modalities (cross-modal interactions). In this paper, we propose the Recurrent Multistage Fusion Network (RMFN) which decomposes the fusion problem into multiple stages, each of them focused on a subset of multimodal signals for specialized, effective fusion. Cross-modal interactions are modeled using this multistage fusion approach which builds upon intermediate representations of previous stages. Temporal and intra-modal interactions are modeled by integrating our proposed fusion approach with a system of recurrent neural networks. The RMFN displays state-of-the-art performance in modeling human multimodal language across three public datasets relating to multimodal sentiment analysis, emotion recognition, and speaker traits recognition. We provide visualizations to show that each stage of fusion focuses on a different subset of multimodal signals, learning increasingly discriminative multimodal representations.
1 Introduction
Multimodal language analysis must capture both within-modality structure and cross-modal interactions across language, visual, and acoustic signals. RMFN addresses this through recursive, specialized fusion stages integrated with recurrent networks, achieving state-of-the-art results across three tasks and datasets.
- Multimodal language modeling spans language, visual, and acoustic signals and supports sentiment, emotion, and speaker-trait recognition tasks.
- Intra-modal interactions capture structure within one modality, whereas cross-modal interactions capture relationships between modalities across synchronized or delayed behaviors.
- Multistage fusion is motivated by evidence that higher-level cross-modal concepts can build on representations formed during earlier processing stages.
- RMFN decomposes fusion into recursive stages that highlight subsets of multimodal signals, fuse them with prior representations, and integrate recurrent networks for temporal and intra-modal interactions.The model jointly represents intra-modal and cross-modal interactions and is differentiable end-to-end.
- RMFN achieves state-of-the-art performance across sentiment analysis, emotion recognition, and speaker-trait recognition on three public multimodal datasets.Ablations and visualizations support the advantages of explicitly defining multiple recursive fusion stages.
2 Related Work
Prior multimodal language approaches include non-temporal models that summarize modality-specific temporal information and newer neural models using recurrent sequence modeling. The related work therefore spans progressively more temporal approaches to multimodal representation.
- Non-temporal Models: Non-temporal approaches simplify multimodal modeling by summarizing temporal observations, often representing each modality through time averages.
- The related-work landscape includes hierarchical sequence summarization models that recursively build hierarchical representations.
- Multimodal Temporal Neural Networks: Recurrent neural networks have been used extensively for language and speech sequence modeling and for sentiment-analysis applications.
3 Recurrent Multistage Fusion Network
RMFN models multimodal language by recursively highlighting subsets of signals, fusing them locally with prior-stage representations, and summarizing the resulting cross-modal interactions. Recurrent networks integrate these fusion representations with temporal and intra-modal information for end-to-end multimodal prediction.
- RMFN represents language, visual, and acoustic inputs as temporal sequences processed by intra-modal recurrent networks.Each network produces a unimodal representation at every time step.
- 3.1 Multistage Fusion Process: The Multistage Fusion Process uses HIGHLIGHT, FUSE, and SUMMARIZE modules to model cross-modal interactions through recursive stages.HIGHLIGHT selects signals, FUSE combines them with prior fusion results, and SUMMARIZE produces the final cross-modal representation.
- HIGHLIGHT: At each stage, HIGHLIGHT recurrently assigns attention weights based on previous assignments and uses them to select attended multimodal signals.The HIGHLIGHT LSTM captures dependencies between stage-specific attention decisions.
- FUSE: FUSE simultaneously performs local fusion of highlighted features and integration with representations from previous stages using a FUSE LSTM.Its gates support local fusion and integration with earlier fusion results.
- SUMMARIZE: After K stages, SUMMARIZE generates z_t, a cross-modal representation containing the interactions discovered during fusion, and feeds it back into the recurrent networks.The final fusion representation is integrated with temporal intra-modal processing through Long Short-term Hybrid Memories.
- 3.4 Optimization: The final representation E concatenates the last LSTHM outputs with the last cross-modal representation for multimodal analysis.E summarizes modeled intra-modal and cross-modal representations and can support supervised or unsupervised analysis.
4 Experimental Setup
The experiments evaluate RMFN across sentiment analysis, emotion recognition, and speaker traits recognition using multimodal video datasets, features aligned across language, visual, and acoustic modalities, and established baselines and metrics. Results tables report RMFN as outperforming prior state-of-the-art systems on sentiment and emotion recognition.
- Evaluation domains: RMFN is evaluated on multimodal sentiment analysis, emotion recognition, and speaker traits recognition across three domains of human multimodal language.
- Datasets: The datasets use monologue videos in which speaker intentions are conveyed through language, visual, and acoustic modalities.
- Features and alignment: Language, visual, and acoustic features are extracted with GloVe, Facet, and COVAREP, respectively, then aligned to word utterance intervals using forced alignment.Aligned video and audio features are computed over each word’s utterance time interval.
- Results: RMFN significantly outperforms the current state of the art across all CMU-MOSI sentiment evaluation metrics.Table 1 marks best results and ΔSOTA improvements over the previous state of the art.
- Results: RMFN significantly outperforms the current state of the art across IEMOCAP emotion evaluation metrics, except for ΔSOTA entries shown in gray.The table highlights best results and improvements over the previous state of the art.
- Baseline models: The comparison includes recurrent, attention-based, tensor-product, gating, context-dependent, and concatenation-based multimodal models.Baselines include MFN, MARN, GME-LSTM(A), TFN, BC-LSTM, EF-LSTM, and stacked or bidirectional EF-LSTM variants.
5 Results and Discussion
RMFN achieves strong results across multimodal tasks and analyses indicate that multistage fusion, cross-modal interaction modeling, and attention-based highlighting contribute to performance. Visualizations show stage- and time-varying attention that captures synchronized, asynchronous trimodal, and bimodal interactions.
- 5.1 Performance on Multimodal Language: RMFN achieves state-of-the-art or competitive results across CMU-MOSI, IEMOCAP, and POM, including sentiment, emotion, and speaker-trait recognition.The authors report state-of-the-art performance in all three tasks, while noting a neutral-emotion exception on IEMOCAP.
- 5.2 Analysis of Multistage Fusion: The number of fusion stages improves cross-modal interaction modeling up to K = 3, after which performance decreases, possibly because of overfitting.The experiments compare multistage fusion with single-stage fusion and report the best observed stage count as K = 3.
- 5.2 Analysis of Multistage Fusion: RMFN outperforms independent cross-modal modeling and ablations without cross-modal fusion or attention highlighting.The comparisons identify multistage fusion, cross-modal interactions, and attention-weight highlighting as important components.
- 5.3 Visualizations: Attention weights change across stages and time, with different features highlighted earlier or later and new multimodal information triggering adaptive reweighting.These observations support stage specialization and show that attention adapts when new multimodal information appears.
- 5.3 Visualizations: The visualizations show synchronized word-acoustic cues, asynchronous smile-emphasis cues that disambiguate language, and alternating acoustic-language emphasis in bimodal interactions.Examples include the word “fun” with emphasis and elongation, asynchronous smile and phonological emphasis, and disappointed tone with a soft voice.
6 Conclusion
RMFN decomposes multimodal fusion into multiple stages focused on subsets of signals, achieving strong performance across three datasets while capturing synchronous and asynchronous interactions.
- RMFN decomposes multimodal fusion into multiple stages, each focused on a subset of multimodal signals.The model builds fusion representations across stages rather than processing all multimodal signals in one step.
- RMFN achieves state-of-the-art performance across three publicly available datasets.
- The multiple stages coordinate to capture both synchronous and asynchronous multimodal interactions.
- Future work proposes merging RMFN with memory-based fusion methods because the approaches have complementary strengths.
8 Supplementary Material
The supplementary material describes multimodal feature extraction, temporal alignment, and complete evaluation results across sentiment, emotion, and personality-trait tasks.
- Feature extraction: Language features use 300-dimensional GloVe embeddings trained on 840 billion Common Crawl tokens.The embeddings represent sequences of individual transcript words as word vectors.
- Feature extraction: Visual features include facial action units, landmarks, head pose, gaze tracking, and HOG features extracted at 30Hz.
- Temporal alignment: Forced alignment provides word-level timestamps, while modality features are averaged over each word utterance interval.Word utterance intervals serve as the shared timestep for the aligned modalities.
- Additional results: RMFN improves over state-of-the-art multi-view and dataset-specific approaches across the CMU-MOSI, IEMOCAP, and POM datasets.The evaluation covers sentiment analysis, emotion recognition, and personality-trait regression metrics.
- Additional results: RMFN significantly outperforms prior state of the art across all sentiment-prediction evaluation metrics on CMU-MOSI.
- Additional results: RMFN achieves state-of-the-art or competitive performance across all emotion-recognition metrics on IEMOCAP.
- Additional results: RMFN achieves state-of-the-art or competitive performance across all personality-trait recognition metrics on POM.