Source-linked AI summary
Memory Fusion Network for Multi-view Sequential Learning
Amir Zadeh, Paul Pu Liang, Navonil Mazumder, Soujanya Poria, Erik Cambria, Louis-Philippe Morency
TL;DR
Multi-view sequential learning requires modeling both view-specific and cross-view interactions across time. MFN assigns an LSTM to each view, uses DMAN to identify cross-view interactions, and stores them in Multi-view Gated Memory. Across public benchmark datasets, MFN outperformed baselines and set state-of-the-art results.
Problem
Multi-view sequential learning must capture interactions within individual views and interactions across different views and time.
Method
MFN uses separate LSTMs for view-specific dynamics, DMAN for cross-view and temporal interactions, and Multi-view Gated Memory to store cross-view information over time.
Results
MFN outperformed proposed baselines and set new state-of-the-art results across all six publicly available multi-view sequential datasets.
Takeaways & Limitations
MFN provides a recurrent architecture that explicitly models view-specific and cross-view interactions throughout multi-view sequences.
Abstract
from arXiv · showhide
Multi-view sequential learning is a fundamental problem in machine learning dealing with multi-view sequences. In a multi-view sequence, there exists two forms of interactions between different views: view-specific interactions and cross-view interactions. In this paper, we present a new neural architecture for multi-view sequential learning called the Memory Fusion Network (MFN) that explicitly accounts for both interactions in a neural architecture and continuously models them through time. The first component of the MFN is called the System of LSTMs, where view-specific interactions are learned in isolation through assigning an LSTM function to each view. The cross-view interactions are then identified using a special attention mechanism called the Delta-memory Attention Network (DMAN) and summarized through time with a Multi-view Gated Memory. Through extensive experimentation, MFN is compared to various proposed approaches for multi-view sequential learning on multiple publicly available benchmark datasets. MFN outperforms all the existing multi-view approaches. Furthermore, MFN outperforms all current state-of-the-art models, setting new state-of-the-art results for these multi-view datasets.
Introduction
Multi-view sequential learning must model both interactions within individual views and interactions across views and time. MFN addresses this by separately encoding each view, identifying cross-view interactions, and storing them over time, outperforming baselines across six public datasets.
- Multi-view data combines heterogeneous views whose complementary information can improve comprehensive and accurate description.
- Multi-view sequential learning requires modeling view-specific interactions within one view and cross-view interactions spanning views and time.
- MFN assigns each view an LSTM, uses DMAN to identify cross-view interactions, and stores them over time in Multi-view Gated Memory.
- MFN integrates view-specific and cross-view information for prediction.
- MFN outperformed baselines and set new state-of-the-art results across six publicly available multi-view sequential datasets.
Related Work
Prior approaches concatenate views, partition shared models, or collapse temporal information, which limits explicit modeling of view-specific and cross-view dynamics. MFN instead assigns one LSTM per view and explicitly models both interaction types over time.
- Researchers have focused on concatenating views, creating multi-view structured models, or collapsing temporal dimensions into view representations.
- Concatenation-based approaches simplify learning by feeding all views as one input to models such as HMMs, SVMs, HCRFs, and recurrent networks.
- Multi-view structured models modify existing approaches or partition LSTM memory into components for different views.
- Temporal-collapsing methods use average features or conventional multi-view techniques, limiting representations for long sequences because temporal information is insufficiently retained.
- MFN assigns one LSTM to each view, learns view-specific interactions in isolation, and uses attention plus memory to model cross-view interactions over time.
Memory Fusion Network (MFN)
MFN models multi-view sequences with separate LSTMs for view-specific dynamics, attention for cross-view interactions, and gated memory for their temporal history.
- System of LSTMs: MFN assigns one LSTM to each view to encode view-specific dynamics over time.
- Delta-memory Attention Network: The Delta-memory Attention Network identifies cross-view and temporal interactions across different dimensions of LSTM memories.
- Delta-memory Attention Network: DMAN compares memories at adjacent timesteps and assigns higher coefficients to dimensions involved in changing cross-view interactions.
- Delta-memory Attention Network: Element-wise attention amplifies relevant memory dimensions while allowing interactions across different timestamps through information retained in LSTM memories.
- Multi-view Gated Memory: The Multi-view Gated Memory stores cross-view interactions over time as a unifying memory for the separate LSTM memories.
- Multi-view Gated Memory: Retain and update gates control how much of the current gated-memory state is preserved or changed by each cross-view update proposal.
Experimental Setup
The experiments evaluate MFN on multimodal language, vision, and acoustic sequences across sentiment, emotion, and speaker-trait tasks using speaker-independent benchmarks and baselines.
- Tasks and datasets: Experiments cover multimodal sentiment analysis, emotion recognition, and speaker-trait analysis across six publicly available datasets.
- Tasks and datasets: All benchmarks use language, visual, and acoustic views from videos of people speaking about a topic.
- Feature extraction: Visual features include facial action units, landmarks, head pose, gaze, and HOG descriptors extracted at 30Hz.
- Feature extraction: Audio features include Mel-frequency cepstral coefficients, pitch, voiced/unvoiced segments, glottal parameters, peak slopes, and dispersion quotients.
- Data splits: The datasets enforce speaker-independent training, validation, and testing splits so the same speaker does not appear in both training and test sets.
- Sequence alignment: Sequences are aligned by word utterances, with expected visual and acoustic features computed for each word across the LSTM inputs.
- Baselines: Baselines include concatenated-view HMM, SVM, HCRF, LDHCRF, and LSTM models, plus multi-view HCRF and multi-view LSTM approaches.
MFN Results and Discussion
MFN achieves state-of-the-art performance across multi-view sequential benchmarks, with consistent classification and regression gains. Its advantages are accompanied by substantially fewer parameters and faster inference, while ablations support the roles of prior memories and Multi-view Gated Memory.
- MFN Results: MFN sets new state-of-the-art results across all evaluated datasets and shows consistent performance for both classification and regression.Other baselines vary by dataset and evaluation task.
- MFN Results: MFN achieves roughly 5e5 parameters and 2858 IPS, compared with roughly 2e7 parameters and 278 IPS for Zadeh2017.The comparison uses a Nvidia GTX 1080 Ti GPU and average data point inferences per second.
- Ablation Studies: Removing prior-time memory reduces performance, indicating a crucial role for the memories at time t −1.This conclusion comes from comparing MFN with MFN (no ∆).
- Ablation Studies: Removing Multi-view Gated Memory demonstrates its essential role in MFN performance.This conclusion comes from comparing MFN with MFN (no mem).
- Ablation Studies: Expanding DMAN's input region from [t −1,t] to [t −q,t] does not significantly improve performance.The authors attribute this to additional memory steps not adding information to DMAN's internal mechanism.
Conclusion
The paper introduces MFN as a multi-view sequential learning architecture that models within-view dynamics, cross-view relations, and their temporal fusion. Experiments on multiple public datasets report state-of-the-art performance across all datasets.
- Conclusion: MFN assigns each view an LSTM to model interactions within that view.This component is called the System of LSTMs.
- Conclusion: DMAN models cross-view relations through time by assigning relevance scores to the memory dimensions of each view's LSTM.Its outputs are computed over the System of LSTMs' memories.
- Conclusion: Multi-view Gated Memory unifies the sequences by updating its content from DMAN outputs.The outputs are calculated over the memories in the System of LSTMs.
- Conclusion: MFN achieves state-of-the-art performance in multi-view sequential learning on all evaluated datasets.The model was compared with various baselines using multiple publicly available datasets.