PaperlayerView exact revision on arXiv

Source-linked AI summary

Memory Fusion Network for Multi-view Sequential Learning

Amir Zadeh, Paul Pu Liang, Navonil Mazumder, Soujanya Poria, Erik Cambria, Louis-Philippe Morency

arXiv:1802.00927v1cs.LGcs.AI

TL;DR

Multi-view sequential learning requires modeling both view-specific and cross-view interactions across time. MFN assigns an LSTM to each view, uses DMAN to identify cross-view interactions, and stores them in Multi-view Gated Memory. Across public benchmark datasets, MFN outperformed baselines and set state-of-the-art results.

Abstract

from arXiv · show

Multi-view sequential learning is a fundamental problem in machine learning dealing with multi-view sequences. In a multi-view sequence, there exists two forms of interactions between different views: view-specific interactions and cross-view interactions. In this paper, we present a new neural architecture for multi-view sequential learning called the Memory Fusion Network (MFN) that explicitly accounts for both interactions in a neural architecture and continuously models them through time. The first component of the MFN is called the System of LSTMs, where view-specific interactions are learned in isolation through assigning an LSTM function to each view. The cross-view interactions are then identified using a special attention mechanism called the Delta-memory Attention Network (DMAN) and summarized through time with a Multi-view Gated Memory. Through extensive experimentation, MFN is compared to various proposed approaches for multi-view sequential learning on multiple publicly available benchmark datasets. MFN outperforms all the existing multi-view approaches. Furthermore, MFN outperforms all current state-of-the-art models, setting new state-of-the-art results for these multi-view datasets.

Introduction

Multi-view sequential learning must model both interactions within individual views and interactions across views and time. MFN addresses this by separately encoding each view, identifying cross-view interactions, and storing them over time, outperforming baselines across six public datasets.

  • Multi-view data combines heterogeneous views whose complementary information can improve comprehensive and accurate description.Source: Introduction · p. 1
  • Multi-view sequential learning requires modeling view-specific interactions within one view and cross-view interactions spanning views and time.Source: Introduction · p. 1
  • MFN assigns each view an LSTM, uses DMAN to identify cross-view interactions, and stores them over time in Multi-view Gated Memory.Source: Introduction · p. 1
  • MFN integrates view-specific and cross-view information for prediction.Source: Introduction · p. 1
  • MFN outperformed baselines and set new state-of-the-art results across six publicly available multi-view sequential datasets.Source: Introduction · p. 1

Related Work

Prior approaches concatenate views, partition shared models, or collapse temporal information, which limits explicit modeling of view-specific and cross-view dynamics. MFN instead assigns one LSTM per view and explicitly models both interaction types over time.

  • Researchers have focused on concatenating views, creating multi-view structured models, or collapsing temporal dimensions into view representations.Source: Related Work · pp. 1–2
  • Concatenation-based approaches simplify learning by feeding all views as one input to models such as HMMs, SVMs, HCRFs, and recurrent networks.Source: Related Work · p. 2
  • Multi-view structured models modify existing approaches or partition LSTM memory into components for different views.Source: Related Work · p. 2
  • Temporal-collapsing methods use average features or conventional multi-view techniques, limiting representations for long sequences because temporal information is insufficiently retained.Source: Related Work · p. 2
  • MFN assigns one LSTM to each view, learns view-specific interactions in isolation, and uses attention plus memory to model cross-view interactions over time.Source: Related Work · p. 2

Memory Fusion Network (MFN)

MFN models multi-view sequences with separate LSTMs for view-specific dynamics, attention for cross-view interactions, and gated memory for their temporal history.

System of LSTMs

Delta-memory Attention Network

Multi-view Gated Memory

  • The Multi-view Gated Memory stores cross-view interactions over time as a unifying memory for the separate LSTM memories.Source: Multi-view Gated Memory · p. 3
  • Retain and update gates control how much of the current gated-memory state is preserved or changed by each cross-view update proposal.Source: Multi-view Gated Memory · pp. 3–4

Experimental Setup

The experiments evaluate MFN on multimodal language, vision, and acoustic sequences across sentiment, emotion, and speaker-trait tasks using speaker-independent benchmarks and baselines.

Tasks and datasets

Feature extraction

  • Visual features include facial action units, landmarks, head pose, gaze, and HOG descriptors extracted at 30Hz.Source: Sequence Features · p. 5
  • Audio features include Mel-frequency cepstral coefficients, pitch, voiced/unvoiced segments, glottal parameters, peak slopes, and dispersion quotients.Source: Sequence Features · p. 5

Data splits

Sequence alignment

Baselines

MFN Results and Discussion

MFN achieves state-of-the-art performance across multi-view sequential benchmarks, with consistent classification and regression gains. Its advantages are accompanied by substantially fewer parameters and faster inference, while ablations support the roles of prior memories and Multi-view Gated Memory.

MFN Results

  • MFN sets new state-of-the-art results across all evaluated datasets and shows consistent performance for both classification and regression.Other baselines vary by dataset and evaluation task.Source: MFN Results and Discussion · p. 7
  • MFN achieves roughly 5e5 parameters and 2858 IPS, compared with roughly 2e7 parameters and 278 IPS for Zadeh2017.The comparison uses a Nvidia GTX 1080 Ti GPU and average data point inferences per second.Source: MFN Results and Discussion · p. 7

Ablation Studies

  • Removing prior-time memory reduces performance, indicating a crucial role for the memories at time t −1.This conclusion comes from comparing MFN with MFN (no ∆).Source: MFN Results and Discussion · p. 7
  • Removing Multi-view Gated Memory demonstrates its essential role in MFN performance.This conclusion comes from comparing MFN with MFN (no mem).Source: MFN Results and Discussion · p. 7
  • Expanding DMAN's input region from [t −1,t] to [t −q,t] does not significantly improve performance.The authors attribute this to additional memory steps not adding information to DMAN's internal mechanism.Source: MFN Results and Discussion · p. 7

Conclusion

The paper introduces MFN as a multi-view sequential learning architecture that models within-view dynamics, cross-view relations, and their temporal fusion. Experiments on multiple public datasets report state-of-the-art performance across all datasets.

Conclusion

  • MFN assigns each view an LSTM to model interactions within that view.This component is called the System of LSTMs.Source: Conclusion · p. 7
  • DMAN models cross-view relations through time by assigning relevance scores to the memory dimensions of each view's LSTM.Its outputs are computed over the System of LSTMs' memories.Source: Conclusion · p. 7
  • Multi-view Gated Memory unifies the sequences by updating its content from DMAN outputs.The outputs are calculated over the memories in the System of LSTMs.Source: Conclusion · p. 7
  • MFN achieves state-of-the-art performance in multi-view sequential learning on all evaluated datasets.The model was compared with various baselines using multiple publicly available datasets.Source: Conclusion · p. 7

Related papers

Loading 1802.00927v1…