Source-linked AI summary
Multi-attention Recurrent Network for Human Communication Comprehension
Amir Zadeh, Paul Pu Liang, Soujanya Poria, Prateek Vij, Erik Cambria, Louis-Philippe Morency
TL;DR
Understanding human communication requires AI to model coordinated language, vision, and acoustic signals together with their interactions over time. MARN addresses this using Multi-attention Blocks and hybrid recurrent memories, achieving state-of-the-art results across six public datasets and 16 communication-related attributes.
Problem
AI still faces a fundamental challenge in understanding human communication because it must model both individual modalities and their interactions.
Method
MARN uses modality-specific Long-short Term Hybrid Memories to store within- and cross-modality dynamics, while Multi-attention Blocks discover multiple cross-view dynamics at each time step.
Results
MARN achieves state-of-the-art results on six publicly available datasets spanning sentiment analysis, speaker trait recognition, and emotion recognition, across 16 attributes.
Takeaways & Limitations
MARN provides a unified recurrent architecture for continuously modeling view-specific and cross-view dynamics in multimodal human communication.
Takeaways & Limitations
The evaluated datasets contain videos with only one speaker in front of the camera.
Abstract
from arXiv · showhide
Human face-to-face communication is a complex multimodal signal. We use words (language modality), gestures (vision modality) and changes in tone (acoustic modality) to convey our intentions. Humans easily process and understand face-to-face communication, however, comprehending this form of communication remains a significant challenge for Artificial Intelligence (AI). AI must understand each modality and the interactions between them that shape human communication. In this paper, we present a novel neural architecture for understanding human communication called the Multi-attention Recurrent Network (MARN). The main strength of our model comes from discovering interactions between modalities through time using a neural component called the Multi-attention Block (MAB) and storing them in the hybrid memory of a recurrent component called the Long-short Term Hybrid Memory (LSTHM). We perform extensive comparisons on six publicly available datasets for multimodal sentiment analysis, speaker trait recognition and emotion recognition. MARN shows state-of-the-art performance on all the datasets.
Introduction
Human communication combines language, vision, and acoustic signals with view-specific and cross-view dynamics that unfold synchronously or asynchronously. MARN is proposed to model both kinds of dynamics continuously through time.
- Multimodal communication: Human communication coordinates language, vision, and acoustic modalities to convey intentions.Language includes words, phrases, and sentences; vision includes gestures and expressions; acoustic signals include paralinguistics and vocal-tone changes.
- Communication dynamics: Coordination across modalities creates view-specific dynamics within modalities and cross-view dynamics between modalities.Cross-view dynamics may be synchronous, such as a smile with a positive sentence, or asynchronous, such as laughter after a sentence.
- Research challenge: Machines must understand both view-specific and cross-view dynamics to comprehend human communication.
- Proposed approach: MARN explicitly models both dynamics continuously through time using modality-specific LSTHMs and the Multi-attention Block.Each LSTHM stores modality-specific dynamics and related cross-view dynamics, while the MAB discovers multiple cross-view dynamics at each recurrence step.
Related Work
Prior multimodal approaches simplify temporal interactions, concatenate features, combine modality decisions, or extend multi-view sequence models. MARN differs by explicitly modeling both view-specific and cross-view dynamics with dedicated recurrent memory and attention components.
- Non-temporal Models: Non-temporal models collapse time to represent cross-modal co-occurrences, limiting their handling of contradictory evidence and long sequences.Averaging modality information through time can make representations less informative over long periods.
- Early Fusion: Early fusion concatenates multimodal features and relies on generic models rather than explicitly modeling view-specific and cross-view dynamics.These approaches may also remove time and can overfit because they lack detailed modeling of modality-specific dynamics.
- Late Fusion: Late fusion combines separate modality-model outputs through decision voting, which does not capture complex cross-view dynamics.
- Multi-view Learning: Multi-view learning extends sequence models such as Hidden Markov Models, Hidden Conditional Random Fields, and LSTMs to multiple modalities.
- MARN: MARN differs by using an LSTHM for each modality and an MAB for explicit, temporally advanced modeling of cross-view dynamics.
MARN Model
MARN models modality-specific and cross-modal dynamics over time using modality-specific hybrid memories and a multi-attention mechanism. Its MAB discovers multiple cross-view dynamics and encodes them for recurrent memory updates.
- Multi-attention Block: The MAB uses all LSTM outputs at a timestep to regress multiple attention coefficients identifying cross-view dynamics.It applies K softmax distributions to the concatenated LSTHM outputs.
- MARN Architecture: MARN combines LSTHM and MAB components in an end-to-end differentiable recurrent pipeline over language, vision, and acoustic modalities.The formulation iterates LSTHM and MAB steps across the sequence.
- Long-short Term Hybrid Memory: MARN assigns each modality an LSTHM that stores both its view-specific dynamics and related cross-view dynamics.The hybrid memory extends LSTM memory for multimodal setups.
- Multi-attention Block: MAB captures cross-view dynamics that may occur simultaneously across modalities or be asynchronous across time.For example, a current smile can connect to an earlier excited voice retained by acoustic memory.
- Multi-attention Block: Attention-weighted LSTHM outputs are split by modality, reduced into dense codes, and combined to generate the neural cross-view dynamics code z_t.The code is passed to the individual LSTHMs as the hybrid factor for subsequent memory updates.
Experimental Methodology
The experiments evaluate MARN on six public multimodal datasets spanning sentiment, speaker traits, and emotion, using aligned language, vision, and acoustic inputs. Performance is compared with neural, probabilistic, kernel, voting, and human-reference baselines using task-appropriate metrics.
- Benchmarks: MARN is evaluated on six publicly available datasets covering multimodal sentiment analysis, speaker trait recognition, and emotion recognition.The datasets use speaker-disjoint train, validation, and test splits.
- Results: MARN achieves significant improvement over comparison methods on the ICT-MMMO, YouTube, and Spanish MOUD sentiment datasets.The MOUD evaluation tests generalization to speakers communicating in another language.
- Feature Preparation: Language inputs use 300-dimensional pre-trained GloVe word vectors derived from manual transcripts.The embeddings are applied to each video’s sequence of transcript words.
- Feature Preparation: Vision features include frame-level emotions and facial action units, while acoustic features include MFCCs, pitch, voicing, and glottal-source descriptors.Facet extracts visual features and COVAREP extracts low-level acoustic features.
- Feature Preparation: Inputs are aligned at the word level, with audio aligned to transcript words and visual and acoustic features assigned the same granularity.Speech pauses are represented as zero-valued word vectors.
- Evaluation: Classification reports accuracy and F1, while regression reports MAE and Pearson’s r, with lower values preferred only for MAE.The experiments compare MARN with state-of-the-art approaches and multiple fusion and recurrent baselines.
- Results: On CMU-MOSI, MARN achieves new state-of-the-art results across all reported metrics.The result is presented as evidence of the model’s multimodal sentiment-comprehension capability.
Experimental Results
MARN achieves state-of-the-art results across multimodal sentiment analysis, speaker trait recognition, and emotion recognition, with one reported exception in dominance correlation.
- CMU-MOSI: MARN achieves new state-of-the-art results on CMU-MOSI across all reported metrics.The authors describe this as evidence of capability in understanding sentiment aspects of multimodal communication.
- Sentiment Analysis: MARN significantly improves over state-of-the-art approaches on two English sentiment analysis datasets and on the Spanish MOUD dataset.The comparison covers ICT-MMMO, YouTube, and MOUD.
- Speaker Traits: MARN achieves state-of-the-art accuracy on all 11 POM speaker trait recognition tasks, including persuasiveness and credibility.The tasks assess traits inferred from multimodal communicative behaviors.
- Emotion Recognition: MARN achieves state-of-the-art performance for IEMOCAP emotion classification and continuous emotion regression, except for dominance correlation.Dominance correlation is competitive but not state of the art.
Discussion
Ablations and visualizations indicate that continuous cross-view modeling, attention coefficients, and multiple attentions are important, while attention requirements vary by dataset and task.
- Ablation Findings: Across all datasets, removing the Multi-attention Block makes MARN learn only simple modality rules and perform worse than the full model.The authors therefore identify continuous modeling of cross-view dynamics as crucial for understanding human communication.
- Ablation Findings: Removing attention coefficients severely degrades MARN, indicating that attentions are important for accurately modeling cross-view dynamics.The comparison is between MARN and the MARN (no A) baseline.
- Ablation Findings: MARN with multiple attentions outperforms a one-attention version, and parameter increases did not improve the one-attention model.The authors attribute the difference to better cross-view-dynamics modeling rather than simply more parameters.
- Ablation Findings: Different tasks and datasets require different numbers of attentions, depending on dataset characteristics and modality interconnections.This addresses the fourth research question concerning attention count.
- Attention Visualization: The attention visualization shows that attentions are diverse, change over time, and can focus on cross-view dynamics involving only two modalities.The figure uses blue for activated attentions and red for non-activated attentions.
- Reported Comparisons: The POM and IEMOCAP table captions state that MARN outperforms previous baselines and that best scores are highlighted.These tables report speaker personality trait recognition and multimodal emotion recognition results, respectively.
- Attention Visualization: Some attention coefficients remain inactive because their dimensions carry view-specific rather than cross-view dynamics.These dimensions receive no weight in forming cross-view dynamics.
- Attention Visualization: Different attentions can act complementarily, with some activating broad variable sets and others activating fewer sets.The authors describe these patterns as residual effects.
Conclusion
The paper presents MARN as a recurrent architecture that models view-specific and cross-view dynamics continuously through time. It reports state-of-the-art results across six datasets and 16 communication-related attributes.
- Conclusion: MARN uses an LSTHM for each modality and a Multi-attention Block to identify cross-view dynamics at each time step.The block produces a multimodal neural code for the LSTHM hybrid memory.
- Conclusion: MARN achieves state-of-the-art results on six publicly available datasets across 16 attributes related to understanding human communication.The reported scope spans multimodal sentiment, speaker traits, and emotion recognition.