Source-linked AI summary
Real-Time Emotion Recognition via Attention Gated Hierarchical Memory Network
Wenxiang Jiao, Michael R. Lyu, Irwin King
TL;DR
RTER seeks utterance-level emotion recognition in ongoing conversations without access to future context, so historical memories must be represented and summarized effectively. The paper proposes AGHMN, combining hierarchical bidirectional recurrent encoding with attention-gated memory summarization. Across two emotion-conversation datasets, AGHMN models outperform compared approaches, with reported gains for the best variants and remaining difficulty on some minority classes.
Problem
Without future context, RTER requires effective historical-memory construction and summarization, while prior CNN, unidirectional GRU, and Soft Attention designs have stated representation or ordering limitations.
Method
AGHMN combines a BiGRU-based Hierarchical Memory Network with AGRU and BiAGRU modules that use attention-weighted recurrent updates to summarize memories.
Results
AGHMN models outperform compared methods on two emotion-conversation datasets; the best variants report gains up to +2.8% Acc, +2.2% F1, and +3.3% mF1.
Takeaways & Limitations
The experiments and attention visualizations support the efficacy of the proposed hierarchical memory and attention-gated summarization components.
Takeaways & Limitations
The models still struggle with minority classes such as disgust on MELD, where additional data or multimodal features may help disambiguate emotions.
Abstract
from arXiv · showhide
Real-time emotion recognition (RTER) in conversations is significant for developing emotionally intelligent chatting machines. Without the future context in RTER, it becomes critical to build the memory bank carefully for capturing historical context and summarize the memories appropriately to retrieve relevant information. We propose an Attention Gated Hierarchical Memory Network (AGHMN) to address the problems of prior work: (1) Commonly used convolutional neural networks (CNNs) for utterance feature extraction are less compatible in the memory modules; (2) Unidirectional gated recurrent units (GRUs) only allow each historical utterance to have context before it, preventing information propagation in the opposite direction; (3) The Soft Attention for summarizing loses the positional and ordering information of memories, regardless of how the memory bank is built. Particularly, we propose a Hierarchical Memory Network (HMN) with a bidirectional GRU (BiGRU) as the utterance reader and a BiGRU fusion layer for the interaction between historical utterances. For memory summarizing, we propose an Attention GRU (AGRU) where we utilize the attention weights to update the internal state of GRU. We further promote the AGRU to a bidirectional variant (BiAGRU) to balance the contextual information from recent memories and that from distant memories. We conduct experiments on two emotion conversation datasets with extensive analysis, demonstrating the efficacy of our AGHMN models.
Introduction
The paper frames real-time emotion recognition as utterance-level emotion detection without future context, making historical-memory construction and summarization central challenges. It proposes AGHMN to improve utterance representations, contextual memory, and memory summarization.
- Real-time emotion recognition detects a speaker’s emotion at each utterance in an ongoing text conversation for developing empathetic machines.
- Without future context, RTER must carefully capture historical utterances in a memory bank and summarize them to retrieve relevant information.
- Existing memory banks use hierarchical word-to-utterance and utterance-to-conversation structures, but common CNN and unidirectional GRU designs have compatibility and context-propagation limitations.
- Bag-of-memories summarization loses the positional and ordering information of memories regardless of how the memory bank is built.
- AGHMN uses a BiGRU utterance reader, a BiGRU fusion layer, and AGRU or BiAGRU memory summarization to improve contextual representation while retaining memory order.
Related Work
Prior conversation-emotion models include static sequence or graph approaches and dynamic models that process utterances as they arrive. AGHMN instead targets improved utterance and memory representations with hierarchical and attention-gated recurrent components.
- Context-dependent emotion models are divided into static models, which use broader conversational context, and dynamic models, which read utterances as they are generated.
- Existing approaches use sequence, graph, self-attention, residual fusion, or party-state tracking to represent conversational context.
- AGHMN differs by using HMN for improved utterance and memory representations and AGRU or BiAGRU for memory summarization.
- Unlike DialogueRNN, AGHMN does not explicitly distinguish speakers, relying on the model to recognize speaker differences itself.
Task Specification
RTER predicts the emotion of the current query utterance from the historical utterances available so far in a conversation. Each utterance is assigned an emotion from a predefined set.
- Given a conversation through turn t, the model predicts the emotion of the t-th query utterance using the other utterances as historical context.
- Each utterance expresses a major emotion from a set such as joy, sadness, and neutral.
Architecture
AGHMN combines a hierarchical encoder with attention-gated recurrent memory summarization to represent utterances and historical conversational context. Its BiGRU-based fusion and BiAGRU components preserve sequential information while balancing recent and distant memories.
- Architecture: AGHMN consists of a word embedding layer, Hierarchical Memory Network, Attention GRU, and classifier.The model is trained with cross-entropy loss for emotion classification.
- Hierarchical Memory Network: The HMN is a two-level encoder whose utterance reader uses a BiGRU to model word sequences in both directions.The forward and backward hidden states are concatenated, max-pooled, and transformed into an utterance embedding.
- Hierarchical Memory Network: The fusion layer builds a memory bank from the K most recent utterances and uses GRU-based interaction to refine their representations.UniF models preceding context, whereas BiF uses a BiGRU so memories also receive context from the opposite direction.
- Attention GRU: AGRU replaces weighted-sum memory summarization with attention-controlled GRU updates that retain memory position and ordering.Its final hidden state becomes the contextual vector used to refine the query representation.
- Attention GRU: BiAGRU balances recent-memory information from AGRU with distant-memory information from a backward AGRU, especially for long conversations.The resulting contextual vectors refine the query before classification.
Experimental Setup
The experiments evaluate AGHMN variants on IEMOCAP and MELD against established baselines using class-level and aggregate emotion-recognition metrics. Training uses CNN-based textual utterance features and fixed context windows with regularized optimization.
- Datasets: The evaluation uses IEMOCAP and MELD conversation emotion datasets with different speaker and emotion configurations.IEMOCAP contains dyadic conversations and six emotion classes, while MELD contains multi-speaker conversations and seven emotion types.
- Compared Methods: Experiments compare four AGHMN variants with scLSTM, CMN, DialogueRNN, and ICON as baselines.The variants combine different memory-bank configurations with AGRU or BiAGRU summarizers.
- Implementation: GRU and AGRU hidden sizes are 100, with context windows of 40 for IEMOCAP and 10 for MELD.The context-window sizes approximate the average conversation lengths in the two datasets.
- Training: Training uses Adam with learning rate 5 × 10^-4, gradient clipping, dropout, validation mF1-based decay, and early stopping.Gradients are clipped at norm 5, dropout is 0.3, and early stopping uses patience 10.
Results
AGHMN models outperform compared methods across IEMOCAP and MELD, with BiF-AGRU and UniF-BiAGRU achieving the strongest dataset-specific results. Analyses support bidirectional memory summarization, BiGRU utterance reading, context-sensitive window sizes, and improved minority-emotion recognition, while MELD disgust remains difficult.
- Overall Results: +4.2% Acc, +3.7% F1, and +5.1% mF1 over DialogueRNN, with BiF-AGRU achieving the best overall IEMOCAP performance.All AGHMN models outperform the compared models on IEMOCAP; gains on happy reach at least +11.5% Acc and +17.3% F1.
- Overall Results: +2.8% Acc, +2.2% F1, and +3.3% mF1 over compared methods, with UniF-BiAGRU best on MELD.The models perform best on most emotion classes and produce more balanced results by mF1, despite degradation on anger.
- AGHMN Variants: UniF-BiAGRU and BiF-AGRU outperform UniF-AGRU, while BiF-BiAGRU is not best and MELD shows limited variant differences.The authors attribute the deeper model’s weaker performance to learning difficulty and MELD’s shorter conversations to reduced module sensitivity.
- Utterance Readers: BiLSTM and BiGRU readers outperform 1-D CNN, with BiGRU performing better and CNN with Soft Attention degrading further.The results indicate that RNN readers are more compatible with the Hierarchical Memory Network and that BiF benefits both recurrent readers.
- Attention Choices & Memory Banks: AGRU outperforms Soft Attention with the UniF memory bank, BiAGRU extends the advantage, and Soft Attention improves when paired with BiF memory.These comparisons support both the attention-gated summarizer and the stronger memory representation produced by BiF.
- Context-Window Size: Performance rises and then falls as context-window size K increases, with best K values of 40 and 50 on IEMOCAP versus 5 and 10 on MELD.The authors associate the shorter MELD optimum with more rapid emotion fluctuation and the longer IEMOCAP optimum with gentler emotion evolution.
Conclusions
AGHMN improves real-time emotion recognition by enhancing utterance features, memory quality, and contextual summarization, with experiments showing strong performance and component efficacy.
- AGHMN outperforms state-of-the-art approaches with significant margins across two emotion conversation datasets.Ablation studies and attention visualization further demonstrate the efficacy of its components.