Source-linked AI summary
HiGRU: Hierarchical Gated Recurrent Units for Utterance-level Emotion Recognition
Wenxiang Jiao, Haiqin Yang, Irwin King, Michael R. Lyu
TL;DR
Utterance-level emotion recognition must handle context-dependent word meanings, rare emotions, and difficult long-range context. The paper proposes hierarchical GRUs with feature-fusion and self-attention variants, achieving improvements across three dialogue datasets and over trimodal CMN on IEMOCAP. The authors also note that subtle emotions remain difficult to recognize.
Problem
Utterance-level emotion recognition must address context-dependent word meanings, rare emotions, and difficult long-range contextual information.
Method
HiGRU uses a lower-level GRU for word sequences and an upper-level GRU for utterance contexts, with HiGRU-f and HiGRU-sf adding feature fusion and self-attention.
Results
HiGRU models improve over state-of-the-art methods by at least 8.7% on IEMOCAP, 7.5% on Friends, and 6.0% on EmotionPush.
Takeaways & Limitations
The proposed models capture individual word or utterance information and long-range context while handling data imbalance across all three tested datasets.
Takeaways & Limitations
The models remain limited in understanding subtle emotions, even when dialogue context is incorporated.
Abstract
from arXiv · showhide
In this paper, we address three challenges in utterance-level emotion recognition in dialogue systems: (1) the same word can deliver different emotions in different contexts; (2) some emotions are rarely seen in general dialogues; (3) long-range contextual information is hard to be effectively captured. We therefore propose a hierarchical Gated Recurrent Unit (HiGRU) framework with a lower-level GRU to model the word-level inputs and an upper-level GRU to capture the contexts of utterance-level embeddings. Moreover, we promote the framework to two variants, HiGRU with individual features fusion (HiGRU-f) and HiGRU with self-attention and features fusion (HiGRU-sf), so that the word/utterance-level individual inputs and the long-range contextual information can be sufficiently utilized. Experiments on three dialogue emotion datasets, IEMOCAP, Friends, and EmotionPush demonstrate that our proposed HiGRU models attain at least 8.7%, 7.5%, 6.0% improvement over the state-of-the-art methods on each dataset, respectively. Particularly, by utilizing only the textual feature in IEMOCAP, our HiGRU models gain at least 3.8% improvement over the state-of-the-art conversational memory network (CMN) with the trimodal features of text, video, and audio.
1 Introduction
Utterance-level emotion recognition must resolve context-dependent word meanings, imbalanced emotion frequencies, and difficult long-range dependencies. HiGRU addresses these challenges with hierarchical GRUs and progressive feature-fusion variants.
- Emotion recognition supports the development of empathetic machines and includes six basic emotion types.
- The same word can express different emotions depending on dialogue context, as “okay” illustrates in Friends.The example associates “okay” with anger, neutral, and joy in different contexts.
- Rare emotions and majority-class dominance require models to remain sensitive to minority emotions.
- HiGRU uses lower- and upper-level bidirectional GRUs to model words within utterances and contextual relationships across utterances.The lower level produces individual utterance embeddings, while the upper level captures sequential and contextual relationships.
- HiGRU-f fuses individual word and utterance inputs with GRU hidden states, while HiGRU-sf additionally applies self-attention.
- 8.7%, 7.5%, and 6.0% improvement over state-of-the-art methods was achieved on IEMOCAP, Friends, and EmotionPush, respectively.Using only text on IEMOCAP, the models improved at least 3.8% over trimodal CMN.
2 Related Work
Prior work either models contextual relationships among dialogue utterances or enriches individual representations, but important limitations remain. HiGRU is positioned as a hierarchical alternative for combining these forms of information.
- Earlier text emotion models often treat texts independently, preventing them from capturing inter-dependence among dialogue utterances.
- Contextual Information Extraction: Contextual models such as bcLSTM and CMN model sequential or speaker-related emotional influence, but CMN is considered too complicated for small dialogue datasets.
- Representation Enrichment: Multimodal features enrich utterance representations, yet prior work reports that textual features dominate visual and audio features for emotion recognition.
- Representation Enrichment: CNN-based representations learn individual utterance embeddings but do not capture contextual information within each utterance well.
- Representation Enrichment: Hierarchical RNNs perform well in several related tasks but remain underexplored for utterance-level emotion recognition in dialogue systems.
- HiGRU-sf combines self-attention and feature fusion, while removing these layers yields HiGRU-f and vanilla HiGRU.
3 Approach
HiGRU performs utterance-level emotion recognition with hierarchical bidirectional GRUs that model words within utterances and contextual utterance sequences. Its variants preserve individual features, add self-attention for long-range context, and use class-weighted training for imbalanced emotions.
- Task Definition: Utterance-level emotion recognition tags each new dialogue utterance with an emotion label from the set of emotions.The task models dialogues as sequences of utterances associated with speakers and emotion labels.
- HiGRU: HiGRU uses a lower-level bidirectional GRU for word sequences and an upper-level bidirectional GRU for contextual utterance sequences.The lower level produces individual utterance embeddings, while the upper level captures sequential and contextual relationships among utterances.
- HiGRU: The vanilla model max-pools contextual word embeddings into individual utterance embeddings, then feeds contextual utterance embeddings through a fully connected layer and softmax.The upper-level hidden states are concatenated before the contextual utterance embedding is classified.
- HiGRU-f: HiGRU-f fuses individual word and utterance embeddings with GRU hidden states to reduce feature dilution and the influence of majority emotions.The fused representations retain individual information at both hierarchy levels.
- HiGRU-sf: HiGRU-sf applies self-attention separately to forward and backward hidden states to capture global context, including in long sequences.The resulting left and right context embeddings are fused with individual features and GRU states.
- Model Training: Training minimizes weighted categorical cross-entropy, assigning larger loss weights to minority emotion classes to address data imbalance.The weighting is inversely proportional to training frequency, with a constant controlling distribution smoothness.
4 Experiments
Experiments evaluate HiGRU models on three textual dialogue emotion datasets using WA and UWA, comparing them with established baselines and examining training-set mixing, encoder scale, and qualitative predictions.
- Datasets: The experiments use IEMOCAP, Friends, and EmotionPush, evaluating four emotions in Friends and EmotionPush while excluding the remaining classes during training.IEMOCAP evaluation uses anger, happiness, sadness, and neutral; Friends and EmotionPush use anger, joy, sadness, and neutral.
- Evaluation metrics: WA and UWA are adopted for fair comparison, with UWA favored on Friends and EmotionPush because their large neutral class compromises WA.WA weights classes by their testing-set proportions, whereas UWA gives classes equal weight.
- Main results: HiGRU models outperform state-of-the-art methods with significant margins across all three datasets.The comparison includes bcLSTM, CMN, and SA-BiLSTM, among other baselines.
- Main results: On IEMOCAP, vanilla HiGRU improves WA by at least 8.7% over CMN with text and 3.8% over CMN with trimodal features.HiGRU-f and HiGRU-sf further improve over vanilla HiGRU by 0.9% and 1.5% in WA, and 1.0% and 1.2% in UWA.
- Main results: On Friends and EmotionPush, HiGRU improves UWA over CNN-DCNN by at least 7.5% and 6.0%, respectively.The model also shows significantly better performance on the minority emotions anger and sadness.
- Additional analyses: Mixing Friends and EmotionPush training data does not necessarily improve performance, while HiGRU-sf remains strong across encoder scales and outperforms bcGRU under a fair setting.Qualitative cases indicate HiGRU-sf uses utterance-level and contextual information to distinguish different emotions conveyed by “okay,” although subtle emotions remain difficult and audio or video may help.
5 Conclusion
HiGRU learns individual utterance embeddings and utterance contexts hierarchically, with variants incorporating word/utterance-level inputs and long-range context. Experiments report significant performance gains across all three tested datasets, while future work targets data scarcity.
- HiGRU uses a lower-level GRU for individual utterance embeddings and an upper-level GRU for utterance contexts.
- HiGRU-f and HiGRU-sf incorporate word/utterance-level inputs and long-range contextual information, respectively.
- The proposed HiGRU models handle data imbalance and textual information effectively, yielding significant performance gains on all three tested datasets.
- Future work will explore semi-supervised learning to address data scarcity in utterance-level emotion recognition.