Source-linked AI summary
DialogXL: All-in-One XLNet for Multi-Party Conversation Emotion Recognition
Weizhou Shen, Junqing Chen, Xiaojun Quan, Zhixian Xie
TL;DR
ERC needs models that handle multi-party speaker dependencies and long conversational histories, which standard pre-trained language-model use and hierarchical structures do not readily support. DialogXL adapts XLNet with utterance-level memory and dialog-aware self-attention, and it outperforms all baselines on four ERC benchmarks. Error analysis also identifies cases where word-level semantic attention can mislead predictions.
Problem
ERC requires modeling multi-party dependencies and long historical context, while existing approaches and pre-trained language models do not readily address both challenges.
Method
DialogXL adapts XLNet with memory-saving utterance recurrence and dialog-aware self-attention for historical context and speaker dependencies.
Results
DialogXL outperforms all baselines across four ERC benchmarks.
Takeaways & Limitations
The results support using enhanced memory and speaker-aware attention to model multi-turn, multi-party conversational emotion recognition.
Takeaways & Limitations
Word-level semantic attention can focus too strongly on relevance between query and historical utterances, causing mistakes.
Abstract
from arXiv · showhide
This paper presents our pioneering effort for emotion recognition in conversation (ERC) with pre-trained language models. Unlike regular documents, conversational utterances appear alternately from different parties and are usually organized as hierarchical structures in previous work. Such structures are not conducive to the application of pre-trained language models such as XLNet. To address this issue, we propose an all-in-one XLNet model, namely DialogXL, with enhanced memory to store longer historical context and dialog-aware self-attention to deal with the multi-party structures. Specifically, we first modify the recurrence mechanism of XLNet from segment-level to utterance-level in order to better model the conversational data. Second, we introduce dialog-aware self-attention in replacement of the vanilla self-attention in XLNet to capture useful intra- and inter-speaker dependencies. Extensive experiments are conducted on four ERC benchmarks with mainstream models presented for comparison. The experimental results show that the proposed model outperforms the baselines on all the datasets. Several other experiments such as ablation study and error analysis are also conducted and the results confirm the role of the critical modules of DialogXL.
Introduction
Emotion recognition in conversation requires modeling contextual and speaker-dependent information that hierarchical approaches and standard pre-trained language models do not readily capture. DialogXL addresses these challenges with utterance-level memory and dialog-aware self-attention, and outperforms baselines across four ERC benchmarks.
- Motivation: ERC identifies the emotion expressed by each utterance, using conversational context because nearby emotions often remain unchanged.Modeling alternating utterances from different parties is therefore critical.
- Limitations of Prior Work: Prior hierarchical models encode utterances separately and sequentially, but ignore direct word dependencies across utterances and hinder the use of pre-trained language models.These limitations affect applying models such as BERT and XLNet to ERC.
- Challenges: Pre-trained language models face multi-party speaker dependencies and input-length limits that can discard information from distant historical utterances.ERC requires both intra- and inter-speaker modeling and access to longer context.
- Results: DialogXL outperforms all baselines across four ERC benchmarks.The paper also reports ablation and error analyses confirming the importance of its critical modules.
- Contributions: DialogXL replaces XLNet’s segment recurrence with memory-saving utterance recurrence that caches up to 1000 historical words.Historical utterance hidden states are stored in a memory bank and reused for query-utterance prediction.
- Contributions: Dialog-aware self-attention uses different reception fields and party roles to capture intra- and inter-speaker dependencies.Its attention design replaces original self-attention’s word-level weighting with dialog-structured dependencies.
Related Work
ERC research has developed through deep learning and multimodal or textual models, while large pre-trained language models have achieved strong results elsewhere. Before this work, such language models had not been applied to emotion recognition in conversation.
- Background: ERC has attracted substantial NLP interest alongside the growth of conversational datasets and dialogue systems.The task is framed as an important recent research problem.
- Deep Learning Approaches: Earlier ERC systems use GRUs, memory networks, recurrent party and global states, hierarchical GRUs, or graph neural networks.These approaches model conversational structure through recurrent, hierarchical, or graph-based architectures.
- Data Modalities: ERC studies differ in modality, with earlier systems combining text, audio, and video while newer work tends to use text only.The cited latest research is described as textual-modality ERC.
- Pre-trained Language Models: Large pre-trained language models such as BERT and XLNet have shown strong performance across NLP tasks, including XLNet’s long-document processing.XLNet uses segment recurrence, while BERT uses bidirectional Transformer encoders and masked-language pre-training.
- Research Gap: Pre-trained language models had been used for dialogue-related tasks but had not yet been applied to emotion recognition in conversation.This establishes the paper’s application gap.
Methodology
DialogXL applies XLNet to multi-turn, multi-party ERC without constructing a hierarchical processing model. It combines an enhanced memory bank with dialog-aware self-attention for historical context and speaker structure.
- Design Overview: DialogXL addresses long historical context and intra- and inter-speaker dependencies with two improvements built on XLNet.The approach avoids building a hierarchical network for conversational processing.
- Architecture: DialogXL processes each incoming conversation utterance using an embedding layer, 12 Transformer layers, and a feed-forward neural network.It identifies the emotion of each utterance in turn.
- Enhanced Memory: An enhanced memory bank stores hidden states from historical utterances for reuse during training and testing.The memory bank is updated through a new utterance recurrence mechanism.
- Dialog-aware Attention: Dialog-aware self-attention derives hidden states at every Transformer layer while modeling the multi-party conversational structure.This mechanism replaces the corresponding XLNet self-attention computation.
Problem Definition
ERC is defined over a sequence of utterances with emotion labels and speaker identities. The task is to predict a query utterance’s emotion from its preceding context and speaker information.
- Conversation Representation: A conversation is a list of N utterances, and each utterance consists of a sequence of tokens.The paper represents the conversation as {u1, u2, ..., uN}.
- Labels and Speakers: Each utterance has an emotion label drawn from a set of emotion labels, while a speaker function identifies its party.Speaker roles form the collection P for the ERC dataset.
- Prediction Objective: ERC predicts the emotion label of query utterance u_t from historical utterances {u1, u2, ..., u_t−1} and corresponding speaker information.The prediction objective explicitly conditions on preceding conversational context.
Model Input
DialogXL prepends each query utterance with [CLS] and passes it through a word-embedding layer before the Transformer layers.
- Each query utterance u_t is prepended with the special token “[CLS]”.
- DialogXL uses only word embeddings in its embedding layer.
Utterance Recurrence
DialogXL replaces XLNet’s segment recurrence with utterance recurrence to preserve complete conversational units and store longer historical context without padding.
- Segment Recurrence: XLNet and Transformer-XL use segment recurrence to cache previous hidden states and revisit them in future computations.
- Utterance Recurrence: Utterance recurrence stores complete historical utterances rather than fixed-length segments, avoiding padding and enabling longer context.
- Utterance Recurrence: The memory update concatenates newly generated hidden states with the current memory at each Transformer layer.
- Utterance Recurrence: Only utterance-token hidden states are stored, while [CLS] and padding positions are excluded to prevent noise.
Dialog-Aware Self-Attention
Dialog-aware self-attention replaces XLNet’s vanilla self-attention to model multi-party conversational context through global, local, speaker, and listener attention patterns.
- Attention Types: Dialog-aware self-attention uses four attention types: global and local fields, plus speaker and listener dependencies.The four types target different receptive-field sizes and intra- versus inter-speaker relationships.
- Global Self-Attention: Global self-attention attends to all historical context and the query utterance without masking.It can incorporate previously distant utterances that may be useful.
- Local Self-Attention: Local self-attention restricts the reception field to the latest ω historical utterances, masking older context.Here, ω is a tunable hyperparameter motivated by the influence of recent utterances on emotion.
- Speaker Self-Attention: Speaker self-attention models intra-speaker dependency by attending only to historical utterances from the present speaker.Attention to utterances made by other speakers is masked.
- Listener Self-Attention: Listener self-attention models inter-speaker dependency by attending only to historical utterances from other speakers.Attention to utterances made by the present speaker is masked.
- Multi-Head Integration: The four attention outputs are concatenated, normalized, and passed through a feed-forward network, with attention types assigned across heads.Each head uses one of global, local, speaker, or listener attention.
Model Training
The model produces emotion predictions from the final “[CLS]” representation and trains its parameters with standard cross-entropy loss.
- Prediction: DialogXL uses the final-layer “[CLS]” hidden state to encode the query and historical context before predicting emotion.A feed-forward neural network maps this encoding to the predicted emotion.
- Objective: Training uses standard cross-entropy loss over the conversations in the training set.M denotes the number of training conversations, and θ denotes DialogXL’s trainable parameters.
Experimental Settings
Experiments evaluate DialogXL against established ERC models and pretrained-language-model baselines across four multi-turn, multi-party datasets using dataset-specific F1 metrics.
- Overview: The experimental section covers implementation details, datasets, metrics, and baseline models.These settings define the comparison framework for evaluating DialogXL.
- Implementation: DialogXL is initialized with XLNet-Base and trained with AdamW, while tuning learning rate, attention-head allocation, memory length, and dropout.Hyperparameters are tuned on held-out validation data for each dataset.
- Datasets: Evaluation uses four multi-turn, multi-party ERC datasets, including IEMOCAP and MELD, with dataset statistics reported in Table 1.IEMOCAP contains two-party conversations and six emotion labels; its final 20 training dialogues supply validation data.
- Data Modality: Experiments use only textual data from the multimodal datasets.This follows recent ERC evaluation practice cited by the paper.
- Memory Constraint: Utterance recurrence memory is capped by computational resources, and earliest hidden states are dropped when the threshold is exceeded.The implementation therefore does not retain an unlimited history.
- Metrics: Metrics are micro-F1 for DailyDialog and weighted-F1 for the other datasets.The metric choice varies by dataset rather than using one common F1 definition.
- Baselines: Baselines include prior ERC models, BERT, and XLNet.The listed prior methods include CMN, DialogueRNN, HiGRU, DialogueGCN, TL-ERC, and KET.
Results and Analysis
DialogXL achieves state-of-the-art performance across all four datasets, with its advantages depending on conversation length and enhanced contextual modeling. Ablations show that dialog-aware attention types contribute differently, while error analysis identifies attention-related failure modes.
- Overall Results: DialogXL reaches a new state of the art on all four evaluated datasets.
- Overall Results: Pre-trained language models substantially improve over other methods on MELD, DailyDialog, and EmoryNLP, but DialogXL’s gains over BERT and XLNet are limited there.Those datasets mostly contain short dialogues of 5 to 9 utterances, allowing BERT and XLNet to encode the full context in most cases.
- Overall Results: DialogXL significantly improves over BERT and XLNet on IEMOCAP, whose conversations contain around 70 utterances.Its utterance recurrence models historical context, while dialog-aware self-attention captures speaker information.
- Effect of the Enhanced Memory: Segment recurrence wastes over 60% of memory across tested lengths, while performance generally stops improving once maximum memory exceeds 700.The experiment varies maximum memory from 100 to 1000, and utterance recurrence has zero theoretical memory waste.
- Ablation Study: Removing any dialog-aware self-attention type reduces performance on both IEMOCAP and MELD, with speaker and listener attention jointly producing the largest drops.This supports the importance of modeling intra- and inter-speaker dependencies.
- Ablation Study: Removing local self-attention causes the largest IEMOCAP F1 decline, whereas removing global self-attention causes the least degradation.The findings indicate that nearby historical context is especially important for IEMOCAP, while speaker and listener attention capture some distant information.
- Error Study: Word-level attention can support correct predictions through semantic relevance but can also cause errors by overemphasizing relevance between query and historical utterances.The error study therefore suggests combining attention with other mechanisms.
- Error Study: DialogXL makes mistakes on roughly 45% of emotional-shift cases, where consecutive utterances from one speaker receive different emotion labels.
Conclusion
DialogXL extends XLNet for multi-turn, multi-party ERC with enhanced memory and dialog-aware self-attention. Experiments on four benchmarks show that it outperforms all baselines, while analyses support both improvements.
- Conclusion: DialogXL introduces enhanced memory and dialog-aware self-attention to model multi-turn, multi-party conversational data.The enhanced memory stores historical contexts more effectively, while dialog-aware self-attention addresses conversational data structures.
- Conclusion: DialogXL outperforms all baselines on four ERC benchmarks.The paper also reports extensive analyses confirming the effectiveness of its two improvements.
- Conclusion: The original segment recurrence stores more than 60% padding in memory, reducing its effectiveness for ERC historical-context encoding.
- Conclusion: Speaker&listener self-attention is more effective than traditional speaker role embedding and may apply to other dialogue tasks.
- Conclusion: Error analysis finds that relying solely on attention can mislead the model.