Source-linked AI summary
EmoBERTa: Speaker-Aware Emotion Recognition in Conversation with RoBERTa
Taewoon Kim, Piek Vossen
TL;DR
Emotion recognition in conversation requires predicting a speaker’s emotion from dialogue context, but existing approaches often use complex architectures. EmoBERTa adds speaker names and utterance separators to RoBERTa-based sequence classification, and it outperforms other models on MELD and IEMOCAP.
Problem
Emotion recognition in conversation must predict a current speaker’s emotion while accounting for preceding or future utterance context.
Method
EmoBERTa prepends speaker names and separates dialogue utterances before fine-tuning RoBERTa for target-utterance emotion classification.
Results
EmoBERTa outperforms other models on both MELD and IEMOCAP, reaching weighted F1 scores of 65.61% and 68.57%, respectively.
Takeaways & Limitations
Encoding speaker information helps EmoBERTa model dialogue context while retaining the original RoBERTa architecture.
Takeaways & Limitations
The experiments cover text modality only, while multimodal ERC is left for future work.
Abstract
from arXiv · showhide
We present EmoBERTa: Speaker-Aware Emotion Recognition in Conversation with RoBERTa, a simple yet expressive scheme of solving the ERC (emotion recognition in conversation) task. By simply prepending speaker names to utterances and inserting separation tokens between the utterances in a dialogue, EmoBERTa can learn intra- and inter- speaker states and context to predict the emotion of a current speaker, in an end-to-end manner. Our experiments show that we reach a new state of the art on the two popular ERC datasets using a basic and straight-forward approach. We've open sourced our code and models at https://github.com/tae898/erc.
1 Introduction
The paper focuses on emotion recognition in conversation, where the goal is to predict a current speaker’s emotion from dialogue context. EmoBERTa incorporates speaker identity into RoBERTa representations and improves performance on two benchmark datasets.
- ERC predicts the emotion of a current speaker engaged in a conversation, an important capability for affective computing and human-robot communication.
- The reported experiments focus on text modality, while incorporating multiple modalities is left for future work.
- EmoBERTa enriches transformer-based ERC models by encoding speaker identity across multiple utterances.The approach adapts RoBERTa sequence representations to model intra- and inter-speaker context.
- The experiments improve the state of the art on two popular ERC benchmark datasets.
2 Related Work
Prior ERC systems often combine specialized architectures or recurrent sequence models, creating feature-decoupling and interpretability or efficiency concerns. EmoBERTa instead uses RoBERTa with speaker information encoded across multiple utterances.
- Existing ERC approaches commonly combine CNNs, RNNs, Transformers, and GNNs, making feature interactions and model behavior harder to understand.
- RNN-based ERC systems decouple word-embedding extraction from sequence modeling and may require external decontextualized embeddings.
- Recurrent training is inefficient because backpropagation through time waits until the final sequence input has been processed.
- Unlike HiTrans and DialogXL, EmoBERTa uses RoBERTa and encodes speaker information across multiple utterances.
3 Methodology
EmoBERTa treats ERC as contextual sequence classification: it constructs dialogue sequences around a target utterance, includes speaker names, and predicts the target emotion with a fine-tuned RoBERTa model.
- The model predicts each utterance label using the current utterance together with preceding or following dialogue context.
- Starting from pretrained roberta-large, EmoBERTa applies a randomly initialized linear softmax layer to the final hidden state corresponding to [CLS].
- EmoBERTa prepends speaker names and separates past, current, and future utterance segments so RoBERTa can use speaker-aware context.
- The input-building procedure appends speaker-labeled utterances until the sequence reaches the maximum token budget, then removes the last appended or prepended item.
- Training minimizes cross-entropy plus L2 weight decay using adaptive gradient descent with warmup scheduling and mixed precision.
- The final model is selected by the highest weighted F1 score on the validation split rather than the lowest training loss.
4 Experiments
The experiments evaluate EmoBERTa on MELD and IEMOCAP, multimodal conversational datasets with imbalanced emotion classes. IEMOCAP has substantially longer dialogues, and its speaker identities are replaced with random names for evaluation.
- IEMOCAP has about 5 times more utterances per dialogue than MELD.
- MELD is a multimodal, multiparty dataset from Friends with seven emotions, evaluated using weighted F1 because its classes are highly imbalanced.
- IEMOCAP is a multimodal, dyadic dataset with six evaluated emotions and highly imbalanced classes, so weighted F1 is used.
- The study follows widely used train, validation, and test splits because IEMOCAP does not officially define them.
5 Results and Analysis
EmoBERTa outperforms other models on MELD and IEMOCAP, while analyses indicate that speaker information and attention to the current speaker support its predictions.
- 5.1 Quantitative Analysis: 65.61 weighted F1 on MELD and 68.57 on IEMOCAP place EmoBERTa above the best reported SOTA without modifying RoBERTa’s architecture.The reported scores are maximum weighted F1 values, while Table 2 reports mean values over five random seeds.
- 5.1 Quantitative Analysis: 65.07 weighted F1 on MELD and 64.02 on IEMOCAP without prepended speaker names indicate that encoding speaker information helps.
- 5.1 Quantitative Analysis: In IEMOCAP, using past utterances performed better than using both past and future utterances because sequence length limited how many contextual utterances could fit.The authors report a larger context-related gain for IEMOCAP than MELD, 12.48 versus 2.15, and attribute this to IEMOCAP’s more contextual nature.
- 5.2 Qualitative Analysis: In qualitative inspection, current-speaker tokens attended to interlocutor tokens in all 20 MELD and 20 IEMOCAP samples.
- 5.2 Qualitative Analysis: 100% of correctly classified MELD samples had last-layer <s> attention to the target speaker, compared with 60% of incorrect samples.The authors interpret this pattern as evidence that current-speaker information becomes increasingly important in higher layers.
6 Conclusion
The conclusion presents EmoBERTa as outperforming other ERC models and highlights attention analysis as a way to inspect which dialogue information supports classification.
- 6 Conclusion: EmoBERTa outperforms other models in emotion recognition in conversation and exposes attention coefficients for inspecting influential dialogue tokens and interlocutor names.
A.1 Training Details
Training used standard fixed settings with mixed-precision RoBERTa-large optimization and Optuna-based learning-rate selection.
- A.1 Training Details: Training ran for five epochs with L2 regularization of 0.01, no frozen weights, and a learning-rate schedule that increased for 20% then decreased for 80%.
- A.1 Training Details: Optuna searched five learning-rate trials between 1e-6 and 1e-4 using 10% training and validation subsets to minimize validation cross-entropy.
- A.1 Training Details: A full five-epoch training run took about 45 minutes on an NVIDIA Tesla V100 using mixed precision.The 16 GB GPU accommodated batches of 4 to 16 samples depending on input length.
A.2 IEMOCAP Speaker Names
Because IEMOCAP does not officially provide speaker names, the experiments assigned common American names to its actors.
- A.2 IEMOCAP Speaker Names: IEMOCAP actors were assigned the five most common male and female American names from the past century.
- A.2 IEMOCAP Speaker Names: The assigned female names were Mary, Patricia, Jennifer, Linda, and Elizabeth; male names were James, John, Robert, Michael, and William.
A.3 Qualitative Analysis on IEMOCAP
On IEMOCAP, last-layer <s> attention to the current target speaker appeared in only 20% of correctly classified samples and 10% of incorrectly classified samples, with unseen test speakers offered as a possible explanation.
- 20% of correctly classified IEMOCAP samples showed last-layer <s> attention to the current target speaker, versus 10% of incorrectly classified samples.
- The target-speaker attention behavior was less prevalent on IEMOCAP than the corresponding MELD analysis.
- The authors attribute the low IEMOCAP ratios to test-split speaker names, such as WILLIAM and ELIZABETH, being unseen during training.
- Figure 2 visualizes the IEMOCAP analysis with current-speaker attention and final-layer <s> attention highlighted across two examples.
- In an incorrectly classified example, the final-layer <s> token did not attend to the current speaker, paralleling the qualitative behavior observed in MELD.