Source-linked AI summary
EmotionLines: An Emotion Corpus of Multi-Party Conversations
Sheng-Yeh Chen, Chao-Chun Hsu, Chuan-Chun Kuo, Ting-Hao, Huang, Lun-Wei Ku
TL;DR
Existing textual emotion datasets usually omit dialogue-wide contextual flow, despite the importance of context for interpreting utterances. EmotionLines addresses this gap with 2,000 annotated dialogues from television scripts and private chats, and contextual baselines show improved weighted accuracy when context is used.
Problem
Most textual emotion datasets label isolated units rather than all utterances in dialogues, limiting analysis of contextual emotion flow.
Method
EmotionLines labels every utterance in Friends scripts and private Facebook Messenger dialogues with one of seven emotions using five Amazon MTurkers.
Results
Weighted accuracy improves from 59.2% to 63.9% on Friends and from 71.5% to 77.4% on EmotionPush when contextual information is used.
Takeaways & Limitations
EmotionLines provides utterance-level sequential emotion labels and baselines for dialogue emotion recognition.
Takeaways & Limitations
EmotionPush conversations involve private participants and required consent and de-identification to protect identities.
Abstract
from arXiv · showhide
Feeling emotion is a critical characteristic to distinguish people from machines. Among all the multi-modal resources for emotion detection, textual datasets are those containing the least additional information in addition to semantics, and hence are adopted widely for testing the developed systems. However, most of the textual emotional datasets consist of emotion labels of only individual words, sentences or documents, which makes it challenging to discuss the contextual flow of emotions. In this paper, we introduce EmotionLines, the first dataset with emotions labeling on all utterances in each dialogue only based on their textual content. Dialogues in EmotionLines are collected from Friends TV scripts and private Facebook messenger dialogues. Then one of seven emotions, six Ekman's basic emotions plus the neutral emotion, is labeled on each utterance by 5 Amazon MTurkers. A total of 29,245 utterances from 2,000 dialogues are labeled in EmotionLines. We also provide several strong baselines for emotion detection models on EmotionLines in this paper.
1. Introduction
EmotionLines addresses the lack of utterance-level emotion labels that preserve conversational context by introducing a textual dialogue dataset annotated across complete dialogues.
- Chit-chat systems need emotion detection to capture conversation flow and speaker feelings.
- Most available textual emotion datasets label isolated words, sentences, posts, or documents rather than every utterance in a dialogue.
- The same utterance, such as “Okay!”, can express different emotions depending on its conversational context.
- IEMOCAP provides utterance-level labels but uses actors performing emotions and annotators viewing videos, risking overacting and reliance on non-textual cues.
- EmotionLines labels every utterance in textual dialogues from Friends scripts and private human-to-human chat logs, and provides dialogue emotion-detection baselines.
2. Related Work
Prior text-based emotion-recognition work includes keyword-based, learning-based, and hybrid approaches, while this paper reports CNN and contextual LSTM baselines on EmotionLines.
- Sentiment analysis of user-generated content has attracted research and business attention because it can capture people’s opinions.
- Text-based emotion-recognition methods have been categorized as keyword-based, learning-based, or hybrid recommendation approaches.
- The paper reports CNN and contextual LSTM performance as baselines on the proposed EmotionLines dataset.
3. Corpus
EmotionLines combines balanced sampling of Friends scripts and private Facebook Messenger dialogues with contextual seven-label annotation and privacy protections.
- The corpus contains 1,000 dialogues from Friends seasons 1–9 and 1,000 dialogues from EmotionPush Facebook Messenger chats.
- Friends scenes are treated as dialogues, categorized by utterance count, and sampled evenly across four dialogue-length ranges.
- EmotionPush messages less than 300 seconds apart are grouped into dialogues assumed not to exceed 30 minutes, then sampled using the Friends procedure.
- Each utterance is labeled in whole-dialogue context with one of seven categories: neutral, joy, sadness, fear, anger, surprise, or disgust.
- Five workers annotate each HIT, with the highest-vote emotion assigned as the gold label; highly divergent votes are assigned to the non-neutral category.
- Private-chat data were consented for research and de-identified through named-entity masking, manual cleaning, and rewriting of participants’ friends’ messages.
4. Analysis
The corpus analysis compares emotion and length distributions across its two sources, evaluates annotator agreement, and preserves complete dialogues in dataset splits.
- Joy and surprise are more frequent than other non-neutral emotions, while EmotionPush has a more skewed label distribution than Friends.
- Fleiss’ kappa scores above 0.33 for both Friends and EmotionPush indicate a solid basis for the subjective labeling task.
- Dataset splits are made by dialogue rather than utterance to preserve dialogue completeness across training, development, and testing sets.
5. Experiments
Experiments compare CNN and CNN-BiLSTM models for dialogue emotion detection using weighted and unweighted accuracy. Adding contextual LSTM information improves weighted accuracy on both datasets.
- Modeling on the Whole Dialogue: The contextual LSTM receives the sequence of utterance representations for each dialogue and feeds each cell output to dense and softmax layers.Utterance representations are formed using word embeddings, convolution, and max-pooling before contextual sequence modeling.
- Experimental setup: CNN and CNN-BiLSTM models are evaluated on EmotionLines using weighted accuracy (WA) and unweighted accuracy (UWA).The experiments report results on the Friends and EmotionPush datasets.
- Performance on EmotionLines: Weighted accuracy rises from 59.2% to 63.9% on Friends when using CNN-BiLSTM instead of CNN.The reported comparison attributes the improvement to incorporating contextual information.
- Performance on EmotionLines: Weighted accuracy rises from 71.5% to 77.4% on EmotionPush when using CNN-BiLSTM instead of CNN.These results are from experiments on raw, not-yet-de-identified data; updated results were to be provided later.
6. Conclusion and Future Work
EmotionLines provides a seven-label emotion dialogue dataset with strong baselines for dialogue emotion detection, while identifying label imbalance as future work.
- EmotionLines contains text content for each utterance annotated with one of seven emotion categories.
- Experiments show that contextual information benefits dialogue emotion recognition.
- Weighted accuracy baselines reach 63.9% on Friends and 77.4% on EmotionPush.
- Future work targets the imbalanced label distribution by collecting more examples of minority emotions such as fear and sadness.