Source-linked AI summary
Emotion Detection on TV Show Transcripts with Sequence-based Convolutional Neural Networks
Sayyed M. Zahiri, Jinho D. Choi
TL;DR
Text-based emotion detection remains difficult and lacks comprehensive annotated dialogue datasets. This paper introduces a seven-emotion Friends corpus and attentive sequence-based CNN models, which outperform the base CNN in the reported experiments. The corpus remains limited by subjective annotation agreement and its relatively small size for more complex models.
Problem
Text-based emotion detection is challenging because emotions are not necessarily conveyed explicitly in text, and few annotated datasets are publicly available.
Method
The paper creates a crowdsourced Friends dialogue corpus and introduces sequence-based CNN models with attention that use information from previous utterances.
Results
The proposed models outperformed the base CNN in the experimental results.
Takeaways & Limitations
The corpus and attentive SCNN models provide a basis for studying emotion detection on consecutive utterances in multiparty dialogue.
Takeaways & Limitations
Annotation agreement was low because emotion judgments were subjective, and the corpus was relatively small for tuning more complex models.
Abstract
from arXiv · showhide
While there have been significant advances in detecting emotions from speech and image recognition, emotion detection on text is still under-explored and remained as an active research field. This paper introduces a corpus for text-based emotion detection on multiparty dialogue as well as deep neural models that outperform the existing approaches for document classification. We first present a new corpus that provides annotation of seven emotions on consecutive utterances in dialogues extracted from the show, Friends. We then suggest four types of sequence-based convolutional neural network models with attention that leverage the sequence information encapsulated in dialogue. Our best model shows the accuracies of 37.9% and 54% for fine- and coarse-grained emotions, respectively. Given the difficulty of this task, this is promising.
1 Introduction
The paper addresses the difficulty and limited data for text-based emotion detection by introducing a multiparty-dialogue corpus and sequence-aware CNN models with attention.
- Text-based emotion detection is challenging because emotions are not necessarily conveyed explicitly in text.
- The study responds to limited annotated datasets by creating a Friends transcript corpus with fine-grained emotion annotations.
- The corpus annotates consecutive multiparty-dialogue utterances with seven emotions and preserves dialogue sequences for sequence classification.
- The proposed models are sequence-based convolutional neural networks with attention that facilitate dependencies among utterances.
- The paper evaluates SCNN models through quantitative and qualitative analyses against basic CNN models and attention mechanisms.
2 Related Work
Prior text-emotion research uses keyword-based, learning-based, and hybrid approaches, while related dialogue research addresses state tracking and dialogue acts; attention has also gained prominence in NLP.
- Text emotion detection commonly uses keyword-based, learning-based, or hybrid methods.
- Learning-based approaches include supervised classification from training examples and unsupervised inference from statistical semantic dependencies.
- Hybrid systems combine keyword-based and learning methods but tend to have complicated architectures for replication.
- Related dialogue research includes CNN-based dialogue tracking, topic models for dialogue states, and statistical dialogue-act modeling.
- Attention mechanisms have become prominent in NLP, including machine translation and question answering.
3 Corpus
The paper builds a crowdsourced seven-emotion corpus from Friends transcripts and analyzes its annotation quality, label distributions, and dialogue emotion dynamics. It also uses voting and ranking to assign gold labels amid disagreement.
- Corpus construction: The corpus contains 97 episodes, 897 scenes, and 12,606 utterances from Friends seasons one through four, each labeled with one of seven emotions.The labels are sad, mad, scared, powerful, peaceful, joyful, and neutral.
- Crowdsourcing: Each utterance was annotated by four Amazon Mechanical Turk workers who selected the most relevant emotion within its scene.Scenes were grouped into four batches according to utterance count and assigned different budgets.
- Inter-annotator agreement: Kappa scores were around 14%, reflecting subjective judgments and the limitations of text-only annotation without speech or images.The authors suggest multimodal data could produce higher agreement.
- Inter-annotator agreement: With four annotators, 85% of annotations had some pairwise agreement, compared with over 70% having none with only two annotators.The authors interpret this as evidence that adding annotators can improve annotation quality.
- Voting and ranking: A voting and ranking scheme deterministically assigned gold labels to 75.5% of the dataset, using majority votes first and annotator least absolute error afterward.The remaining folds used the annotator with minimum LAE as the gold-label source.
- Corpus analysis: Neutral and joyful together exceed 50% of the corpus, while positive, negative, and neutral comprise about 40%, 30%, and 30% in the coarse-grained scheme.Complete agreement is only around 1% for peaceful and powerful, which are often confused with neutral.
- Corpus analysis: Within scenes, a speaker’s current emotion is often affected by that speaker’s previous emotions and by other participants’ preceding emotions.Figure 1 tracks the six main characters’ emotions across utterance positions, with white regions indicating no speech.
- Corpus analysis: The annotation confusion matrix shows the most confusion for dominant neutral and joyful labels, while minor emotions such as sad, powerful, and peaceful show stronger diagonal agreement.Each matrix cell contains a raw count, with rows representing voting-scheme labels and columns representing annotator selections.
4 Sequence-Based Convolutional Neural Networks (SCNN)
The paper introduces sequence-based CNNs that combine current-utterance features with dialogue history, using convolution and attention to model sequential dependencies for emotion classification.
- 4 Sequence-Based Convolutional Neural Networks (SCNN): SCNNs preserve dialogue sequence information by combining the current utterance with previous utterances for classification.The corpus supports sequence classification because utterance order is preserved.
- 4.1 Sequence Unification: Concatenation: The concatenation model joins dense feature vectors from the current and k-1 previous utterances before applying one-dimensional convolution.The convolution fuses current-utterance features with features associated with prior utterances.
- 4.2 Sequence Unification: Convolution: SCNNv uses separate two-dimensional convolutions for current-utterance features and the sequence of previous feature vectors.The two convolution outputs are later combined for classification.
- 4.2 Sequence Unification: Convolution: The convolution-based models concatenate convolution outputs, apply one-dimensional convolution to create fused features, and classify seven emotions with softmax.The fused vector is passed to a softmax layer for emotion classification.
- 4.3 Attention Mechanism: Attention weights current-utterance features according to their relevance given unified feature vectors from previous utterances.The attentive SCNN variants use past feature vectors to learn which current features deserve greater emphasis.
- 4.3 Attention Mechanism: The attention mechanism accepts a dynamically generated representation of previous hidden layers alongside the dense representation of the current utterance.This differs from fixed-size attention inputs described generally for other models.
5 Experiments
Experiments evaluate sequence-based CNN variants with attention against CNN and RNN-CNN baselines on the Friends emotion corpus. The attentive SCNNc model performed best overall, while gains were strongest in shorter scenes and performance remained limited in very long scenes.
- Experimental setup: The corpus is partitioned into training, development, and evaluation sets containing 77, 11, and 9 episodes, respectively.Episodes remain intact within one split to preserve dialogue sequence information.
- Models and baselines: The study compares SCNNc, SCNNv, attentive variants, base CNN, and RNN-CNN using between one and five previous utterances.RNN-CNN uses an LSTM over CNN-generated utterance features.
- Sequence length: The best development-set sequence lengths were three previous utterances for SCNNc and five for SCNNv.Experiments with more than five previous utterances produced no significant improvement.
- Overall results: SCNNc outperformed all other listed models, and attention improved SCNNv relative to base CNN.The paper reports evaluation using accuracy and F1-score for seven- and three-emotion settings.
- Error analysis: Most emotions were confused with neutral; peaceful had the highest neutral confusion rate at 30%, while joyful and powerful had rates of 13.8% and 20.4%.These confusion rates are reported for the best SCNNc model.
- Scene-length analysis: SCNNc with three previous utterances produced roughly a 4% F1-score boost over base CNN in scenes containing [5,15) utterances.In very long scenes with more speakers and speaker transitions, the proposed models did not significantly outperform base CNN.
- Attention analysis: Attention generally weighted the current utterance most heavily, while neutral current utterances received relatively small and similar weights for previous utterances.The heat map covers the first eight consecutive utterances and three previous utterances at each time step.
6 Conclusion
The paper introduces a Friends dialogue corpus for emotion detection and attentive sequence-based SCNN models. It reports that the proposed models outperform base CNN, while identifying annotation quality and corpus size as future improvement targets.
- Conclusion: The paper introduces a corpus for emotion detection from spoken dialogues and proposes attentive SCNN models that incorporate sequence information.The corpus is drawn from Friends transcripts.
- Conclusion: Experimental results showed that the proposed models outperformed the base CNN.The conclusion states this as the main experimental finding.
- Future work: Future work will add annotators, evaluate additional attention mechanisms, and expand the corpus with more Friends seasons.These plans target corpus quality, model evaluation, and dataset size.