Source-linked AI summary
GoEmotions: A Dataset of Fine-Grained Emotions
Dorottya Demszky, Dana Movshovitz-Attias, Jeongwoo Ko, Alan Cowen, Gaurav Nemade, Sujith Ravi
TL;DR
Emotion research needs large-scale, consistently labeled datasets with fine-grained taxonomies to support machine understanding of subtle affect. The paper introduces GoEmotions, analyzes annotation reliability, and evaluates transfer learning and a BERT baseline, finding strong dataset utility but substantial room for improvement.
Problem
Emotion expression is subtle and complex, while existing resources show annotation-quality gaps and lack a large-scale, consistently labeled dataset with a fine-grained taxonomy.
Method
The authors compile 58k Reddit comments labeled with 27 emotions or Neutral, analyze annotation reliability using PPCA, and fine-tune BERT for emotion classification.
Results
The dataset demonstrates reliable dissociation among all 27 emotion categories, while BERT reaches an average F1-score of .46 over the proposed taxonomy.
Takeaways & Limitations
GoEmotions generalizes across domains and emotion taxonomies, but fine-grained emotion classification remains not fully addressed by current state-of-the-art NLU models.
Takeaways & Limitations
The dataset is not representative of global diversity and contains biases, potentially problematic content, and labeling limitations associated with Reddit, filtering, and annotators.
Abstract
from arXiv · showhide
Understanding emotion expressed in language has a wide range of applications, from building empathetic chatbots to detecting harmful online behavior. Advancement in this area can be improved using large-scale datasets with a fine-grained typology, adaptable to multiple downstream tasks. We introduce GoEmotions, the largest manually annotated dataset of 58k English Reddit comments, labeled for 27 emotion categories or Neutral. We demonstrate the high quality of the annotations via Principal Preserved Component Analysis. We conduct transfer learning experiments with existing emotion benchmarks to show that our dataset generalizes well to other domains and different emotion taxonomies. Our BERT-based model achieves an average F1-score of .46 across our proposed taxonomy, leaving much room for improvement.
1 Introduction
GoEmotions addresses the need for a large, consistently labeled dataset with fine-grained emotion categories. The authors introduce the dataset, analyze annotation quality and structure, test transfer across domains and taxonomies, and establish BERT baselines.
- Existing emotion datasets are generally small and limited to coarse taxonomies, motivating a larger fine-grained resource.The paper identifies both dataset size and taxonomy coverage as limitations in prior resources.
- GoEmotions contains 58k manually annotated Reddit comments labeled with 27 emotion categories or Neutral.The taxonomy includes positive, negative, and ambiguous categories for nuanced emotion understanding.
- Principal Preserved Component Analysis supports reliable dissociation among all 27 emotion categories.The analysis is presented as evidence that the annotations are suitable for emotion classification.
- Hierarchical clustering finds intensity-related emotions close together and larger clusters aligned with sentiment categories.These relations may support grouping emotions into higher-level categories for downstream tasks.
- The BERT-base model achieves average F1-scores of .46 for the proposed taxonomy, .64 for six coarse categories, and .69 for sentiment.The reported results also indicate substantial room for improvement.
- Transfer experiments show that GoEmotions generalizes across emotion taxonomies and domains including tweets and personal narratives.The authors position the data as a baseline when specialized-domain annotations are limited.
2 Related Work
Prior emotion datasets span many domains but are often small, noisy, weakly labeled, or restricted to coarse taxonomies. GoEmotions responds with a larger manually annotated Reddit resource using multiple annotations for quality assurance.
- Most manually constructed emotion datasets are relatively small, while CrowdFlower contains 39k labeled examples and was found comparatively noisy.Other resources may instead use automatic weak labeling based on emotion-related hashtags.
- GoEmotions is manually built as the largest human-annotated dataset, with multiple annotations per example for quality assurance.The dataset uses Reddit comments as its source domain.
- Existing emotion datasets cover domains such as Twitter, news, dialogs, narratives, and self-reported experiences.The paper situates GoEmotions among resources with varied domains and annotation settings.
- Existing taxonomies commonly use six basic emotions or affective dimensions such as valence and arousal.The paper contrasts these approaches with newer efforts to represent a more complex emotion semantic space.
- BERT has achieved strong performance on emotion prediction, and the paper reports that it outperforms the authors’ biLSTM model.
3 GoEmotions
GoEmotions is a curated Reddit dataset designed to balance coverage across comments, sentiments, emotions, and subreddits before manual annotation. Its taxonomy and annotation procedure support multi-label, fine-grained emotion judgments.
- The dataset contains 58K Reddit comments labeled for one or more of 27 emotions or Neutral.Comments came from a Reddit dump spanning 2005 through January 2019, after removing deleted and non-English comments.
- The curation pipeline filtered offensive content and masked proper names and religion terms, while retaining vulgar comments as potentially informative for negative emotions.Raters viewed unmasked comments during rating.
- Comments were filtered to 3–30 tokens, while downsampling balanced comment lengths, sentiments, emotions, and subreddit representation.The remaining 315k comments from 482 subreddits were randomly sampled for annotation.
- The taxonomy was designed to maximize emotional coverage, represent kinds of expression, and limit overlap and the total number of labels.Psychology literature and pilot annotations informed the selection procedure.
- Three raters labeled each example, with two additional raters assigned when no emotion label received agreement.Raters could select multiple emotions or Neutral and could mark especially difficult examples.
- The interface grouped categories by sentiment and expression tendencies, while explicitly noting that these groupings were not clear-cut.
4 Data Analysis
The data analysis evaluates label agreement, emotion relationships, latent dimensions, and lexical correlates. It finds high annotation reliability, structured emotion relationships, and variation in how verbally explicit emotions are.
- 83% of examples have a single emotion label, and 94% have at least two raters agreeing on a single label.Neutral accounts for 26% of emotion labels and is excluded from subsequent semantic-space analyses.
- Emotion frequencies vary substantially: admiration is 30 times more frequent than grief despite balancing during data selection.The authors attribute this disparity to natural differences in emotion expression.
- Interrater correlation is preferred to Cohen’s kappa for interpretability and variable numbers of raters, while the two measures correlate at Pearson r = 0.85, p < 0.001.
- Interrater correlation is highest for gratitude, admiration, and amusement, and lowest for grief and nervousness.Frequency and agreement correlate but are not equivalent; fear and annoyance illustrate exceptions.
- Correlations show that intensity-related emotions are positively associated, whereas emotions with opposite sentiment are negatively correlated.Examples include annoyance–anger, joy–excitement, and nervousness–fear.
- Hierarchical clustering places intensity-related emotions near one another and larger clusters near sentiment categories.Ambiguous emotions such as surprise are reported as closer to positive than negative emotions.
- All 27 principal preserved components are highly significant, with Bonferroni-corrected p-values below 1.5e-6 for every dimension.The result suggests that the emotion categories are highly dissociable.
- Emotions with more strongly associated lexical tokens tend to have higher interrater correlation, while less lexically explicit emotions tend to have lower agreement.The authors suggest that verbally implicit emotions may require more context for interpretation.
5 Modeling
The paper presents a BERT-based baseline for fine-grained emotion classification, evaluates performance across taxonomies, and identifies substantial variation across emotions and models.
- Data Preparation: Filtering out labels selected by only one annotator retains 93% of the original data for the baseline experiments.The authors also release the 4K examples lacking annotator agreement because they may contain difficult or emotion-ambiguous cases.
- Results: BERT achieves an average F1-score of .46 on the full GoEmotions taxonomy, .64 on Ekman-grouped categories, and .69 on sentiment-grouped categories.Performance is higher when fine-grained categories are grouped into coarser taxonomies.
- Results: The BERT model performs best on emotions with overt lexical markers, including gratitude (.86), amusement (.8), and love (.78).It performs worst on lower-frequency emotions such as grief (0), relief (.15), and realization (.21).
- Error Analysis: The full-taxonomy model confuses less frequent emotions with more frequent emotions related in sentiment and intensity.Examples include grief with sadness, pride with admiration, and nervousness with fear.
- Results: The biLSTM baseline performs significantly worse than BERT, achieving average F1-scores of .41, .53, and .6 for the full, Ekman-grouped, and sentiment-grouped taxonomies.These results compare the two model classes across all three taxonomy levels.
6 Transfer Learning Experiments
Transfer-learning experiments test whether GoEmotions supports emotion classification in other domains and taxonomies, especially when target-domain labels are scarce. The results favor GoEmotions-based initialization over target-only training in low-data settings.
- Benchmark Datasets: The experiments use nine benchmark datasets varying in size, domain, quality, and taxonomy, while discussing three representative datasets in detail.The additional benchmark results show similar trends and are included in Appendix H.
- Experimental Setup: The experiments vary target-domain training sizes from 100, 200, 500, and 1000 examples to 80% of each dataset, using 10 random splits per size.Confidence intervals are based on repeated experiments using these splits.
- Experimental Setup: FREEZE first fine-tunes BERT on GoEmotions before training only the final layer on the target dataset, whereas NOFREEZE also updates the lower layers.The BASELINE fine-tunes BERT only on the target dataset.
- Results: The transfer-learning results suggest that GoEmotions generalizes across domains and taxonomies and can help when target-domain data or labeling resources are limited.Figure 3 reports average F1-scores across emotion categories with 95% confidence intervals from 10 runs on 10 random splits.
7 Conclusion
The paper concludes that GoEmotions is a large, fine-grained, manually annotated resource with reliable annotations and demonstrated transfer across domains and taxonomies. It also acknowledges dataset biases and limited representativeness as important boundaries on use.
- Contributions: The annotation analysis supports reliability for the full taxonomy, while transfer-learning experiments demonstrate generalizability across domains and taxonomies.The paper also provides a strong BERT baseline, but reports substantial room for future improvement.
- Limitations: The dataset is not representative of global diversity and contains biases associated with Reddit, filtering, identity-label assessment, and its all-native-English-speaking annotator pool.The disclaimer states that these factors likely affect labeling, precision, and recall for trained models.
B Taxonomy Selection & Data Collection
The taxonomy was developed through multiple pilot rounds that incorporated prior emotion categories and annotator suggestions, then removed rarely selected or difficult-to-distinguish emotions before final annotation.
- Taxonomy Construction: The taxonomy was selected through a multi-round process beginning with salient emotions from prior work and Ekman categories, plus an open field for annotator suggestions.The first pilot annotated 3K examples and the second annotated 2K new examples, with updates after each round.
- Taxonomy Construction: The authors removed emotions that were scarcely selected or had low interrater agreement because they were too similar to other emotions or difficult to detect from text.Removed categories included boredom, doubt, heartbroken, indifference, and calmness.
- Final Annotation: The final dataset was labeled in a third collection round using the revised taxonomy.The authors link the high final interrater agreement partly to considering interpretability during taxonomy construction.
C Cohen’s Kappa Values
The paper reports positive interrater agreement and examines how filtering and model-layer importance relate to the dataset and analyses.
- All Cohen’s kappa values are greater than 0, indicating agreement between raters.The kappa values also correlate highly with interrater correlation values.
- All BERT layers are similarly important for the task, with center of gravity = 6.19.
- Filtering removes labels selected by only a single annotator before model training and testing.
G Confusion Matrix
The multilabel confusion matrix counts true–predicted emotion co-occurrences, normalizes them by true-label rows, and reveals systematic confusions among related emotions.
- For each true label, the matrix increases counts for every predicted label, reflecting multilabel co-occurrence.For example, true joy and admiration with predicted joy and pride increments four matrix cells.
- Row normalization divides each true-emotion row by its total count to account for disparate label frequencies.
- The model tends to confuse emotions related in sentiment and intensity, including grief–sadness, pride–admiration, and nervousness–fear.
- Hierarchical clustering of the normalized confusion matrix produces clusters relatively similar to those in the dataset’s emotion taxonomy.This similarity holds even though training uses only labels with annotator agreement.
H Transfer Learning Results
Transfer learning from GoEmotions improves performance across emotion benchmarks, especially with limited target-domain data, although the preferred fine-tuning setup varies with data size and dataset.
- Transfer learning helps across all evaluated datasets, especially when target-domain training data are limited.
- FREEZE significantly outperforms both BASELINE and NOFREEZE on CrowdFlower and Electoral Tweets for all training sizes except “max”.
- For other datasets, FREEZE tends to help only through a couple hundred examples, while NOFREEZE tends to outperform BASELINE with 500–1000 examples.
- With more target-domain data, NOFREEZE and BASELINE performance becomes closer.