Source-linked AI summary
Using millions of emoji occurrences to learn any-domain representations for detecting sentiment, emotion and sarcasm
Bjarke Felbo, Alan Mislove, Anders Søgaard, Iyad Rahwan, Sune Lehmann
TL;DR
NLP tasks often lack sufficient manually annotated data, motivating distant supervision from emotional expressions. The paper pretrains a single model by predicting diverse emoji labels and transfers its representations to target tasks, achieving state-of-the-art performance across eight benchmarks. The analyses identify emoji-label diversity as important for performance.
Problem
NLP tasks are limited by scarce manually annotated data, motivating distant supervision from emotional expressions.
Method
The paper pretrains a model on emoji prediction and transfers its learned representations using layer-wise fine-tuning, including the chain-thaw approach.
Results
The single pretrained model achieves state-of-the-art performance across eight benchmark datasets spanning sentiment, emotion, and sarcasm detection.
Takeaways & Limitations
Diverse emoji labels are important for learning transferable emotional representations, and the released pretrained model can support emotion-related NLP tasks.
Takeaways & Limitations
The architecture’s factors for outperforming a standard two-layer LSTM were not determined and were left for future work.
Abstract
from arXiv · showhide
NLP tasks are often limited by scarcity of manually annotated data. In social media sentiment analysis and related tasks, researchers have therefore used binarized emoticons and specific hashtags as forms of distant supervision. Our paper shows that by extending the distant supervision to a more diverse set of noisy labels, the models can learn richer representations. Through emoji prediction on a dataset of 1246 million tweets containing one of 64 common emojis we obtain state-of-the-art performance on 8 benchmark datasets within sentiment, emotion and sarcasm detection using a single pretrained model. Our analyses confirm that the diversity of our emotional labels yield a performance improvement over previous distant supervision approaches.
1 Introduction
The paper extends distant supervision from narrow emotional labels to diverse noisy labels, learning richer emotional representations that transfer across sentiment, emotion, and sarcasm benchmarks.
- Distant supervision on noisy labels often enables better performance on target tasks.
- Using a more diverse set of noisy labels helps models learn richer representations of emotional content.
- A single pretrained model’s learned representation generalizes across 5 domains.
- The model obtains improvements over state-of-the-art methods across emotion, sarcasm, and sentiment detection tasks.
2 Related work
Earlier work used emoticons, hashtags, and emojis as noisy labels, often with manually specified emotional categories. The paper contrasts this with an approach that captures diverse emoji usage without prior corpus knowledge.
- Prior research used emotional expressions as noisy labels, including binarized emoticons, hashtags, and emojis.
- Previous studies manually assigned emotional categories to expressions, drawing on theories such as Ekman’s and Plutchik’s emotion systems.
- Manual categorization is difficult and time-consuming for sophisticated emotional combinations and may misinterpret or omit usage details.
- The paper’s approach requires no prior corpus knowledge and captures diverse usage of 64 emoji types.
- Emoji-table embeddings require emojis at test time and do not capture changes in emoji meaning over time.
3 Method
The method pretrains DeepMoji by predicting diverse emoji labels from tweets, then transfers the learned representation to target tasks through layer-wise fine-tuning. The model combines bidirectional LSTMs with attention and skip-connections, while balancing emoji classes and expanding vocabulary for new domains.
- Pretraining data: Emoji prediction uses tweets containing emojis as noisy labels, with one saved training example per unique emoji type in a tweet.This preserves multiple emotional labels while keeping pretraining a single-label classification task.
- Pretraining data: The pretraining dataset is balanced by equally representing emojis in validation and test sets and upsampling the remaining training data.The balancing is intended to prevent the model from focusing only on the most frequent emojis.
- Model: DeepMoji uses 256-dimensional word embeddings, two bidirectional LSTM layers, and an attention layer with skip-connections.The attention mechanism weights words when constructing a text representation for the final Softmax classifier.
- Model: Attention scores form a probability distribution over words, whose weighted representations are summed into a text vector used for classification.The paper reports that adding attention and skip-connections improves transfer-learning capabilities.
- Transfer learning: Chain-thaw sequentially unfreezes and fine-tunes one layer at a time before jointly fine-tuning the entire model.Weights are reloaded to the best validation setting after each convergence point, and the method increases target-task accuracy at extra computational cost.
- Transfer learning: Chain-thaw can add up to 10000 training-set words to a target vocabulary with little risk of overfitting.This supports vocabulary expansion for new domains during transfer learning.
4 Experiments
The experiments evaluate emoji prediction and transfer learning across eight datasets spanning sentiment, emotion, and sarcasm. DeepMoji outperforms state-of-the-art systems across the benchmarks, including domains substantially different from its Twitter pretraining data.
- 4.1 Emoji prediction: 1.2 billion relevant tweets were filtered from 56.6 billion raw tweets, with duplicate storage by emoji producing a 1.6-billion-tweet pretraining dataset.Validation and test sets each contain 640K tweets, with 10K examples per emoji type.
- 4.1 Emoji prediction: 43.8% top-5 accuracy for the largest DeepMoji model exceeds fastText’s 36.2% on emoji prediction.The comparison attributes the difference to DeepMoji’s LSTM and attention layers, which capture word context beyond embeddings.
- 4.2 Benchmarking: The benchmark covers eight datasets across five domains and evaluates emotion and sarcasm with averaged F1, while sentiment uses accuracy.The datasets include emotion, sentiment, and Internet Argument Corpus sarcasm benchmarks.
- 4.2 Benchmarking: The emotion evaluation includes only Fear, Joy, and Sadness because the remaining emotions occur in less than 5% of observations.The authors identify data scarcity as a particular issue for emotion-analysis benchmarks.
- 4.2 Benchmarking: DeepMoji outperforms the state of the art across all benchmark datasets, while chain-thaw consistently gives the highest transfer-learning performance, often only slightly above or equal to the last-layer approach.Reported results are averaged across five runs.
- 4.2 Benchmarking: p < 0.001 on every benchmark dataset supports that the results are statistically significantly better than the state of the art.The significance test uses bootstrap testing with 10,000 samples.
5 Model Analysis
The analyses show that diverse emoji supervision improves transferable emotional representations, while architecture and pretraining contribute to performance across benchmark tasks. Additional experiments examine clustering, coverage, phrase modeling, and agreement with human ratings.
- Importance of emoji diversity: 433M tweets with eight positive/negative emojis formed a size-controlled comparison for testing label diversity.The reduced dataset was designed to isolate emoji-set diversity from dataset-size differences.
- Importance of emoji diversity: DeepMoji-PosNeg performed worse than DeepMoji across all eight benchmarks, indicating richer transfer representations from diverse emoji types.The comparison used the ‘last’ transfer-learning approach, which maps learned features to target classes.
- Model architecture: DeepMoji outperformed a standard 2-layer LSTM across all benchmark datasets, although both architectures performed equally on pretraining.The authors suggest the architectural advantage may be specific to transfer learning rather than supervised classification with ample data.
- Analyzing the effect of pretraining: Word coverage increased when vocabulary from pretraining was added, but word coverage alone may understate pretrained representation benefits.Pretraining words generally had thousands or millions of observations, unlike many words appearing once in small benchmark training sets.
- Analyzing the effect of pretraining: 63% versus 93% accuracy on SS-Youtube showed that DeepMoji’s phrase coverage contributed beyond pretrained word embeddings.The comparison used a fastText classifier that was nearly equivalent to using only DeepMoji’s embedding layer.
- Comparing with human-level agreement: 82.4% agreement with aggregate MTurk sentiment ratings exceeded the 76.1% agreement of a randomly selected human rater.The model was evaluated against the average of the other nine ratings for each tweet.
6 Conclusion
The paper concludes that emoji-rich social-media text can pretrain models to represent emotional content. It reports that emoji diversity matters for performance and releases the pretrained DeepMoji model for emotion-related NLP tasks.
- Millions of emoji-containing social-media texts can pretrain models to learn representations of emotional content.
- Comparisons with a model trained on a subset of emojis indicate that emoji-set diversity is important for method performance.
- The pretrained DeepMoji model is released for use in various emotion-related NLP tasks.
A.1 Preprocessing Emotion Datasets
The emotion datasets are converted or organized into categorical benchmarks using valence-arousal classes and established emotion labels. The appendix also describes ISEAR as a multinational self-report dataset.
- Olympic Games tweets are converted from multiple emotions into four low/high valence-arousal classes.A class is assigned when its average rater evaluation is at least 2.0, with Low=1, Medium=2, and High=3.
- ISEAR contains self-reported emotional experiences mapped to one of seven possible emotions.The dataset was created through interviews conducted in 37 countries by a large group of psychologists.
A.2 Pretraining as Regularization
Pretraining helps regularize target-task models, which otherwise overfit quickly. Chain-thaw adds further regularization by fine-tuning layers sequentially.
- Pretraining helps regularize the target-task model and prevents it from quickly overfitting.
- Chain-thaw further increases regularization by fine-tuning the model layer by layer.
A.3 Emoticon to Emoji mapping
The study maps emoticon-like supervision to a reduced emoji set to examine the importance of label diversity. Figure 5 presents the emojis selected for this experiment.
- A.3 Emoticon to Emoji mapping: The reduced pretraining subset contains tweets with one of 8 emojis similar to positive or negative emoticons.The positive emoticons include :) :-) :D and =), while the negative emoticons include :( and :-(.
- A.3 Emoticon to Emoji mapping: The selected emojis were identified from the dataset based on their similarity to the emoticons used in prior distant-supervision studies.These emojis were used to create the reduced subset for analyzing diverse emoji labels.
- A.3 Emoticon to Emoji mapping: Figure 5 shows the emojis used to study the importance of a diverse noisy label set.
A.4 Emoji Clustering
The model’s emoji predictions are analyzed through correlations and hierarchical clustering to examine how emotional categories organize. The resulting hierarchy is broadly stable across model versions.
- A.4 Emoji Clustering: Predictions on a 640K-tweet pretraining test set are used to compute a correlation matrix of predicted probabilities.
- A.4 Emoji Clustering: Hierarchical clustering with average linkage converts the prediction correlation matrix into the dendrogram shown in Figure 6.The dendrogram represents distances on the correlation matrix.
- A.4 Emoji Clustering: The overall clustering structure remains very stable across model versions, with only a few emojis changing places.
- A.4 Emoji Clustering: The dendrogram groups emojis into overall categories and subcategories based on emotional content.
- A.4 Emoji Clustering: Figure 7 displays the correlation matrix of model predictions on the pretraining test set.