Source-linked AI summary
Are Emojis Predictable?
Francesco Barbieri, Miguel Ballesteros, Horacio Saggion
TL;DR
The relation between words and emojis has received little NLP study, despite emojis' widespread use in social media. The paper trains bidirectional LSTMs to predict emojis evoked by tweets. The models outperform two baselines and human annotators on the task.
Problem
The relation between words and emojis remains virtually unexplored from an NLP standpoint, despite emojis' widespread use in social media.
Source: §1 · p. 1Method
The paper uses bidirectional LSTMs with word and character representations to predict the emoji evoked by a tweet.
Source: §1 · p. 1, §3.1 · p. 2Results
BLSTMs outperform bag-of-words and semantic-vector baselines and human annotators; char-BLSTM scores F1 of 0.65 versus 0.50 for humans.
Source: §1 · p. 1, §4.2 · p. 5Takeaways & Limitations
The results suggest an important relation between word sequences and emojis, and that LSTMs can generalize emoji usage better than humans on this prediction task.
Source: §5 · p. 5
Abstract
from arXiv · showhide
Emojis are ideograms which are naturally combined with plain text to visually complement or condense the meaning of a message. Despite being widely used in social media, their underlying semantics have received little attention from a Natural Language Processing standpoint. In this paper, we investigate the relation between words and emojis, studying the novel task of predicting which emojis are evoked by text-based tweet messages. We train several models based on Long Short-Term Memory networks (LSTMs) in this task. Our experimental results show that our neural model outperforms two baselines as well as humans solving the same task, suggesting that computational models are able to better capture the underlying semantics of emojis.
1 Introduction
Although emojis are widely used alongside text in online communication, their relationship with text-based messages has received little NLP study. This paper investigates that relationship by predicting emojis evoked by tweets, and reports that its BLSTMs outperform baselines and human annotators.
1 Introduction
- The paper investigates the largely unexplored relation between words and emojis by predicting which emojis text-based tweets evoke.Earlier NLP work had focused on emoji semantics, usage, or sentiment, rather than the interplay between text and emojis.Source: §1 · p. 1
- Emoji ambiguity motivates studying whether an artificial agent can recognize emoji use in spontaneous conversation.The authors suggest emoji prediction could contribute to natural language understanding and tasks such as sentiment analysis and social-media retrieval.Source: §1 · p. 1
- BLSTMs with word and character representations outperform bag-of-words, semantic-vector, and human baselines on emoji prediction.Source: §1 · p. 1
2 Dataset and Task
The task uses tweets labeled by their single included emoji, drawn from a geographically and temporally bounded dataset of frequent-emoji tweets.
2 Dataset and Task
- The final dataset contains 584,600 tweets with exactly one of the 20 most frequent emojis, selected from 40 million retrieved tweets.The source tweets were geo-localized in the United States and posted between October 2015 and May 2016.Source: §2 · p. 2
- The study also uses subsets containing the 10 most frequent emojis (502,700 tweets) and the 5 most frequent emojis (341,500 tweets).Source: §2 · p. 2
- For the prediction task, the emoji is removed from each tweet's token sequence and used as the label for training and testing.Source: §2 · p. 2
3 Models
The models predict a tweet's emoji using sequence-aware bidirectional LSTMs and compare them with two logistic-regression baselines that do not use word order.
3 Models
- The model section contrasts recurrent neural networks, which use word sequences and the full context, with two baselines.Source: §3 · p. 2
3.1 Bi-Directional LSTMs
- The bidirectional LSTM combines forward and backward message encodings, then uses the resulting representation to predict emoji probabilities.Word embeddings may be learned with the model or combined with pretrained vectors; character-based representations use bidirectional LSTMs to capture orthographic similarities between words.Source: §3.1 · p. 2
3.2 Baselines
- The baselines ignore word order; the semantic-vector baseline represents tweets using pretrained Twitter vectors.Source: §3.2 · p. 3
3.2.1 Bag of Words
- The bag-of-words baseline uses TF-IDF token features, including punctuation, with L2-regularized logistic regression.Source: §3.2.1 · p. 3
3.2.2 Skip-Gram Vector Average
- The AVG baseline averages Skip-gram token vectors trained on 65 million tweets, then classifies the message representation with L2-regularized logistic regression.Source: §3.2.2 · p. 3
4 Experiments and Evaluation
The experiments compare neural and baseline systems across emoji-prediction settings, then compare the best system with human annotators. Neural models outperform the baselines, and char-BLSTM surpasses humans on the five-emoji task.
4.1 First Experiment
- Neural models outperform the baselines across all experimental setups, with pretrained char-BLSTM the most accurate system.The bidirectional LSTMs outperform BOW and AVG; the character-based model captures orthographic variants, while pretrained vectors provide semantic knowledge.Source: Table 2 · p. 3, §4.1 · pp. 3–4
- The ranking measure averages how many emojis score above the gold emoji; a moderate ranking can coexist with an emoji being almost never predicted.One such emoji has a ranking of 4.69; frequency also affects ranking, while F-measure does not appear to depend on frequency.Source: §4.1 · p. 4
- Specific vocabulary can help predict the blue-heart emoji, while the Christmas-tree example shows a rare emoji can be recognized when its text includes “Christmas.”The Christmas tree appears only three times in the test set; the model recognizes it twice and misses a tweet without obvious clues.Source: §4.1 · p. 4
4.2 Second Experiment
- F1 was 0.65 for char-BLSTM versus 0.50 for humans on the five most frequent emojis.The comparison used 1,000 tweets, with annotators asked to predict the emoji after reading each tweet with its emoji removed.Source: §4.2 · pp. 4–5, Table 4 · p. 5
- Humans chose the laughing emoji 208 times although it occurred 100 times in the test set, whereas the char-BLSTM’s doubtful predictions favored it as more probable.The authors suggest the model may encode information about the probabilities of the two emojis.Source: §4.2 · p. 5
5 Conclusions
The LSTMs outperform humans on emoji prediction, and the authors propose extending the model to predict multiple emojis and use emoji position in tweets.
5 Conclusions
- LSTMs outperform humans on emoji prediction, suggesting automatic systems generalize emoji usage better than people.The authors also describe a unique relation between word sequences and emojis.Source: §5 · p. 5
- Emojis are used differently across communities, while their use and semantics remain little understood.The paper relates words and emojis through the task of predicting the emoji evoked by a tweet.Source: §5 · p. 5
- Future work will extend prediction to multiple emojis per tweet and explore emoji position, since nearby words may help prediction.Source: §5 · p. 5