Source-linked AI summary
emoji2vec: Learning Emoji Representations from their Description
Ben Eisner, Tim Rocktäschel, Isabelle Augenstein, Matko Bošnjak, Sebastian Riedel
TL;DR
Social-media NLP systems use pre-trained word embeddings, but these resources lack complete emoji coverage despite increasing emoji use. The paper introduces emoji2vec, which learns emoji embeddings from Unicode descriptions and evaluates them in downstream sentiment analysis. Despite using much less data and a simpler model, emoji2vec outperforms prior tweet-trained emoji embeddings on Twitter sentiment analysis.
Problem
Pre-trained word-embedding resources used in social NLP contain few or no emoji representations, despite the increasing use of emojis in social media.
Method
emoji2vec learns trainable emoji embeddings from Unicode emoji descriptions and keyword phrases, aligning them with 300-dimensional Google News word2vec embeddings.
Results
Despite being simpler and trained on much less data, emoji2vec outperforms Barbieri et al. (2016) on Twitter sentiment analysis.
Takeaways & Limitations
The resulting embeddings can augment word2vec-based downstream applications, including social NLP tasks where emojis are used frequently.
Takeaways & Limitations
Training on English-language definitions while ignoring temporal and context-dependent meanings may fail to capture the full semantic characteristics of emojis.
Abstract
from arXiv · showhide
Many current natural language processing applications for social media rely on representation learning and utilize pre-trained word embeddings. There currently exist several publicly-available, pre-trained sets of word embeddings, but they contain few or no emoji representations even as emoji usage in social media has increased. In this paper we release emoji2vec, pre-trained embeddings for all Unicode emoji which are learned from their description in the Unicode emoji standard. The resulting emoji embeddings can be readily used in downstream social natural language processing applications alongside word2vec. We demonstrate, for the downstream task of sentiment analysis, that emoji embeddings learned from short descriptions outperforms a skip-gram model trained on a large collection of tweets, while avoiding the need for contexts in which emoji need to appear frequently in order to estimate a representation.
1 Introduction
Emoji use has grown sharply in social media, while common pre-trained word embeddings lack complete Unicode emoji representations. The paper introduces emoji2vec to address this gap and evaluates its usefulness for Twitter sentiment analysis.
- Emoji usage in social media increased substantially, with emojis appearing in over 10% of Twitter posts and over 50% of Instagram text.
- Emoji research finds that emojis express relationally useful conversational roles and have culturally and contextually variable meanings.
- Many social-media NLP systems rely on pre-trained word embeddings from resources such as word2vec or GloVe.
- Neither common resource contains a complete set of Unicode emoji representations, motivating the addition of robust emoji embeddings.
- The paper releases emoji2vec and evaluates its representations on Twitter sentiment analysis, emoji analogies, and visualization of the embedding space.
2 Related Work
Prior emoji embedding work used distributional contexts, whereas this approach learns representations from emoji descriptions and keyword phrases. The authors emphasize robust coverage of infrequently used emojis and competitive sentiment-analysis accuracy with less data.
- Earlier emoji embedding efforts trained skip-gram-like representations on large Instagram or Twitter corpora.
- The approach estimates emoji representations directly from descriptions, providing robust representations for supported symbols, including infrequently used ones.
- The training data includes emoji names and keyword phrases, illustrated by business, man, and suit keywords for an uncommon emoji.
- The method trains on only a few thousand descriptions rather than millions of tweets and still obtains higher accuracy on Twitter sentiment analysis.
- Related description-based representation work builds word and concept vectors from dictionary descriptions and keyword phrases.
- Because training uses English-language definitions and ignores temporal definitions, the method may miss cultural and temporal aspects of emoji meaning.
3 Method
emoji2vec maps emoji symbols into the same space as 300-dimensional Google News word2vec embeddings by matching trainable emoji vectors to description vectors. It uses a logistic objective with sampled mismatches and lightweight optimization.
- The method maps emoji symbols into the 300-dimensional Google News word2vec space and creates 6088 descriptions for 1661 emoji symbols.
- For each emoji description, the model sums the Google News word2vec vectors of its words to form the description representation.
- 3.1 Model: The model learns a trainable vector for each emoji and scores description matches using the sigmoid of the representations’ dot product.
- 3.1 Model: Training uses logistic loss, with labels indicating whether a description is valid for an emoji.
- 3.2 Optimization: Randomly mismatched emoji-description pairs provide negative instances because the original training set contains no invalid descriptions.
- 3.2 Optimization: The model uses TensorFlow, Adam-based stochastic gradient descent, early stopping, and takes less than 3 minutes to train on a 2013 MacBook Pro.
4 Evaluation
The evaluation combines intrinsic emoji-description classification, Twitter sentiment analysis, and qualitative analyses of embedding geometry and analogies. The learned representations achieve strong description matching, improve sentiment classification especially for emoji-containing tweets, and form meaningful semantic clusters.
- The evaluation covers emoji-description classification, Twitter sentiment analysis, t-SNE visualization, and emoji analogy examples.
- 4.1 Emoji-Description Classification: 85.5% accuracy and 0.933 area-under-the-curve measure valid versus invalid emoji-description pairs.The classifier uses a 0.5 threshold for accuracy, while varying the threshold produces the ROC curve.
- 4.2 Sentiment Analysis on Tweets: Augmenting word2vec with emoji embeddings improves overall Twitter sentiment accuracy and substantially improves performance on tweets containing emojis.emoji2vec generally outperforms Barbieri et al. (2016) despite using much less data and a simpler model.
- 4.3 t-SNE Visualization: t-SNE projections show clusters for flags, smileys, fruits, astrological signs, animals, vehicles, and families.The projection preserves relative distances while reducing the embeddings to two dimensions.
- 4.4 Analogy Task: In emoji analogy examples, the seemingly correct emoji is often among the three closest vectors but is not always ranked first.For each query, the authors retrieve the five closest emojis.
5 Conclusion
The paper releases emoji2vec embeddings trained from Unicode descriptions and reports that they outperform a tweet-trained alternative despite using much less data. The approach captures several semantic clusters but does not capture context-dependent meanings and has difficulty disentangling number symbols.
- emoji2vec provides embeddings for 1661 emoji symbols, trained directly from Unicode descriptions rather than tweet contexts.The embeddings can augment existing word2vec representations for downstream applications.
- Despite a simpler model and much less training data, emoji2vec outperforms Barbieri et al. (2016) on Twitter sentiment analysis.
- The approach is not restricted to emoji symbols because it directly processes Unicode descriptions.The authors propose investigating other Unicode symbol embeddings.
- The learned space contains semantic clusters, but number symbols are not properly disentangled.
- English-language definitions and omission of temporal and culturally coopted meanings may leave emoji semantics incompletely represented.The authors identify context-dependent meanings such as sarcasm and cultural appropriation as areas for future work.
Data Release and Reproducibility
The released emoji2vec package includes pre-trained embeddings, training data, and code, with a format compatible with word2vec tooling.
- Pre-trained embeddings, training data, and code are released in the emoji2vec repository.
- The emoji2vec format is compatible with word2vec and can be loaded into gensim or similar libraries.