Source-linked AI summary
Tweet2Vec: Character-Based Distributed Representations for Social Media
Bhuwan Dhingra, Zhong Zhou, Dylan Fitzpatrick, Michael Muehl, William W. Cohen
TL;DR
Social-media text contains rare words, spelling variation, slang, and special characters that challenge word-level representations. Tweet2vec addresses this by composing whole-tweet embeddings from character sequences and training them to predict user-associated hashtags. It outperforms a word-level baseline, especially on posts with many rare or out-of-vocabulary words, while also handling spelling, segmentation, emojis, and special characters.
Problem
Rare words, misspellings, slang, emojis, and other unusual social-media sequences make word-level representations difficult to use effectively.
Method
Tweet2vec uses character inputs and a bidirectional GRU encoder to produce whole-tweet embeddings supervised by hashtag prediction.
Results
Tweet2vec outperforms the word-based approach, doing significantly better when input posts contain many rare words.
Takeaways & Limitations
The character encoder is robust to word-segmentation errors, spelling mistakes, emojis, and special characters while remaining comparable for in-vocabulary tokens.
Abstract
from arXiv · showhide
Text from social media provides a set of challenges that can cause traditional NLP approaches to fail. Informal language, spelling errors, abbreviations, and special characters are all commonplace in these posts, leading to a prohibitively large vocabulary size for word-level approaches. We propose a character composition model, tweet2vec, which finds vector-space representations of whole tweets by learning complex, non-local dependencies in character sequences. The proposed model outperforms a word-level baseline at predicting user-annotated hashtags associated with the posts, doing significantly better when the input contains many out-of-vocabulary words or unusual character sequences. Our tweet2vec encoder is publicly available.
1 Introduction
Tweet2vec learns tweet representations by composing characters, targeting social media’s rare words, misspellings, slang, emojis, and other unusual sequences. It uses hashtag supervision and improves hashtag prediction over a word-level baseline, especially for difficult inputs.
- Motivation: Rare words and sequences make social-media representation learning difficult because most vocabulary types and token sequences are infrequent or unseen.Neural compositional models address this by assigning non-zero probability to unseen sequences.
- Approach: Tweet2vec composes tweets from constituent characters to generalize to out-of-vocabulary words and unseen test-time sequences.The approach models long character sequences, including whitespace, rather than relying only on word types.
- Motivation: Twitter’s slang, misspellings, similar token forms, emojis, and special characters make it a useful test bed for character-based models.The training set contained about 0.92 million unique word types across 2 million tweets.
- Advantages: Character-based methods can be language-independent and avoid NLP preprocessing such as word segmentation.These properties complement robustness to noisy social-media text.
- Training objective: The model trains on hashtag prediction because posts sharing hashtags are assumed to have nearby embeddings.The objective maximizes cross-entropy for predicting a post’s hashtags from its latent representation.
- Model: A Bi-GRU processes the full character sequence in both directions, combines the final states into a tweet embedding, and predicts hashtags through softmax.Compared with a word-level baseline, the model improves held-out hashtag prediction.
2 Related Work
Prior work established distributed word and character representations across NLP tasks, but word-based methods require large lookup tables and struggle with noisy social-media text. Tweet2vec instead composes whole-tweet representations directly from characters without morpheme parsing or word lookup tables.
- Word-based representations: Word-vector methods extend to larger text units but require storing extremely large tables of vectors for word types.This creates a scalability problem for representation learning over broad vocabularies.
- Character models: Character composition can compact word lookup tables and scale better, while tweet2vec extends the idea from word embeddings to entire tweet representations.The related model uses bidirectional LSTMs; tweet2vec uses characters to represent complete tweets.
- Applications: Character models had already been applied to named entity recognition, part-of-speech tagging, text classification, and language modeling.These applications motivate extending character composition to social-media representation learning.
- Alternatives: Morpheme-based rare-word representations improve over word models but require a morpheme parser and still involve a large morpheme space.Those requirements are poorly matched to noisy Twitter text.
- Hashtag supervision: Earlier hashtag-prediction work learned text embeddings from word lookup tables and showed that hashtag supervision can support another task.Tweet2vec retains hashtag supervision while replacing word composition with character composition.
3 Tweet2Vec
Tweet2vec encodes a tweet’s character stream with forward and backward GRUs, combines their terminal states into one embedding, and predicts hashtag probabilities with a softmax layer. The design converts characters into contextual sequence representations before supervised hashtag prediction.
- Character input: The encoder processes character vectors with forward and backward GRUs, with the backward unit reading the sequence in reverse.Characters begin as one-hot inputs over an alphabet and are projected into a character vector space.
- GRU encoder: Each GRU updates its hidden state sequentially using reset and update gates and a candidate output state.These gates control how the current character vector and previous hidden state form the next state.
- Tweet embedding: The final forward and backward states are combined through a fully connected layer to produce the tweet embedding.The learned embedding dimension is set equal to the GRU hidden-state dimension in the experiments.
- Hashtag prediction: The embedding is passed through a linear layer with one output per hashtag, followed by softmax posterior probabilities.The model optimizes categorical cross-entropy between predicted and true hashtags, with L2 regularization.
4 Experiments and Results
The experiments compare a character-level tweet encoder with a word-level encoder for hashtag prediction across dataset, model-size, and input-vocabulary conditions. Tweet2vec performs especially well on rare-word posts and remains stronger as model size and task complexity increase.
- Experimental setup: The word-level baseline splits tweets into whitespace-delimited tokens, while tweet2vec encodes character sequences with comparable parameter budgets.The baseline uses a 20K-word lookup vocabulary; tweet2vec uses character inputs and GRU states.
- Evaluation: Hashtag prediction is evaluated on held-out posts using average precision@1, recall@10, and mean rank of correct hashtags.Predictions are ranked because a post may have more than one correct hashtag.
- Vocabulary conditions: Tweet2vec significantly outperforms the word model on rare-word posts and performs comparably on frequent-word posts.The rare-word and frequent-word test sets each contain 2,000 posts selected by their number of out-of-vocabulary words.
- Model size: Tweet2vec always outperforms the word-based model at matched parameter counts, with little performance variation as model size increases.This comparison uses precision@1 across the tested parameter settings.
- Task complexity: Tweet2vec outperforms its word-based counterpart in small, medium, and large hashtag-prediction settings.The settings vary training data size and the number of output labels, with different test sets for each setting.
- Qualitative analysis: Example predictions indicate that the character model handles segmentation errors, spelling mistakes, emojis, and special characters while matching word-level performance on in-vocabulary tokens.The qualitative examples identify robustness to noisy and nonstandard social-media text as a strength.
5 Conclusion
The paper presents tweet2vec, a character-level encoder trained with hashtag supervision to represent social-media posts. It outperforms the word-based approach, particularly when posts contain many rare words, while the authors note that the study focuses on English-language posts.
- Tweet2vec is a character-level encoder for social-media posts trained using associated hashtags as supervision.
- Tweet2vec outperforms the word-based approach and does significantly better when input posts contain many rare words.
- The study focuses only on English-language posts, although the model requires no language-specific preprocessing and can be extended to other languages.
- The authors provide an off-the-shelf encoder and code for computing tweet vector representations.