Source-linked AI summary
Tweet2Vec: Learning Tweet Embeddings Using Character-level CNN-LSTM Encoder-Decoder
Soroush Vosoughi, Prashanth Vijayaraghavan, Deb Roy
TL;DR
Tweet2Vec targets the difficulty of representing noisy, idiosyncratic tweets without extensive task-specific feature engineering. It learns general-purpose character-level CNN-LSTM encoder-decoder embeddings, which outperform competing methods across semantic relatedness and sentiment classification while supporting off-the-shelf classifiers.
Problem
Tweets’ short, noisy, and idiosyncratic nature makes standard methods ill-suited, while task-specific feature engineering is time consuming and inefficient.
Method
Tweet2Vec learns general-purpose tweet vectors with a character-level CNN-LSTM encoder-decoder, using convolutional layers and an LSTM encoder to produce a tweet representation.
Results
Tweet2Vec embeddings with simple logistic regression outperformed top-performing methods for Twitter semantic relatedness and sentiment classification without extensive feature engineering.
Takeaways & Limitations
The generic representations can be applied to tasks of different nature and outperform ParagraphVec and competing methods across the evaluated Twitter tasks.
Takeaways & Limitations
Future work is planned to add word-order augmentation and attention, indicating these capabilities are not included in the current method.
Abstract
from arXiv · showhide
We present Tweet2Vec, a novel method for generating general-purpose vector representation of tweets. The model learns tweet embeddings using character-level CNN-LSTM encoder-decoder. We trained our model on 3 million, randomly selected English-language tweets. The model was evaluated using two methods: tweet semantic similarity and tweet sentiment categorization, outperforming the previous state-of-the-art in both tasks. The evaluations demonstrate the power of the tweet embeddings generated by our model for various tweet categorization tasks. The vector representations generated by our model are generic, and hence can be applied to a variety of tasks. Though the model presented in this paper is trained on English-language tweets, the method presented can be used to learn tweet embeddings for different languages.
1. INTRODUCTION
Tweets are noisy, short, and idiosyncratic, making standard methods and task-specific feature engineering inefficient. Tweet2Vec addresses this with general-purpose character-level CNN-LSTM tweet representations that can support classification tasks.
- Tweets’ short, noisy, and idiosyncratic nature makes standard information retrieval and data mining methods ill-suited to Twitter.
- Extensive task-specific feature engineering is time consuming and inefficient because new features must be engineered for every task.
- Tweet2Vec generates general-purpose tweet vectors that can be used for any classification task with standard off-the-shelf classifiers.
- Tweet2Vec uses a character-level CNN-LSTM encoder-decoder to learn tweet representations.
- Word-level embedding and encoder-decoder methods are ill-suited to tweets’ extreme noise and idiosyncrasy, motivating Tweet2Vec’s character-level approach.
2. CNN-LSTM ENCODER-DECODER
Tweet2Vec combines character-level convolutional feature extraction with LSTM sequence encoding and decoding to learn tweet representations. Characters are encoded as padded one-hot matrices, processed by CharCNN, compressed into a 256-dimensional embedding, and used to reconstruct tweets character by character.
- Character-Level Input: Tweets use 70-character one-hot encoding and are padded into 150 × 70 matrices before entering the model.The maximum length includes 140 tweet characters plus padding.
- Character-Level CNN: Four one-dimensional convolutional layers extract character n-gram and higher-order textual features from sliding windows.Filters are applied across character windows, with deeper layers learning increasingly abstract features.
- Character-Level CNN: Pooling is restricted to the first two CNN layers so later outputs retain sequential structure for LSTM processing.The final CharCNN output has size 10 × 512 and is fed to the LSTM.
- Encoder: The LSTM processes CNN feature rows as time steps, using the final 256-dimensional hidden representation as the tweet embedding.The final encoder output encodes the entire tweet.
- Decoder: A two-layer LSTM decoder predicts each tweet character sequentially from the encoded representation and prior hidden vectors.Softmax outputs form a decoded tweet matrix that is compared with the original or synonym-replaced tweet.
3. DATA AUGMENTATION & TRAINING
The model is trained on augmented English-language tweets, with synonym replacement providing additional training variants. Parameters are optimized by minimizing character-level cross-entropy using Adam.
- Data Augmentation: Training uses 3 million randomly selected English-language tweets augmented by replacing selected words with WordNet synonyms.Replicated tweets exclude non-replaceable items such as stopwords, usernames, and hashtags.
- Training: The training objective is cross-entropy loss optimized with the Adam algorithm.The loss compares the true one-hot character distribution with the softmax output.
4. EXPERIMENTS
Tweet2Vec was evaluated on Twitter semantic relatedness and sentiment classification using standard classifiers over learned tweet representations. It outperformed competing methods on both tasks without extensive task-specific feature engineering.
- Evaluation tasks: The experiments evaluated Tweet2Vec on tweet semantic relatedness and tweet sentiment classification.The semantic relatedness task used paraphrase judgments, while sentiment classification assigned positive, negative, or neutral labels.
- Semantic Relatedness: 18K tweet pairs were used for semantic-relatedness training and 1K for testing, with 35% paraphrases and 65% non-paraphrases.
- Semantic Relatedness: Tweet2Vec outperformed the top four SemEval 2015 models and ParagraphVec on semantic relatedness without extensive task-specific feature engineering.Tweet-pair vectors were represented using their element-wise product and absolute difference, then classified with logistic regression.
- Sentiment Classification: 9,520 training tweets and 2,380 test tweets were used for sentiment classification, with positive, negative, and neutral classes.SemEval measured performance using the average F1-score of the positive and negative classes.
- Sentiment Classification: Tweet2Vec outperformed the top four SemEval 2015 models and ParagraphVec on sentiment classification without feature engineering.A logistic regression classifier was trained using the Tweet2Vec representations.
5. CONCLUSION AND FUTURE WORK
The paper concludes that Tweet2Vec provides general-purpose character-level tweet embeddings that perform strongly across two Twitter classification tasks. It also identifies word-order augmentation and attention as future extensions.
- Conclusion: Tweet2Vec generates general-purpose tweet representations with a character-level CNN-LSTM encoder-decoder architecture.The paper describes this as its novel method for learning and applying character-level tweet embeddings.
- Conclusion: Character-level modeling is presented as better suited to noisy, peculiar tweets than word-level embedding methods.The model is also described as robust to synonyms through WordNet-based data augmentation.
- Conclusion: Tweet2Vec representations supported off-the-shelf logistic regression classifiers that outperformed top-performing methods on semantic relatedness and sentiment classification.The evaluations used two SemEval 2015 tasks, and the model was trained on 3 million tweets.
- Future Work: Future work includes reordering tweet words for word-order robustness and adding attention to improve word alignment during decoding.The paper states that attention could improve overall performance.