Source-linked AI summary
BB_twtr at SemEval-2017 Task 4: Twitter Sentiment Analysis with CNNs and LSTMs
Mathieu Cliche
TL;DR
The paper addresses accurate Twitter sentiment classification across SemEval-2017’s English subtasks. It combines CNNs and LSTMs with large-scale embedding pre-training, distant supervision, and ensembling. The resulting system ranked first on all five English subtasks.
Problem
The paper seeks highly accurate Twitter sentiment classification for a competition with five English subtasks and broad practical applications.
Method
The system pre-trains word embeddings on 100 million unlabeled tweets, fine-tunes them with distant supervision and task data, trains CNNs and LSTMs, and ensembles them.
Results
The final ensemble of 10 CNNs and 10 LSTMs achieved first rank in all five English subtasks.
Takeaways & Limitations
Combining diverse CNN and LSTM models with different hyperparameters and pre-training strategies produced the paper’s top-ranked competition system.
Takeaways & Limitations
Future work should examine more organic CNN–LSTM combinations and how unlabeled and distant data amounts affect performance.
Abstract
from arXiv · showhide
In this paper we describe our attempt at producing a state-of-the-art Twitter sentiment classifier using Convolutional Neural Networks (CNNs) and Long Short Term Memory (LSTMs) networks. Our system leverages a large amount of unlabeled data to pre-train word embeddings. We then use a subset of the unlabeled data to fine tune the embeddings using distant supervision. The final CNNs and LSTMs are trained on the SemEval-2017 Twitter dataset where the embeddings are fined tuned again. To boost performances we ensemble several CNNs and LSTMs together. Our approach achieved first rank on all of the five English subtasks amongst 40 teams.
1 Introduction
Twitter sentiment polarity is an accessible but practically valuable NLP task, motivating competition research toward highly accurate classifiers. SemEval-2017 evaluates this problem across five subtasks, while deep learning methods have shown strong performance in sentiment analysis.
- Twitter sentiment analysis supports applications including event monitoring and extracting trading signals from tweets about public companies.
- SemEval-2017 promotes research on Twitter sentiment classification because these applications often benefit from the best possible accuracy.
- The competition comprises five subtasks involving standard classification, ordinal classification, and distributional estimation.
- Deep learning techniques have outperformed traditional methods across several NLP tasks, including sentiment analysis.
- CNNs and LSTMs are highlighted as two popular deep learning techniques for sentiment analysis.
2 System description
The system combines CNN and bidirectional LSTM architectures for tweet classification, using word embeddings, convolutional or recurrent processing, and softmax outputs. CNNs extract salient n-gram features, while bidirectional LSTMs incorporate information from both sentence directions.
- CNN: Tweets are tokenized into words and mapped to 200-dimensional embeddings, producing padded input matrices with 80 word positions.
- CNN: CNN convolutions apply filters across word windows, use ReLU activations, and produce feature vectors over possible tweet windows.
- CNN: Multiple filter sizes and filtering matrices capture features spanning smaller or larger tweet regions.
- CNN: Max-pooling selects the strongest feature from each convolution regardless of its location, effectively extracting important n-grams.
- CNN: The CNN pooled representation passes through a 30-unit hidden layer and softmax classifier, with dropout used after pooling and the hidden layer.
- LSTM: The LSTM processes each word using its embedding and previous hidden state, with a 200-dimensional recurrent representation.
- LSTM: The LSTM uses input, forget, and output gates plus a cell state to retain information over short or long terms.
- LSTM: The bidirectional LSTM combines forward and backward hidden states, yielding a 400-dimensional vector for a 30-unit hidden layer and softmax output.
3 Training
The system uses staged training that progresses from unsupervised embedding pre-training through distant supervision and human-labeled SemEval training, then ensembles diverse models to improve accuracy.
- Distant training: A 10-million-tweet distant dataset was extracted as 5 million positive and 5 million negative tweets using emoticons as noisy labels.The distant dataset was created from the unlabeled tweets by associating sentiment with positive or negative emoticons.
- Unsupervised training: 100 million unlabeled English tweets were used to pre-train embeddings with Word2vec, FastText, and GloVe.Word2vec predicts context words, FastText adds subword information, and GloVe uses global word-word co-occurrence statistics.
- Supervised training: The final stage fine-tuned CNN and LSTM models on SemEval-2017 human-labeled data, first freezing and then unfreezing embeddings.Embeddings were frozen for approximately five epochs, followed by approximately five epochs with a tenfold lower learning rate; inverse-frequency class weighting addressed imbalance.
- Ensembling: Soft voting combined 10 CNNs and 10 LSTMs with varied initializations, training durations, filter sizes, and embedding pre-training algorithms.The ensemble was designed to reduce variance and boost accuracy.
4 Subtask specific tricks
For topic-associated subtasks, the system augments tweets with missing topic words and adds topic-membership indicators to the word embeddings.
- Subtask outputs: Output dimensions are 3 for subtask A, 2 for subtasks B and D, and 5 for subtasks C and E.For quantification subtasks D and E, output probabilities are converted into sentiment distributions using probability averaging.
- Topic handling: For subtasks B, C, D, and E, missing words from the associated topic are appended to the tweet during preprocessing.This topic augmentation was one of two steps reported to improve cross-validation accuracy.
- Topic handling: Topic-associated subtasks concatenate regular word embeddings with a five-dimensional embedding space encoding topic membership.Two indicator vectors distinguish words that are part of the topic from words that are not.
5 Results
Historical experiments show benefits from distant training, class weighting, and ensembling, while the final system achieved the best scores across all five English subtasks in 2017.
- Historical evaluation: The ensemble model effectively outperformed the individual models because their outputs were sufficiently uncorrelated to provide a small score boost.The most uncorrelated models came from different supervised architectures and different unsupervised embedding algorithms.
- Historical evaluation: Removing class weights or the distant training stage significantly lowered historical test-set scores.These experiments support class weighting and distant training as useful additions to the training procedure.
- Historical evaluation: GloVe produced a lower score than both FastText and Word2vec and was therefore excluded from the ensemble model.The historical comparison focused on subtask A using average F1 for the positive and negative classes.
- 2017 evaluation: The system achieved the best scores on all five English subtasks in the 2017 test set.For subtask A, the submission tied another team overall but ranked higher on accuracy and F1 score.
6 Conclusion
The paper presents an ensemble-based deep-learning system for SemEval-2017 Twitter sentiment analysis that obtained first rank across all English subtasks. It identifies more integrated CNN–LSTM combinations and analysis of unlabeled and distant data as future directions.
- Conclusion: The authors’ goal was to use deep-learning models and modern training strategies to build the best possible sentiment classifier for tweets.The system was developed for the SemEval-2017 Twitter sentiment analysis competition.
- Conclusion: The final model was an ensemble of 10 CNNs and 10 LSTMs using different hyper-parameters and pre-training strategies.The system participated in all English subtasks and obtained first rank in each.
- Future work: Future work includes combining CNNs and LSTMs more organically than through an ensemble and analyzing how unlabeled and distant data amounts affect performance.The proposed directions include a model similar to one described by Stojanovski et al. (2016).