Source-linked AI summary
SemEval-2016 Task 4: Sentiment Analysis in Twitter
Preslav Nakov, Alan Ritter, Sara Rosenthal, Fabrizio Sebastiani, Veselin Stoyanov
TL;DR
SemEval-2016 Task 4 addresses how to analyze Twitter sentiment for both individual opinions and aggregate topic-level prevalence. It organizes this problem into five subtasks combining classification or quantification with two- or five-point sentiment scales, and reports that top systems generally used deep learning. The dataset construction involved filtering and annotation choices that introduced documented limitations.
Problem
Twitter sentiment applications often seek aggregate positive and negative prevalence about topics, while classification and quantification are distinct tasks requiring different evaluation perspectives.
Method
The task evaluates five subtasks combining tweet-level or set-level sentiment analysis with two- or five-point scales, using a dataset of manually filtered topics and annotated tweets.
Results
Top-ranked systems generally used deep learning, including convolutional or recurrent neural networks and word embeddings, often without tuning specifically to ordinal or quantification requirements.
Takeaways & Limitations
Future quantification tasks could generate multiple artificial test topics by varying removed tweets, increasing experimentation while keeping manual annotation constant.
Takeaways & Limitations
Lexicon-based filtering biases the dataset to some degree, although tweets also contain sentiment expressions outside the lexicon.
Abstract
from arXiv · showhide
This paper discusses the fourth year of the ``Sentiment Analysis in Twitter Task''. SemEval-2016 Task 4 comprises five subtasks, three of which represent a significant departure from previous editions. The first two subtasks are reruns from prior years and ask to predict the overall sentiment, and the sentiment towards a topic in a tweet. The three new subtasks focus on two variants of the basic ``sentiment classification in Twitter'' task. The first variant adopts a five-point scale, which confers an ordinal character to the classification task. The second variant focuses on the correct estimation of the prevalence of each class of interest, a task which has been called quantification in the supervised learning literature. The task continues to be very popular, attracting a total of 43 teams.
1 Introduction
SemEval-2016 Task 4 builds on established Twitter sentiment classification while introducing ordinal classification and quantification variants. These variants reflect both ordered sentiment scales and applications interested in aggregate class prevalence.
- Sentiment classification detects whether a textual item expresses a positive or negative opinion generally or about a given entity.
- Twitter sentiment classification supports applications in political science, social sciences, market research, and related fields.
- The task had attracted the highest participation of any task in the previous three SemEval campaigns.
- SemEval-2016 retained earlier single-label classification subtasks while introducing new problems involving five-point ordinal scales and quantification.
- Quantification estimates the prevalence or class distribution of unlabelled tweets, which better matches applications using aggregate topic-level data.
- For Subtasks B–E, tweets are supplied with their topic labels, so participants need not determine whether each tweet concerns that topic.
2 Task Definition
Task 4 comprises five subtasks spanning tweet-level classification and set-level quantification across two- and five-point sentiment scales. Subtask A reruns prior work, while Subtasks B–E introduce or extend topic-focused and distribution-estimation settings.
- Task 4 defines five subtasks covering overall sentiment, topic-directed sentiment, five-point sentiment, binary quantification, and five-class quantification.
- Subtask A predicts positive, negative, or neutral sentiment for an individual tweet and reruns the most popular prior task.
- Subtask B predicts positive or negative sentiment toward a known topic in an individual tweet.
- Subtask C predicts topic-directed sentiment on an ordered five-point scale from HIGHLYNEGATIVE to HIGHLYPOSITIVE.
- Subtasks D and E estimate topic-level sentiment distributions over two classes and five classes, respectively.
- The new subtasks form a 2×2 design crossing classification versus quantification with two-point versus five-point granularity.
3 Datasets
The Tweet 2016 dataset combined prior SemEval data with newly collected and annotated tweets for five subtasks. Collection filtered tweets into distinct topics, while annotation used multiple annotators, quality controls, consolidation rules, and staged test-data release.
- Dataset construction: The Tweet 2016 dataset comprised TRAIN, DEV, DEVTEST, and TEST, combining prior-edition data for training and development with newly created training and testing sets.
- Tweet collection: Tweets were collected through the public streaming Twitter API, with training and development data gathered from July to October 2015 and test data from October to December 2015.
- Tweet collection: Manual filtering produced 200 meaningful topics with at least 100 tweets each, excluding incomprehensible, ambiguous, overly general, or merely topic-mentioning tweets.
- Tweet collection: Training and test topics did not overlap, so the test set contained tweets about topics different from those used for training and development.
- Annotation: Mechanical Turk annotated TRAIN, DEV, and DEVTEST, whereas CrowdFlower annotated TEST with hidden tests used to identify unreliable annotators.
- Caveat: Filtering by a sentiment lexicon biased the dataset to some degree, although tweets also contained sentiment expressions outside the lexicon.
- Annotation: For five-point annotations, labels with three-of-five agreement were accepted; otherwise, labels were mapped to −2 through 2, averaged, and rounded to the nearest integer.
- Annotation: The 20,632-tweet TEST set contained 2,760 unanimous annotations, 9,944 majority labels, and 7,928 cases resolved by averaging.
4 Evaluation Measures
SemEval-2016 Task 4 evaluates five sentiment subtasks using task-specific measures, with topic-wise computation and macroaveraging for Subtasks B–E. The section also documents a scoring-script bug that may have affected Subtask B rankings.
- Evaluation protocol: For Subtasks B–E, measures are computed independently for each topic and then averaged across topics; an earlier scoring script used the wrong Subtask B measure.The script computed F1 instead of ρPN, and this could have penalized participants who used it for parameter optimization after submissions closed.
- Subtask A: Message polarity classification: Subtask A assigns each tweet exactly one of POSITIVE, NEUTRAL, or NEGATIVE and evaluates performance with the adopted F1-based measure.The measure uses precision and recall for the POSITIVE and NEGATIVE classes, computed from confusion-matrix cells.
- Subtask B: Tweet classification according to a two-point scale: Subtask B is binary classification, assigning each tweet to POSITIVE or NEGATIVE and using macroaveraged recall.Its score is interpreted so that 1 denotes a perfect classifier, 0 a perverse classifier, and 0.5 both a trivial and an expected random classifier; it is more robust to class imbalance than accuracy and F1.
- Subtask C: Tweet classification according to a five-point scale: Subtask C is ordinal classification over five ordered sentiment classes, where errors are weighted by the distance between predicted and true labels.The evaluation measure is macroaveraged mean absolute error, with lower values better and robustness to class imbalance.
- Subtask D: Tweet quantification according to a two-point scale: Subtask D estimates the relative frequency of POSITIVE and NEGATIVE tweets rather than classifying every tweet, using Kullback-Leibler Divergence.KLD measures distribution-estimation error, ranges from 0 to +∞, and is computed separately for each topic before averaging.
- Subtask E: Ordinal quantification: Subtask E performs ordinal quantification by estimating the prevalence of each class on an ordered five-point sentiment scale.The classes are HIGHLYPOSITIVE, POSITIVE, NEUTRAL, NEGATIVE, and HIGHLYNEGATIVE.
5 Participants and Results
SemEval-2016 Task 4 attracted 43 teams, with participation spanning five subtasks and a clear shift toward deep learning, distant supervision, and word embeddings. Results also show limited tailoring to ordinal and quantification-specific task properties.
- Participation: Quantification-related subtasks drew 24 participations, while ordinal subtasks drew 21, but only three teams entered all five subtasks.Twenty-three teams participated in only one subtask, usually Subtask A.
- System trends: Deep learning dominated participating systems, especially convolutional and recurrent neural networks, while kernel machines became less common.Theano and Keras were the most frequently used deep-learning libraries.
- System trends: Distant supervision and pre-trained word embeddings were ubiquitous, whereas preprocessing remained largely conventional.Common preprocessing included negation-scope detection, normalization, n-gram extraction, part-of-speech n-grams, and sentiment lexicons.
- Subtask A: In Subtask A, 34 systems beat the baseline on all three measures except one system that scored below it on accuracy.The top team used an ensemble of convolutional neural networks, word2vec embeddings, and distant supervision.
- Subtask A: Top-ranked systems generally transferred well across earlier test sets, but improvements could reflect either doubled training data or advances in learning methods.For a given test set, scores improved by 1–3 absolute points in its second year of use before stabilizing or declining slightly.
- Subtask B: In Subtask B, convolutional networks and word embeddings were common among leading systems, while only one top-ten team used a support vector machine.The top-scoring team combined convolutional neural networks, topic modeling, and word2vec embeddings.
- Subtask C: Subtask C showed limited use of ordinal methods: only two of 11 teams explicitly exploited the task’s ordinal structure.PUT and ISTI-CNR used ordinal regression approaches, while other teams mostly applied general-purpose multiclass methods.
6 Conclusion and Future Work
SemEval-2016 Task 4 broadened Twitter sentiment analysis with five subtasks, including new ordinal and quantification variants. The task’s top systems generally used deep learning, while proposed follow-ups target better-controlled quantification evaluation and new linguistic settings.
- Task scope: Five subtasks included three significant departures from previous editions, covering ordinal sentiment classification and quantification individually or together.The task retained earlier overall-sentiment and topic-sentiment subtasks while introducing these new variants.
- Main findings: Most top-ranked systems used deep learning, including convolutional or recurrent neural networks and general-purpose or task-specific word embeddings.Several systems achieved strong scores without exploiting ordinal or quantification-specific properties.
- Future work: For quantification subtasks, generating artificial test topics with altered class prevalences would expose test-set prevalence and distribution-drift effects.The proposal selectively removes tweets from chosen classes to create topics with controlled prevalence changes.
- Future work: Varying tweet removals could expand experimentation while keeping the amount of manual annotation constant.This would produce many test topics from controlled modifications of existing data.
- Future work: Future subtasks could distinguish topic-neutral tweets from lukewarm fair opinions or support sentiment transfer across languages.The paper specifically suggests multilingual and cross-lingual variants, including English–Arabic transfer.