Source-linked AI summary
SemEval-2017 Task 4: Sentiment Analysis in Twitter
Sara Rosenthal, Noura Farra, Preslav Nakov
TL;DR
The paper addresses the need for a broad, comparable benchmark for Twitter sentiment across overall sentiment, topic sentiment, ordinal scales, and quantification. It reruns the SemEval-2016 subtasks, adds Arabic and public user-profile information, and reports a popular evaluation with 48 participating teams. The task supplies datasets and evaluation settings for comparing sentiment systems across English and Arabic.
Problem
Twitter sentiment research needs comparable resources spanning overall sentiment, topic sentiment, multiple ordinal scales, and sentiment distributions, including underexplored Arabic topic sentiment.
Method
The task reruns five SemEval-2016 subtasks in English and Arabic, provides annotated datasets, and makes public Twitter user-profile information available.
Results
48 teams participated, while the top English Subtask A teams twtr and DataStories achieved a macro-average recall of 0.681.
Takeaways & Limitations
The task provides a large accessible benchmark of over 70,000 tweets across two languages for evaluating and comparing sentiment-analysis methods.
Takeaways & Limitations
The impact of user information remains difficult to judge because the best approaches did not use it, and Arabic received only nine participants.
Abstract
from arXiv · showhide
This paper describes the fifth year of the Sentiment Analysis in Twitter task. SemEval-2017 Task 4 continues with a rerun of the subtasks of SemEval-2016 Task 4, which include identifying the overall sentiment of the tweet, sentiment towards a topic with classification on a two-point and on a five-point ordinal scale, and quantification of the distribution of sentiment towards a topic across a number of tweets: again on a two-point and on a five-point ordinal scale. Compared to 2016, we made two changes: (i) we introduced a new language, Arabic, for all subtasks, and (ii)~we made available information from the profiles of the Twitter users who posted the target tweets. The task continues to be very popular, with a total of 48 teams participating this year.
1 Introduction
SemEval-2017 Task 4 reran the prior sentiment subtasks while adding Arabic and public Twitter user information. The task targets overall sentiment, topic sentiment, ordinal classification, and tweet-level sentiment distributions.
- The task distinguishes categorical sentiment from topic-specific sentiment about people, products, or events, typically using positive, negative, and neutral labels.
- Earlier SemEval tasks included non-English languages such as Arabic, but did not target tweets or sentiment toward a topic.
- SemEval-2017 Task 4 reran the SemEval-2016 subtasks for overall sentiment, topic classification, ordinal regression, and quantification.
- Arabic was introduced across the subtasks to expand resources for multilingual and topic-focused sentiment analysis, especially for morphologically complex languages.
- Public Twitter profile information, including demographics and network-related information, was made available to examine its impact on sentiment analysis.
2 Task Definition
SemEval-2017 Task 4 defines five subtasks, each offered in English and Arabic, covering tweet classification and topic-level sentiment quantification on two-point and five-point scales.
- The five subtasks cover overall tweet sentiment, two-point topic classification, five-point topic ordinal classification, and two quantification tasks.
- Subtask A: Subtask A classifies a tweet as POSITIVE, NEGATIVE, or NEUTRAL.
- Subtasks B–C: Subtasks B and C classify sentiment toward a topic on two-point and five-point scales, respectively.
- Subtasks D–E: Subtasks D and E estimate the distribution of tweets across two or five sentiment classes for a topic.
- Each subtask runs for both English and Arabic; Subtask A continued from earlier editions, while Subtasks B–E had previously run in SemEval-2016 Task 4.
3 Datasets
The task provides English and newly collected Arabic Twitter datasets annotated for sentiment at multiple scales and toward specified topics. Collection, filtering, annotation, and quality-control procedures were standardized across the data.
- The datasets contain tweets annotated for sentiment on 2-point, 3-point, and 5-point scales, with new Arabic training data and new English and Arabic test sets.
- Topics were selected from popular English and Arabic Twitter trends and included named entities, geopolitical entities, and other public issues.
- Duplicate and highly similar tweets were filtered, and topics were retained only when at least 100 tweets remained.
- Test topics differed from training and development topics for both English and Arabic.
- CrowdFlower annotators labeled overall and topic-level polarity, with instructions emphasizing opinions about the topic itself rather than contextual events.
- Each tweet received at least five annotations, while hidden tests and pilot runs supported quality control and instruction refinement.
PPY ¯ Federer: HIGHLYPOSITIVE
The supplied passages describe annotation consolidation and dataset-use procedures, including Arabic label concentration and separate English and Arabic data resources.
- Arabic contributors produced very few HIGHLYPOSITIVE and HIGHLYNEGATIVE annotations despite receiving examples of those labels.
- When three of five annotators agreed, the label was accepted; otherwise, five-point labels were averaged after mapping them to −2 through 2.
- English training and development data combined all previous editions, while no prior-year progress set-aside was retained.
- Participants received a default Arabic train-development split and scripts for downloading tweets and basic author information.
4 Evaluation Measures
The task evaluates sentiment classification and quantification with measures tailored to class imbalance, ordinal distance, and topic-level prevalence estimation. Scores are generally computed per topic and macroaveraged across topics.
- Macroaveraging: For topic-based subtasks, each evaluation measure is computed independently per topic and then averaged across topics to produce the final score.This macroaveraging procedure applies across the classification and quantification evaluations described here.
- Subtasks A–B: AvgRec averages recall across sentiment classes and is preferred because it is more robust to class imbalance than standard accuracy and F1.For Subtask A it covers POSITIVE, NEGATIVE, and NEUTRAL; for two-class Subtask B it averages POSITIVE and NEGATIVE recall.
- Subtask C: MAEM evaluates five-point ordinal classification by averaging absolute distances between predicted and true class labels across true classes.It is robust to class imbalance, whereas ordinary mean absolute error is equivalent only on perfectly balanced datasets; lower values are better.
- Subtask D: KLD measures error between true and predicted class distributions for binary quantification, with additive smoothing ensuring the measure is always defined.KLD ranges from 0 for coinciding distributions to +∞, and lower values are better.
- Subtask E: EMD evaluates ordinal quantification by measuring distance between estimated and true prevalence distributions over totally ordered sentiment classes.Its values range from 0 to |C| −1, with lower values better, and results are averaged across topics.
5 Participants and Results
The task attracted 48 teams, with deep learning and ensembles prominent among English systems and Arabic systems relying mainly on language-specific approaches. Results varied across subtasks, languages, and baselines, with strong gains in several quantification settings.
- Participation: 48 teams participated, including 46 in an English subtask and 9 in an Arabic subtask.Subtask A was the most popular, with 38 English and 8 Arabic participants.
- Methods: At least 20 teams used deep learning or neural networks, while SVM and Liblinear remained popular alternatives.Other systems used Maximum Entropy, Logistic Regression, Random Forest, Naïve Bayes, and Conditional Random Fields.
- Subtask A: BB_twtr and DataStories tied for first in English Subtask A with macro-average recall of 0.681, and both used deep learning.BB_twtr used an LSTM/CNN ensemble, whereas DataStories used deep LSTMs with attention.
- Subtask A: NileTMRG ranked first in Arabic Subtask A with a score of 0.583, and all Arabic teams beat every baseline across metrics.Its system combined Naïve Bayes with lexical and sentiment features and expanded training data to about 13K examples using external tweets.
- Subtasks B and C: The best Subtask B accuracy reached 0.882 for English and 0.768 for Arabic, with higher scores partly reflecting the two-class setup.Almost all teams beat the majority-class baseline for Subtask B, whereas only two beat the NEUTRAL baseline for Subtask C.
6 Conclusion and Future Work
SemEval-2017 Task 4 remained popular while expanding sentiment evaluation to Arabic and user information. The authors identify larger Arabic training resources, cross-lingual data, richer user information, and related tasks as useful future directions.
- 48 teams participated, and the benchmark provided over 70,000 tweets across English and Arabic for method comparison.
- Arabic attracted nine participants, while four teams used user information introduced in this edition.
- Further exploration should collect more Arabic training data and encourage cross-lingual training data.
- Future editions could add languages, provide further user information, and include irony and emotion detection.
- Deep learning remained popular among state-of-the-art approaches, and the authors expected this trend to continue.