Source-linked AI summary
SemEval-2015 Task 10: Sentiment Analysis in Twitter
Sara Rosenthal, Saif M Mohammad, Preslav Nakov, Alan Ritter, Svetlana Kiritchenko, Veselin Stoyanov
TL;DR
The paper presents SemEval-2015 Task 10 as a five-subtask evaluation of sentiment in Twitter and related short social-media text. It organizes rerun and newly introduced prediction tasks, constructs datasets with crowdsourced and MaxDiff annotations, and reports participation and system findings. The shared task attracted 41 teams, while sentiment lexicons and preprocessing were especially important features.
Problem
Sentiment analysis in social media requires evaluation across contextual, message-level, topic-directed, aggregate-topic, and prior-polarity judgments.
Method
The task combines crowdsourced annotations for subtasks A–D with MaxDiff judgments that produce rankings and real-valued scores for subtask E.
Results
41 teams participated, and sentiment lexicons plus tweet preprocessing and normalization were among the most important features.
Takeaways & Limitations
The shared task evaluates sentiment analysis across multiple granularities, from contextual phrases and messages to topics and Twitter-term polarity.
Takeaways & Limitations
Lexicon-based filtering biased the dataset to some degree, and teams received different numbers of training tweets depending on download timing.
Abstract
from arXiv · showhide
In this paper, we describe the 2015 iteration of the SemEval shared task on Sentiment Analysis in Twitter. This was the most popular sentiment analysis shared task to date with more than 40 teams participating in each of the last three years. This year's shared task competition consisted of five sentiment prediction subtasks. Two were reruns from previous years: (A) sentiment expressed by a phrase in the context of a tweet, and (B) overall sentiment of a tweet. We further included three new subtasks asking to predict (C) the sentiment towards a topic in a single tweet, (D) the overall sentiment towards a topic in a set of tweets, and (E) the degree of prior polarity of a phrase.
1 Introduction
SemEval-2015 Task 10 addressed sentiment analysis in varied social-media text through five subtasks, combining two established tasks with three topic- and lexicon-focused additions.
- Social media enables researchers to study opinions from very large numbers of individuals at a scale previously unavailable.
- Twitter sentiment analysis must handle misspellings, poor grammar, emoticons, acronyms, and slang.
- 41 teams participated across five subtasks, with most teams participating in more than one.
- The task reran phrase- and message-level sentiment prediction and added topic sentiment for single messages, topic trends across message sets, and prior-polarity scoring.
- The paper covers task definitions, dataset creation, participating systems, results, lessons learned, related efforts, and future research directions.
2 Task Description
The task defined five sentiment-prediction problems spanning contextual phrases, individual messages, topic-directed sentiment, aggregate topic trends, and prior polarity of Twitter terms.
- Subtask A classifies a word or phrase in message context as positive, negative, or neutral.
- Subtask B classifies an entire message as positive, negative, or neutral/objective, choosing the stronger polarity when both occur.
- Subtask C predicts positive, negative, or neutral sentiment toward a specified topic in one message.
- Subtask D classifies overall sentiment toward a topic in contemporaneous messages as strongly or weakly positive, neutral, or strongly or weakly negative.
- Subtask E assigns each word or phrase a 0-to-1 score indicating its strength of association with positive sentiment.
3 Datasets
The paper describes collecting and annotating short social-media messages, focusing on the 2015 datasets while referring readers to earlier papers for 2013 and 2014 details.
- The dataset section explains how short social-media messages were collected and annotated for the 2015 task.
- Discussion of the 2013 and 2014 datasets is deferred to the corresponding earlier SemEval papers.
3.1 Data Collection
Datasets were built from topic-related social-media messages, filtered to address neutrality and topic quality, and supplemented with selected sentiment terms and crowdsourced annotation instructions.
- Data Collection: Tweets were gathered from popular topics identified through named entities extracted from millions of tweets using a Twitter-tuned NER system.
- Data Collection: Popular topics were selected from named entities frequently mentioned around specific dates, and later test messages used different topics and periods than training.
- Data Collection: Neutral-class skew was reduced by removing messages without sentiment-bearing words identified through SentiWordNet.
- Data Collection: For subtasks C and D, researchers removed incomprehensible, ambiguous, overly general, weakly related, and underpopulated topics, requiring at least 10 tweets.
- Data Collection: Lexicon-based filtering biases the dataset to some degree, although messages still contain sentiment expressions outside the lexicon.
- Data Collection: Mechanical Turk instructions asked workers to identify subjective words or phrases, mark their spans, and assign polarity labels.
- Data Collection: Figure 1 presents the Mechanical Turk instructions followed by a screenshot.
- Data Collection: Subtask E terms included ordinary, hashtagged, misspelled, abbreviated, shortened, slang, and negated expressions sampled across sentiment-value ranges.
3.2 Annotation
Subtasks A–D were annotated through Mechanical Turk, with multiple workers labeling phrase- and message-level sentiment. The annotations were filtered and consolidated using intersection or majority voting, while subtask E used relative term judgments to derive sentiment rankings.
- Subtasks A–D: Five Mechanical Turk workers annotated each tweet for subjective phrases, phrase polarity, overall polarity, and topic-directed polarity.These annotations were collected concurrently for subtasks A–D.
- Subtasks A–D: Annotations with overlapping phrases, missing sentiment labels, or extreme subjectivity markings were discarded.Discarded cases included messages marked subjective without subjective phrases and messages with every word marked subjective.
- Subtasks A–D: Subtask A used intersection consolidation, requiring agreement from three of five Turkers for subjectivity and polarity labels.This procedure often shortened annotated phrases to single words because long spans were difficult to agree on.
- Subtasks A–D: Union and highest-volume-Turker alternatives performed worse than intersection in manual analysis.The consolidated annotations became gold labels for subtasks A–D.
- Subtasks A–D: Subtasks B and C used majority voting with a minimum 3/5 agreement, discarded unresolved cases, and merged objective with neutral.The paper separately reports dataset statistics for subtasks A–D.
- Subtask E: Subtask E relied on MaxDiff judgments, asking annotators to identify the most and least positive term among four and infer pairwise rankings.Relative judgments were used because absolute sentiment scores are difficult for annotators to assign consistently.
3.3 Lower & Upper Bounds
The paper estimates attainable performance by comparing gold annotations with Turker annotations and presenting examples that distinguish phrase-, message-, and topic-level labels.
- Lower & Upper Bounds: Agreement bounds compare gold annotations with the best, worst, and average Turker for each HIT.Because the gold standard was derived by agreement, standard statistics such as Kappa could not be calculated.
- Lower & Upper Bounds: Examples report phrase-level subjective spans and polarity alongside message-level polarity for Twitter, SMS, and LiveJournal messages.
- Lower & Upper Bounds: Twitter examples show that sentiment toward a topic can differ from sentiment expressed by the message overall.
3.4 Tweets Delivery
Twitter’s terms of service prevented direct delivery of annotated tweets, so participants received indexes, labels, tweet IDs, and an API download script. Download timing gave teams different training-set sizes, but the reported impact was not major.
- Tweets Delivery: Twitter’s terms of service prevented direct delivery of annotated tweets to participants.
- Tweets Delivery: Participants received annotation indexes and labels, corresponding Twitter IDs, and a script for retrieving tweets through the Twitter API.
- Tweets Delivery: Different download times gave teams different numbers of training tweets.The analysis found no major impact, and some high-scoring teams had less training data than lower-scoring teams.
4 Scoring
The task used three-way sentiment classification and macro-averaged positive/negative F1 for subtasks A–C, while subtasks D and E evaluated topic trends and term rankings with specialized measures.
- Classification scoring: Systems assigned positive, negative, or objective/neutral labels, and the main classification score was macro-averaged F1 over positive and negative classes.
- Classification scoring: The overall classification score was F = (Fpos + Fneg)/2, averaging positive- and negative-class F1 scores.The scorer also outputs precision, recall, and F1 by class and test set.
- Subtask D: Subtask D predicts the ratio of positive to positive-plus-negative tweets for each topic, excluding neutral tweets from the formula.
- Subtask D: AvgDiff measures the average absolute difference between predicted and gold topic ratios, while AvgLevelDiff applies five coarse sentiment categories first.The five categories range from strongly positive to strongly negative.
- Subtask E: Subtask E compares system and aggregated human term rankings using Kendall’s τ officially and Spearman’s rank correlation unofficially.
5 Participants and Results
The 2015 task attracted 41 teams across five subtasks, with strong but uneven results across phrase-, message-, and topic-level sentiment prediction. New topic-focused subtasks were substantially harder than message-level classification, while systems commonly exceeded baselines.
- Participation: 41 teams participated across five subtasks, with participation ranging from 6 teams in subtask D to 40 in subtask B.Subtask A had 11 participants, C had 7, and E had 10.
- Subtask A: Phrase-Level Polarity: All subtask A systems exceeded the majority-class baseline by 25–40 absolute F1 points across datasets.unitn achieved 84.79 F1 on Twitter2015-test, narrowly ahead of KLUEless at 84.51.
- Subtask A: Phrase-Level Polarity: Subtask A’s best results varied by dataset: unitn led both progress Twitter datasets, while GTI led Twitter2014-sarcasm.None of the leading unitn, KLUEless, or IOA systems ranked in the top three on the sarcasm dataset.
- Subtask A: Phrase-Level Polarity: Compared with 2014, subtask A improved on Twitter2014-test from 86.63 to 87.12, while SMS and sarcasm performance declined.Twitter2013-test was nearly unchanged, and LiveJournal remained at 85.61.
- Subtask B: Message-Level Polarity: Subtask B’s top four systems formed a narrow cluster, led by Webis at 64.84 F1, while nearly all systems beat the baseline.The other leading scores were unitn 64.59, lsislif 64.27, and INESC-ID 64.17.
- Subtask B: Message-Level Polarity: On sarcastic 2015 tweets, IOA led subtask B with 65.77 F1, followed by INESC-ID at 64.91 and NLP at 63.62.The evaluation subset contained 75 tweets flagged as sarcastic by human annotators.
- Subtask C: Topic-Level Polarity: Subtask C was difficult: only three of seven teams beat the majority baseline, and the top three exceeded the fourth team by 15–25 absolute F1 points.TwitterHawk scored 50.51, KLUEless 45.48, and Whu-Nlp 40.70.
- Subtask C: Topic-Level Polarity: Subtask C was harder than B, with top-three F1 scores of 41–51 versus 64–65, despite similar majority-class baselines of 26.7 and 30.3.The paper explicitly attributes the difference to task difficulty rather than class distribution.
6 Discussion
Most systems used supervised learning, while sentiment lexicons, text normalization, and preprocessing were recurring ingredients. The task allowed unrestricted additional resources, and participation remained comparable to prior years despite redistribution across subtasks.
- Methods: Supervised learning remained dominant, with SVMs, maximum entropy, CRFs, and linear regression among the common approaches.Top systems in several subtasks also used deep neural networks and word embeddings.
- Methods: Sentiment-lexicon features were the most important features, alongside bag-of-words, hashtags, negation handling, word shape, punctuation, and elongated-word features.Tweet preprocessing and normalization were also important parts of the processing pipeline.
- Evaluation Setting: Participants could use any additional data, resources, and tools because the task did not distinguish constrained from unconstrained systems.
- Participation: 41 teams participated in 2015, compared with 46 in 2014 and 44 in 2013, while subtask B remained dominant with 40 teams.Participation in A decreased as new, related subtasks C and D and subtask E divided the field.
- Related Efforts: The task was part of a broader Sentiment Track that included tasks on implicit event polarity, figurative language, and aspect-based sentiment.A related paraphrase and semantic similarity task also focused on tweets.
7 Conclusion
SemEval-2015 Task 10 organized five Twitter sentiment subtasks spanning contextual, document-level, topic-oriented, and lexicon-oriented prediction. The conclusion points toward ordinal scales and aggregate quantification to better match application and research needs.
- The task covered five subtasks: contextual term sentiment, whole-message sentiment, topic polarity, topic quantification, and real-valued prior sentiment scores.
- The planned five-point topic scale would shift sentiment prediction from binary classification to ordinal regression.
- The planned continuation of trend detection would shift from classifying individual items to quantifying the percentage of positive and negative tweets.
- A five-point trend-detection task would formulate the problem as ordinal quantification and better serve business and research applications.