Source-linked AI summary
SemEval-2014 Task 9: Sentiment Analysis in Twitter
Sara Rosenthal, Preslav Nakov, Alan Ritter, Veselin Stoyanov
TL;DR
Sentiment analysis in social media must handle short, informal messages and limited evidence beyond traditional sentiment corpora. SemEval-2014 extends a phrase- and message-level annotated corpus, adds regular, sarcastic, and LiveJournal test sets, and reports improved progress-set results but lower performance on sarcastic tweets. The paper highlights sarcasm handling as an important direction while noting uncertainty about why unconstrained systems did not outperform constrained ones.
Problem
Short, informal social-media messages and the lack of tweet-focused sentiment corpora make sentiment expression difficult to study automatically.
Method
The task extends an annotated tweet and SMS corpus and evaluates three-way phrase- and message-level sentiment systems across Twitter, SMS, sarcastic tweets, and LiveJournal sentences.
Results
Progress-test performance consistently improved, with absolute gains of 0-1 points for subtask A and 2-3 points for subtask B, while sarcastic-tweet performance showed a sharp drop for most teams.
Takeaways & Limitations
Better handling of sarcastic language is identified as an important direction for future Twitter sentiment analysis.
Takeaways & Limitations
The discussion leaves unclear why unconstrained systems did not outperform constrained systems, citing possible differences in external data or the constrained/unconstrained definition.
Abstract
from arXiv · showhide
We describe the Sentiment Analysis in Twitter task, ran as part of SemEval-2014. It is a continuation of the last year's task that ran successfully as part of SemEval-2013. As in 2013, this was the most popular SemEval task; a total of 46 teams contributed 27 submissions for subtask A (21 teams) and 50 submissions for subtask B (44 teams). This year, we introduced three new test sets: (i) regular tweets, (ii) sarcastic tweets, and (iii) LiveJournal sentences. We further tested on (iv) 2013 tweets, and (v) 2013 SMS messages. The highest F1-score on (i) was achieved by NRC-Canada at 86.63 for subtask A and by TeamX at 70.96 for subtask B.
1 Introduction
Social-media sentiment analysis is difficult because informal, short messages contain creative language and limited context. SemEval-2014 extends the task’s corpus and evaluates systems on sarcastic tweets alongside other social-media sources.
- Social-media text includes creative spelling, punctuation, misspellings, slang, new words, URLs, and platform-specific abbreviations.
- Tweets and SMS messages are short, typically resembling sentences or headlines rather than documents.
- Existing sentiment corpora provide valuable resources but generally do not focus on tweets.
- The SemEval Tweet corpus contains tweets and SMS messages with phrase-level and message-level polarity annotations, and was extended with tweets and LiveJournal sentences.
- The 2014 evaluation added sarcastic tweets because the #sarcasm hashtag indicates that a message should not be taken literally and its polarity should be flipped.
2 Task Description
The task evaluates sentiment at both marked-expression and whole-message levels using three-way polarity classification. Systems may be constrained or unconstrained according to the additional training data they use.
- Subtask A classifies a marked word or phrase as positive, negative, or neutral in context, with boundaries supplied.
- Subtask B classifies an entire message as positive, negative, or neutral, choosing the stronger polarity when both positive and negative sentiment occur.
- Constrained systems train classifiers using the provided data, while unconstrained systems may use additional training data in supervised, semi-supervised, or unsupervised ways.
- A lexicon built from other data remains compatible with a constrained system when used as features, but auto-labeling extra messages for classifier training makes the system unconstrained.
3 Datasets
The evaluation combines existing and newly collected social-media data, including regular, sarcastic, and LiveJournal material. Sarcastic tweets receive modified annotation procedures to assess sarcasm independently of the hashtag.
- Training and development reuse the SemEval-2013 Twitter datasets and the annotated NUS SMS test set.
- The 2014 datasets add a Twitter test set, tweets containing #sarcasm, and LiveJournal sentences to test in-domain, sarcastic, and cross-source performance.
- New tweets were collected from popular topics containing sentiment-bearing words filtered with SentiWordNet.
- Sarcastic tweets were shown to Mechanical Turk annotators without the #sarcasm hashtag, who judged sarcasm and rated it as definite, probable, or absent.
- Annotations were combined by intersection, accepting a word when it appeared in two-thirds of the annotations.
- Participants received annotation indexes, Twitter IDs, and a download script rather than the annotated tweets, so access varied with download timing.
4 Scoring
Both subtasks use three-way sentiment classification but score systems through positive and negative F1, averaged into an overall score. Shuffled test sets assess generalization across sources without revealing their origins.
- Systems classify marked phrases or entire messages as positive, negative, or objective/neutral in both subtasks.
- Positive and negative F1-scores are computed from precision and recall, then averaged as F = (Fpos + Fneg)/2.
- The evaluation combines the two 2013 and three 2014 test sets into one shuffled test set.
- Shuffling prevents source identification and tests how systems trained on standard tweets generalize to sarcastic tweets and LiveJournal sentences.
- The provided scorer outputs the overall score and a confusion matrix for each of the five test sets.
5 Participants and Results
The task evaluated constrained and unconstrained systems across five test sets, with results reported separately for subtasks A and B. The best systems were constrained, while some improperly trained systems produced unusually high Twitter2013-test scores.
- Test sets and ranking: Tables 4 and 5 report results for two 2013 progress test sets and three new 2014 official test sets, including macro- and micro-averaged scores.Systems were officially ranked by their Twitter-2014 testset score; other rankings were secondary.
- Test sets and ranking: The 2013 test sets were included without prior notice to reduce overtuning, although they had been available for development.Participants were explicitly forbidden to use them for training.
- Evaluation caveat: Some participants violated the ban on using the 2013 test sets for training, producing unusually high Twitter2013-test scores that prompted corrected resubmissions.The tables identify those resubmissions.
- Constrained versus unconstrained systems: 7 of 27 subtask-A submissions and 8 of 50 subtask-B submissions were unconstrained, but unconstrained systems were not consistently better than constrained systems.Some unconstrained systems performed worse or the same, so both types were included in one ranking and marked accordingly.
- Subtask results: 27 submissions from 21 teams entered subtask A, and all participating systems outperformed the majority-class baseline by a sizable margin.Seven subtask-A submissions were unconstrained, while the best systems were constrained.
- Subtask results: 50 submissions from 44 teams entered subtask B, and all systems exceeded the majority-class baseline, although some were very close to it.Eight subtask-B submissions were unconstrained, while the best systems were constrained.
6 Discussion
The discussion finds that supervised learning, constrained systems, and established Twitter-oriented methods remained dominant, while progress-set performance improved but newer test conditions exposed lower scores and dataset-specific strengths.
- Methods: SVM, MaxEnt, and Naive Bayes remained the most popular classifiers, while two deep-learning submissions ranked second and tenth on subtask B.The deep-learning systems were coooolll and ThinkPositive.
- Methods: Participants used varied lexical, syntactic, and Twitter-specific features, with heavy reliance on sentiment lexicons and Twitter-specific preprocessing.Preprocessing included tokenization, stemming, lemmatization, stopword removal, POS tagging, URL and emoticon handling, normalization, abbreviation lookup, and punctuation removal.
- Continuity: 18 of 46 teams were returning teams, possibly contributing to the similarity of preprocessing, NLP tools, classifiers, and features across 2013 and 2014.The paper explicitly presents this as a probable partial explanation.
- Test-set differences: 86.63 versus 90.14 and 70.96 versus 72.12 were the best F1 scores on Twitter-2014 and Twitter-2013 for subtasks A and B, respectively.The paper notes that Twitter-2014 results were lower despite higher baselines, possibly because teams overfit to the available 2013 test data or used outdated sentiment dictionaries.
- Generalization: NRC-Canada performed well across all test sets, whereas TeamX was strong only on Twitter datasets after tuning for class imbalance in tweets.This contrast indicates differing cross-dataset performance profiles among leading systems.
7 Conclusion
SemEval-2014 Task 9 attracted 46 teams and introduced in-domain Twitter, out-of-domain LiveJournal, and sarcastic-tweet test sets. Performance was mostly comparable on LiveJournal but dropped sharply on sarcastic tweets, motivating future sarcasm-focused evaluation.
- 46 teams participated: 21 teams submitted 27 systems for subtask A, while 44 teams submitted 50 systems for subtask B.
- Three new 2014 test sets covered regular Twitter, LiveJournal, and sarcastic tweets.
- Performance was mostly comparable on LiveJournal but dropped sharply for most teams on sarcastic tweets.
- The authors planned a 2015 sarcasm sub-evaluation and another test domain.
- The authors considered removing the constrained/unconstrained distinction in the 2015 task.
- They also proposed studying sentiment trends toward topics because Twitter contains multiple opinions about a topic.