Source-linked AI summary
SemEval-2013 Task 2: Sentiment Analysis in Twitter
Preslav Nakov, Zornitsa Kozareva, Alan Ritter, Sara Rosenthal, Veselin Stoyanov, Theresa Wilson
TL;DR
Sentiment-analysis research lacked suitable social-media datasets for reliable comparison, especially for informal Twitter and SMS text and expression-level polarity. The task created and released a crowdsourced corpus with phrase- and message-level annotations, and attracted 149 submissions from 44 teams; results were better for expression-level than message-level sentiment, with Twitter generally outperforming out-of-domain SMS.
Problem
Social-media sentiment research lacked suitable datasets, including publicly available Twitter or SMS corpora with expression-level sentiment annotations.
Method
The task created the SemEval Tweet corpus with crowdsourced contextual phrase-level and overall message-level polarity annotations for Twitter and SMS.
Results
Results were generally better on Twitter than out-of-domain SMS and significantly better for expression-level subtask A than message-level subtask B.
Takeaways & Limitations
The released datasets attracted 149 submissions from 44 teams and were intended as a useful community testbed beyond SemEval.
Takeaways & Limitations
Lexicon-based filtering biased the dataset to some degree, although expressions outside the lexicon remained.
Abstract
from arXiv · showhide
In recent years, sentiment analysis in social media has attracted a lot of research interest and has been used for a number of applications. Unfortunately, research has been hindered by the lack of suitable datasets, complicating the comparison between approaches. To address this issue, we have proposed SemEval-2013 Task 2: Sentiment Analysis in Twitter, which included two subtasks: A, an expression-level subtask, and B, a message-level subtask. We used crowdsourcing on Amazon Mechanical Turk to label a large Twitter training dataset along with additional test sets of Twitter and SMS messages for both subtasks. All datasets used in the evaluation are released to the research community. The task attracted significant interest and a total of 149 submissions from 44 teams. The best-performing team achieved an F1 of 88.9% and 69% for subtasks A and B, respectively.
1 Introduction
Social-media sentiment research faced limited datasets and informal, structured text challenges. SemEval-2013 Task 2 addressed these gaps with a released corpus containing expression- and message-level polarity annotations for Twitter and SMS.
- Twitter and SMS messages create NLP challenges through short length, informal language, creative spelling, slang, URLs, hashtags, and abbreviations.
- Social-media messages also contain structured information about follows, retweets, and tags that can support empirical study of social interactions and opinion.
- Existing sentiment corpora largely focused on traditional text, while Twitter datasets were small, proprietary, Spanish-only, noisy, or limited to message-level labels.
- The paper describes the task, dataset creation, evaluation methodology, participating approaches, and their scores.
2 Task Description
The task evaluated sentiment at two granularities: marked expressions and entire messages. Teams could train constrained systems using the provided data or unconstrained systems using additional training data.
- Subtask A classified a marked word or phrase as positive, negative, or neutral in context, with boundaries supplied in advance.
- Subtask B classified an entire message as positive, negative, or neutral, selecting the stronger polarity when both positive and negative sentiment appeared.
- Each team could submit one constrained and one unconstrained system per subtask.
- Constrained systems could use the provided training data and external lexicons, whereas unconstrained systems could add training data through supervised, semi-supervised, or unsupervised methods.
- The constrained/unconstrained distinction concerns data used to train the classifier; lexicons alone do not make a system unconstrained unless they label additional messages for training.
3 Dataset Creation
The corpus combined automatically selected topical Twitter data with SMS test messages and crowdsourced annotations. Annotation aggregation used agreement thresholds and majority labels, while lexicon filtering improved balance but introduced dataset bias.
- 3.1 Data Collection: Tweets were collected through the public streaming API from January 2012 to January 2013, using named entities and popular topics to select sentiment-bearing content.
- 3.1 Data Collection: SentiWordNet filtering retained messages containing a positive or negative sentiment score greater than 0.3 in at least one word sense to reduce class imbalance.
- 3.1 Data Collection: Filtering based on an existing sentiment lexicon biases the dataset to some degree, although sentiment expressions outside the lexicon remain.
- 3.2 Dataset: The dataset includes Twitter training and testing messages plus SMS testing messages, with only partial overlap between the two subtasks after low-agreement messages were removed.
- 3.3 Annotation Process: Each sentence was labeled by five qualified Mechanical Turk workers for subjective phrases, phrase polarity, and overall sentence polarity.
- 3.3 Annotation Process: For subtask A, phrases and their polarities were accepted when at least two-thirds of annotations agreed; subtask B used majority polarity and discarded ties.
- 3.3 Annotation Process: Intersection-based annotation aggregation was more accurate than union or selecting the most prolific worker, but even the best annotator could not achieve perfect accuracy.
4 Scoring
Systems performed three-way classification for marked phrases or full messages, while evaluation emphasized positive and negative F1. Neutral labels remained part of prediction despite being excluded from the reported F-score.
- Each system classified a marked phrase or entire message as positive, negative, or objective, and was scored on positive/negative predictions against the other classes.
- Positive and negative F1 scores were computed from class-specific precision and recall, then averaged as F = (Fpos + Fneg)/2.
- Ignoring Fneutral did not remove neutral labels from the task, because misclassifying neutral examples as positive or negative lowers Fpos and Fneg.
- The supplied scorer output the overall F-score, a three-class confusion matrix, and validation of submission formatting.
5 Participants and Results
The task attracted diverse systems across Twitter and SMS, with contextual phrase-level polarity substantially easier than message-level polarity. Performance was generally higher on in-domain Twitter data than out-of-domain SMS data, and NRC-Canada was strongest overall.
- Subtask B: Message Polarity: Subtask B was significantly harder than subtask A, likely because messages can contain both positive and negative sentiment.
- Subtask A: Contextual Polarity: 23 teams submitted 21 constrained and 7 unconstrained systems for subtask A on Twitter.
- Subtask A: Contextual Polarity: One system was semi-supervised, while most systems were supervised and commonly used SVM, Naive Bayes, or Maximum Entropy classifiers.
- Subtask A: Contextual Polarity: 88.9% was NRC-Canada's best constrained F1-measure on the Twitter test set, while AVAYA achieved 87.4% in the unconstrained setting.
- Subtask A: Contextual Polarity: 88.4% was the best constrained F-measure on SMS, achieved by GU-MLT-LT, while AVAYA's best unconstrained F1 was 85.8%.
- Overall: NRC-Canada was ranked first in three of the four conditions and was the strongest team overall.
6 Discussion
Participants predominantly used supervised systems with standard classifiers, diverse linguistic and Twitter-specific features, sentiment lexicons, and preprocessing for noisy messages. Most systems predicted positive, negative, and neutral labels, while constrained systems generally performed best.
- The best systems for both subtasks and both test datasets were constrained, although additional data could still be useful and sometimes worsened unconstrained runs.
- Most participants used supervised systems, with only five semi-supervised systems and one unsupervised system reported.
- Most participants predicted positive, negative, and neutral labels, although some omitted neutral because the final F1-score averaged only positive and negative predictions.
- Popular classifiers included SVM, Max-Ent, linear classifiers, and Naive Bayes, alongside manual rules and classifier ensembles.
- Systems combined word, word-shape, syntactic, Twitter-specific, and sentiment-related features, with extensive use of sentiment lexicons.
- Most participants preprocessed noisy Twitter messages through operations including tokenization, normalization, stemming, and removal of platform-specific artifacts.
7 Conclusion
The paper introduced SemEval-2013 Task 2 on Twitter sentiment analysis, attracting broad participation and releasing newly created datasets for research use. The task received 149 submissions from 44 teams, and its datasets were intended to support research beyond SemEval.
- The task attracted 149 submissions from 44 teams and released its newly created datasets under a Creative Commons Attribution 3.0 Unported License.
- The authors expected the released datasets to be useful to researchers beyond SemEval.