Source-linked AI summary
NRC-Canada: Building the State-of-the-Art in Sentiment Analysis of Tweets
Saif M. Mohammad, Svetlana Kiritchenko, Xiaodan Zhu
TL;DR
Sentiment analysis of tweets and SMS requires detecting sentiment for whole messages and for terms within messages. The paper develops two SVM classifiers using surface-form, semantic, and sentiment features, including large automatically generated lexicons. On tweets, the submissions ranked first in both tasks, while lexicon features produced major gains and the systems also performed strongly on untuned SMS data.
Problem
The paper addresses three-way positive, negative, or neutral sentiment detection for whole messages and terms within messages.
Method
The authors build two SVM classifiers using surface-form, semantic, and sentiment features, including lexicons generated from hashtagged and emoticon-labeled tweets.
Results
The tweet submissions ranked first in both tasks, with macro-averaged F-scores of 69.02 for message-level and 88.93 for term-level sentiment.
Takeaways & Limitations
Automatically generated sentiment lexicons were particularly useful, providing more than 5 F-score points of additional message-level tweet performance beyond all other features.
Takeaways & Limitations
For the SMS dataset, the classifier model was trained only on tweets, limiting the extent of ngram benefits on SMS.
Abstract
from arXiv · showhide
In this paper, we describe how we created two state-of-the-art SVM classifiers, one to detect the sentiment of messages such as tweets and SMS (message-level task) and one to detect the sentiment of a term within a submissions stood first in both tasks on tweets, obtaining an F-score of 69.02 in the message-level task and 88.93 in the term-level task. We implemented a variety of surface-form, semantic, and sentiment features. with sentiment-word hashtags, and one from tweets with emoticons. In the message-level task, the lexicon-based features provided a gain of 5 F-score points over all others. Both of our systems can be replicated us available resources.
1 Introduction
The paper presents two SVM classifiers for message-level and term-level sentiment analysis, motivated by widespread interest in analyzing tweets across domains. The systems ranked first on tweet data in both SemEval-2013 tasks, while automatically generated sentiment lexicons provided a substantial message-level gain.
- Hundreds of millions of people use microblogging websites, creating interest in sentiment analysis across commerce, health, and disaster management.
- The authors created SVM classifiers for three-way sentiment detection at both message and term levels.The message-level classifier handles tweets and SMS, while the term-level classifier detects sentiment toward a term within a message.
- The tweet submissions ranked first in both SemEval-2013 tasks, achieving macro-averaged F-scores of 69.02 for message-level sentiment and 88.93 for term-level sentiment.
- The automatically generated hashtag and emoticon lexicons alone improved message-level tweet performance by more than 5 F-score points beyond all other features.The lexicons were made freely available.
2 Sentiment Lexicons
The paper builds sentiment lexicons from manually curated resources and automatically labeled tweets, using hashtags and emoticons as sentiment signals. The resulting lexicons capture unigram, bigram, and non-contiguous word-pair associations at large scale.
- Sentiment lexicons are lists of words associated with positive and negative sentiments, including the NRC Emotion, MPQA, and Bing Liu lexicons.The manually created resources contain approximately 14,000, 8,000, and 6,800 words, respectively.
- Hashtagged emotion words can indicate the sentiment of an entire tweet, so the authors used positive and negative hashtags to build a corpus.The seed collection included terms such as #good, #excellent, #bad, and #terrible.
- The NRC Hashtag Sentiment Lexicon was generated from 775,000 pseudo-labeled tweets using word–sentiment association scores.Tweets were labeled according to positive or negative seed hashtags.
- The hashtag lexicon contains 54,129 unigram, 316,531 bigram, and 308,808 non-contiguous-pair entries.
- The Sentiment140 Lexicon was generated from 1.6 million emoticon-labeled tweets and contains unigram, bigram, and non-contiguous-pair entries.Its entries number 62,468 unigrams, 677,698 bigrams, and 480,010 non-contiguous pairs.
3 Task: Automatically Detecting the Sentiment of a Message
The message-level task classifies tweets and SMS as positive, negative, or neutral using a linear SVM with diverse textual, sentiment, and negation features. Sentiment lexicons and negation handling were especially influential, while performance transferred to SMS despite tweet-only training.
- The task determines whether a message expresses positive, negative, or neutral sentiment.
- The classifier used a linear-kernel SVM trained on annotated tweets, with tweet preprocessing and feature-vector representations.URLs and userids were normalized, and tweets were tokenized and part-of-speech tagged with the CMU tool.
- Features included word and character ngrams, capitalization, part-of-speech tags, hashtags, lexicons, emoticons, elongated words, clusters, and negation contexts.Negated contexts added a NEG suffix to subsequent words and sentiment or emotion features.
- More than 8.5% gains came from sentiment lexicon features in ablation experiments across tweets and SMS.Tweets benefited mostly from automatic lexicons, whereas SMS benefited more from manually constructed lexicons.
- Ngrams were the second most important feature group for message-level classification, but their impact was smaller on SMS because training used tweets only.
- Attention to negations improved performance on both tweet and SMS datasets.Removing hashtags, emoticons, and elongated-word features had almost no impact, likely because other features captured their information.
4 Task: Automatically Detecting the Sentiment of a Term in a Message
The term-level task classifies whether a word or phrase expresses positive, negative, or neutral sentiment in context. The SVM combines target, contextual, lexical, syntactic, and surface-form features, achieving top tweet performance and strong transfer to unseen SMS data.
- The task detects positive, negative, or neutral sentiment expressed by a word or phrase within a message.
- Features: The classifier uses a linear-kernel SVM with features extracted from both the target term and a four-word context window on each side.The context window size was selected through development-set experiments.
- Features: The feature set combines ngrams, character patterns, elongated words, emoticons, punctuation, capitalization, negation, position, term splitting, and sentiment lexicons.The lexicons include automatically created and manually created resources.
- Results: 88.93 was the submission’s macro-averaged F-score on tweets, ranking first among 29 submissions from 23 teams.The same unchanged model ranked second on SMS with an F-score of 88.00; the first SMS system scored 88.39.
- Ablation results: Ngram features were most useful, while removing sentiment lexicons reduced F-score by 3.95 points on tweets and 4.64 on SMS.Removing word or character ngrams alone caused only a small drop, indicating overlapping information; negation modeling improved scores by 0.72 and 1.57 points, respectively.
- Ablation results: Target-only features outperformed context-only features, while adding context improved F-scores by roughly 2 to 4 points.
5 Conclusions
The paper presents two state-of-the-art SVM classifiers for message-level and term-level sentiment detection. On tweet data, the submissions ranked first in both SemEval-2013 subtasks, with sentiment lexicon and ngram features producing the largest gains.
- The authors created separate SVM classifiers for detecting message sentiment and sentiment expressed by a term within a message.
- The tweet submissions ranked first in both SemEval-2013 sentiment-detection subtasks.
- Sentiment lexicon and word- and character-ngram features produced the largest performance gains.