Source-linked AI summary
On the Challenges of Sentiment Analysis for Dynamic Events
Monireh Ebrahimi, Amir Hossein Yazdavar, Amit Sheth
TL;DR
The paper addresses the challenge of analyzing sentiment in election tweets when topics and candidate references change over time. It examines candidate-specific classification, training-data updates, user political leanings, and sentiment alongside emotion. In its #IVOTED analysis, Trump had more tweets and individuals expressing joy even though early sentiment analysis favored Hillary.
Problem
Sentiment classification of election tweets is difficult because conversation topics and candidate references change over time, shifting which features are useful.
Source: §1 · p. 3Method
Results
In the #IVOTED analysis, Trump had more tweets and individuals expressing joy, while sentiment analysis favored Hillary in the early hours.
Source: §7 · p. 6Takeaways & Limitations
The authors consider emotion a better criterion than sentiment for predicting actions such as voting, noting emotional differences among tweets with the same polarity.
Source: §7 · p. 6In the 2016 election analysis, only 43% of tweets containing #Imwithher were positive for Clinton and 27% used it sarcastically; using hashtags as classifier features decreased accuracy.
Source: §4 · p. 5
Abstract
from arXiv · showhide
With the proliferation of social media over the last decade, determining people's attitude with respect to a specific topic, document, interaction or events has fueled research interest in natural language processing and introduced a new channel called sentiment and emotion analysis. For instance, businesses routinely look to develop systems to automatically understand their customer conversations by identifying the relevant content to enhance marketing their products and managing their reputations. Previous efforts to assess people's sentiment on Twitter have suggested that Twitter may be a valuable resource for studying political sentiment and that it reflects the offline political landscape. According to a Pew Research Center report, in January 2016 44 percent of US adults stated having learned about the presidential election through social media. Furthermore, 24 percent reported use of social media posts of the two candidates as a source of news and information, which is more than the 15 percent who have used both candidates' websites or emails combined. The first presidential debate between Trump and Hillary was the most tweeted debate ever with 17.1 million tweets.
1. Fast-paced change in dataset
Election-tweet sentiment classification is difficult because topics and candidate references change rapidly, shifting which features matter. The authors argue for active learning and regular training-data updates, while noting annotation costs and the robustness of SVMs to dataset shift.
1. Fast-paced change in dataset
- Changing topics make important sentiment features obsolete and cause new ones to be missed, creating a dataset-shift problem when training and test distributions differ.Features can also disappear and reappear as supporters change what they discuss; political sentiment is often implicit rather than expressed with sentiment words [21].Source: §1 · p. 3
- Among the classification models considered, SVM is described as one of the most robust for dataset shift.Source: §1 · p. 3
- Active learning requires human annotation, which is costly and difficult because short, vague political tweets require annotators to understand political context.An MPQA lexicon-based test reached only 0.49 accuracy, which the authors attribute to the limited use of sentiment words in political tweets.Source: §1 · p. 3
- Regularly refreshing the training data, usually every few days, was the most effective measure for keeping the classifier reasonably good during the 2016 election.The first model trained during the primaries performed quite poorly during and after the conventions; new data were collected opportunistically, including during presidential debates.Source: §1 · p. 4
2. Candidate-dependence
A tweet can express different sentiment toward different candidates, and posts naming multiple candidates can introduce off-target features. Candidate-specific classifiers therefore need to select target-relevant features during both training and testing.
2. Candidate-dependence
- About 48% of post-convention instances contained variants of both Clinton’s and Trump’s names, allowing features about one candidate to interfere with sentiment classification for the other.Source: §2 · p. 4
- Context-based target-dependent methods outperform syntax-based methods on informal texts such as tweets.Syntax-based methods rely on POS tagging or parsing, while context-based methods define the left and right context for each target.Source: §2 · p. 4
- For tweets naming multiple candidates, include each instance in both candidate classifiers but retain only target-relevant features during training and testing.The authors identify dependency or proximity as ways to select on-target features and exclude off-target ones.Source: §2 · p. 5
3. The importance of identifying the user’s political preference
The paper estimates users’ political leanings from the orientations of accounts they follow, aiming to add user-preference information to social-media-based election prediction.
3. The importance of identifying the user’s political preference
- The authors categorize users as far left-leaning, left-leaning, independent, right-leaning, or far right-leaning using the orientation of accounts they follow.Source: §3 · p. 5
- They estimate left- or right-leaning preference from the ratio of correspondingly leaning followees to the user’s total followers, then compare that ratio with a threshold.Source: §3 · p. 5
- The authors state that political-preference information helps improve social-media-based election prediction.Source: §3 · p. 5
4. Content-related challenges (hashtags)
Election hashtags can be sarcastic, making them unreliable as automatic labels or classifier features. In the 2016 analysis, hashtag-based features reduced accuracy.
4. Content-related challenges (hashtags)
- Hashtags used for sarcasm undermined distant supervision: only 43% of tweets containing #Imwithher were positive for Clinton, while 27% used it sarcastically.The authors report that using such hashtags as classifier features decreased accuracy rather than increasing it.Source: §4 · p. 5
5. Content-related challenges (links)
Tweets linking to external documents may be incomplete without their content, making sentiment difficult to infer. The authors propose expanding tweets with linked-document content as a performance-improving feature.
5. Content-related challenges (links)
- Around 36% of 2016 election tweets contain a URL to an external link.Source: §5 · p. 6
- Without linked content, some tweets are incomplete and their sentiment is difficult even for human annotators to infer.Source: §5 · p. 6
- The authors hypothesize that adding linked-document content, keywords, or titles as features will improve performance.They note that tweet classifiers had not expanded tweets using URLs, although link expansion had been used for other problems.Source: §5 · p. 6
6. Content-related challenges (sarcasm)
Sarcasm complicates sentiment analysis because its polarity may affect only part of a tweet, and its scope remains difficult to determine. The authors report misclassification among sarcastic tweets and propose excluding them from training.
6. Content-related challenges (sarcasm)
- Sarcasm may reverse polarity, but its scope can cover only part of a tweet or its hashtags, leaving its effect on sentiment analysis an open issue.Source: §6 · p. 6
- 7% of Trump’s tweets and 6% of Clinton’s tweets were sarcastic.Source: §6 · p. 6
- Among sarcastic tweets, the system classified 39% for Trump and 32% for Clinton incorrectly.Source: §6 · p. 6
- The authors hypothesize that excluding sarcastic instances from training would remove noise and improve training-set quality.Source: §6 · p. 6
7. Interpretation-related challenges (Sentiment Analysis versus Emotion Analysis)
Emotion analysis provides finer-grained information than positive, negative, and neutral sentiment. In the #IVOTED analysis, emotion patterns differed from sentiment patterns, and the authors considered emotion a better criterion for predicting voting action.
7. Interpretation-related challenges (Sentiment Analysis versus Emotion Analysis)
- Emotion analysis of #IVOTED showed more tweets and individuals expressing joy for Trump than for Clinton.Source: §7 · p. 6
- Although sentiment analysis favored Hillary in the early hours, emotion analysis showed better support for Trump.Source: §7 · p. 6
- The authors considered emotion a better criterion for predicting voting action and said emotional differences within the same sentiment polarity supported their 2016 election prediction.Source: §7 · p. 6
8. Interpretation-related challenges (Vote counting versus engagement counting)
Election monitoring should account for both tweet volume and user activity when relating online opinions to actual votes. The authors recommend reporting both tweet-level and user-level normalized sentiment.
8. Interpretation-related challenges (Vote counting versus engagement counting)
- An election monitoring system should report user-level normalized sentiment alongside tweet-level sentiment.Source: §8 · p. 7
- The authors cite two explanations for why highly active users may matter: frequent tweeting can make opinion prediction more reliable, and active people may be more influential and more likely to vote.Source: §8 · p. 7
- The end-user analyzer must consider both user-level and tweet-level factors when making predictions.Source: §8 · p. 7
9. The importance of location
Election prediction must account for each state's electoral-vote weight. Twitris's spatial aspect played a crucial role in assisting end users during the 2016 election.
9. The importance of location
- Each state's influence in an election must be considered through its number of electoral votes when predicting the result.Tweet or user-profile locations can estimate a user's approximate location; Twitris's spatial aspect played a crucial role in assisting election prediction in 2016.Source: §9 · p. 7
10. Trustworthiness-related challenges (Bots)
The authors describe social bots as automated accounts that may emulate or try to change public attitude, and report using a hybrid empirical approach to identify tweets originating from an API. Twitris was commercially licensed, and Prof. Sheth was a cofounder of the licensee, Cognovi Labs.
10. Trustworthiness-related challenges (Bots)
- Social bots automatically generate social-media content and try to emulate and possibly change public attitude.The authors also describe bot wars between political sides.Source: §10 · p. 7
- The authors' hybrid empirical approach checked whether a tweet originated from an API and reportedly gave fairly good results.The passage directs readers to an external discussion but does not provide quantitative results.Source: §10 · p. 7
- Cognovi Labs licensed Twitris, and Prof. Sheth was a cofounder of the company.The relationship is disclosed in the acknowledgements.Source: §10 · p. 7