Source-linked AI summary
A new ANEW: Evaluation of a word list for sentiment analysis in microblogs
Finn Årup Nielsen
TL;DR
Existing sentiment lexicons vary in vocabulary and whether they encode polarity or sentiment strength, creating uncertainty about which lists suit microblogs. The paper constructs and evaluates a Twitter-oriented strength-scored list against ANEW and other methods using AMT-labeled tweets. The new list slightly outperforms ANEW with simple matching, while SentiStrength performs best.
Problem
Existing sentiment lexicons differ in coverage and scoring, and their suitability for informal microblog sentiment-strength detection is unclear.
Method
The paper constructs a manually scored Twitter-oriented lexicon containing sentiment strength, Internet slang, and obscene words, then evaluates it against AMT-labeled tweets.
Results
The new list slightly outperformed ANEW, while SentiStrength achieved the highest correlation with AMT labels at 0.610.
Takeaways & Limitations
Simple word-list scoring can perform slightly better with the Twitter-oriented list than with ANEW, but the more elaborate SentiStrength approach performs best.
Takeaways & Limitations
The application of the new list omitted negation detection, emoticon handling, and spelling-variation handling used by newer methods.
Abstract
from arXiv · showhide
Sentiment analysis of microblogs such as Twitter has recently gained a fair amount of attention. One of the simplest sentiment analysis approaches compares the words of a posting against a labeled word list, where each word has been scored for valence, -- a 'sentiment lexicon' or 'affective word lists'. There exist several affective word lists, e.g., ANEW (Affective Norms for English Words) developed before the advent of microblogging and sentiment analysis. I wanted to examine how well ANEW and other word lists performs for the detection of sentiment strength in microblog posts in comparison with a new word list specifically constructed for microblogs. I used manually labeled postings from Twitter scored for sentiment. Using a simple word matching I show that the new word list may perform better than ANEW, though not as good as the more elaborate approach found in SentiStrength.
1 Introduction
Microblog sentiment can be estimated with sentiment lexicons, but existing lists differ in coverage and scoring. The paper introduces a new strength-scored list containing Internet slang and obscene words and evaluates it with manually labeled Twitter data.
- Approaches: Lexicon-based sentiment analysis scores postings by matching their words and phrases against labeled sentiment values.The paper contrasts this approach with supervised machine learning and identifies SentiStrength as an example of lexicon-based analysis.
- Research gap: Existing affective word lists differ in vocabulary coverage, especially for Internet slang, obscene words, and sentiment strength.Some lists encode only positive/negative polarity rather than graded sentiment strength.
- Contribution: The paper constructs a new word list with sentiment strength and Internet slang and obscene words for microblog analysis.The list had been used on Twitter but had not yet been validated.
- Evaluation: Manually labeled texts from Amazon Mechanical Turk provide the evaluation data for comparing sentiment-analysis methods.The study uses AMT-labeled language data to assess competing word lists.
2 Construction of word list
The new word list was built incrementally for Twitter and manually assigned valence scores from −5 to +5. Its vocabulary is negatively biased, and its scores correlate strongly with ANEW when directly matched.
- List construction: The study’s newest list contains 2477 unique words and 15 phrases, with manually assigned valence scores from −5 to +5.The publicly distributed AFINN-96 version contains 1468 words, including a few phrases.
- List construction: The list was expanded from obscene and positive words using COP15 Twitter postings, an affective word list, Urban Dictionary slang, and DeRose’s emotion-word list.Additions included acronyms such as WTF, LOL, and ROFL.
- Valence distribution: 1598 words, or 65%, were negative, compared with 878 positive words; most positive words received +2 and most negative words received −2.Strong obscene words were typically rated −4 or −5.
- Comparison with ANEW: A Spearman rank correlation of 0.81 was obtained between directly matched scores in the new list and ANEW.Porter stemming and WordNet lemmatization did not significantly change the results.
- Comparison with ANEW: Splitting the lists at valence thresholds revealed a few discrepant words, with stemming producing additional mismatches such as alien/alienation.The reported discrepancies include aggressive, mischief, ennui, hard, silly, alert, mischiefs, and noisy.
3 Twitter data
The study evaluates lexicon scores against 1,000 independently collected tweets labeled repeatedly through AMT. Tweet sentiment is computed from word-level valences using several aggregation schemes.
- Twitter data: The tweets were not used during construction of the word list.This separates list construction from evaluation data.
- Scoring: The primary tweet score is the sum of matched word valences divided by the number of words.Alternative schemes include unnormalized sums, non-zero-word normalization, extreme-word selection, and quantization to +1, 0, and −1.
4 Results
Across 1,000 AMT-labeled tweets, the new list slightly outperformed ANEW in simple word-list scoring, while SentiStrength achieved the strongest correlation. ANEW scoring was better on the shared vocabulary intersection.
- Vocabulary coverage: 422 of 4,095 unique tweet words matched the new 2,477-word list, compared with 398 matches for ANEW’s 1,034-word list.The corpus contained 15,768 tokenized words in total.
- Performance: SentiStrength achieved the highest correlations with AMT labels: Pearson 0.610 and Spearman 0.616.The new list reached Pearson 0.564 and Spearman 0.596, while ANEW reached 0.525 and 0.544.
- Performance: The new list exceeded ANEW on AMT correlation, with Pearson 0.564 versus 0.525 and Spearman 0.596 versus 0.544.OpinionFinder and General Inquirer performed below both lists.
- Scoring robustness: Across scoring schemes, ANEW’s Pearson correlations ranged from 0.522–0.526, while the new list ranged from 0.543–0.581.For the new list, unnormalized sum scoring was highest and extreme scoring lowest; normalized sum scoring produced Spearman 0.596.
- Intersection analysis: On the 299-word intersection, ANEW valences produced Pearson 0.52 versus 0.49 for the new-list valences.This comparison separates scoring differences from vocabulary differences.
5 Discussion
The new word list slightly outperformed ANEW with simple word matching, but SentiStrength achieved the best overall performance. Its advantage may reflect Internet slang and obscene words, although ANEW remains preferable for validated psycholinguistic scoring.
- SentiStrength achieved the highest correlation with AMT labels at 0.610, ahead of the simple word-list approaches.
- The new word list performed slightly better than ANEW, while General Inquirer and OpinionFinder performed worse despite having larger lexicons.
- The intersection analysis indicated that ANEW had better scoring, so the new list’s full-lexicon advantage may come from Internet slang and obscene words.
- The application omitted negation detection, emoticon handling, and spelling-variation handling used by newer methods.
- The word list’s performance might improve slightly as additional words are added.
- ANEW remains preferable for scientific psycholinguistic studies because its scoring has been validated across several persons; its standard deviation was not used here.