Source-linked AI summary
Comparing and Combining Sentiment Analysis Methods
Pollyanna Gonçalves, Matheus Araújo, Fabrício Benevenuto, Meeyoung Cha
TL;DR
Existing sentiment methods lack an apples-to-apples comparison for polarity detection in online social-network messages. The paper compares eight methods using coverage and agreement, then combines them; coverage varies widely, and no single method consistently performs best.
Problem
Existing sentiment methods had limited apples-to-apples comparison in online social networks, leaving their relative applicability, advantages, disadvantages, and limitations unclear.
Method
The study compares eight sentiment methods across two large online-social-network datasets using polarity, coverage, and agreement, and develops a combined method with an open comparison API.
Results
Coverage ranged from 4% to 95%, agreement from 33% to 80%, and no single method consistently achieved high coverage and agreement across sources.
Takeaways & Limitations
Combining methods provides the best coverage with competitive agreement, while iFeel enables open access to comparative sentiment-method results.
Takeaways & Limitations
Combining all methods is not necessarily optimal because accuracy and precision decrease as more methods are combined, despite an F-measure above 0.7.
Abstract
from arXiv · showhide
Several messages express opinions about events, products, and services, political views or even their author's emotional state and mood. Sentiment analysis has been used in several applications including analysis of the repercussions of events in social networks, analysis of opinions about products and services, and simply to better understand aspects of social communication in Online Social Networks (OSNs). There are multiple methods for measuring sentiments, including lexical-based approaches and supervised machine learning methods. Despite the wide use and popularity of some methods, it is unclear which method is better for identifying the polarity (i.e., positive or negative) of a message as the current literature does not provide a method of comparison among existing methods. Such a comparison is crucial for understanding the potential limitations, advantages, and disadvantages of popular methods in analyzing the content of OSNs messages. Our study aims at filling this gap by presenting comparisons of eight popular sentiment analysis methods in terms of coverage (i.e., the fraction of messages whose sentiment is identified) and agreement (i.e., the fraction of identified sentiments that are in tune with ground truth). We develop a new method that combines existing approaches, providing the best coverage results and competitive agreement. We also present a free Web service called iFeel, which provides an open API for accessing and comparing results across different sentiment methods for a given text.
1. INTRODUCTION
The paper addresses the lack of apples-to-apples comparison among sentiment methods for OSN polarity analysis by evaluating eight methods on coverage and agreement. Results show substantial variation across methods, motivating a combined approach and the iFeel comparison API.
- OSN sentiment analysis supports applications such as tracking public opinions, mood fluctuations, and reactions to events, products, and services.
- Machine-learning methods adapt to specific contexts but require labeled data, whereas lexical methods avoid training data but struggle across contexts such as slang-heavy OSN text.
- The study compares eight sentiment methods on two large OSN datasets using coverage and agreement for positive-versus-negative polarity.
- Coverage ranges from 4% to 95% on real events, so the selected method can determine how much data is analyzed.
- Existing methods also disagree on event polarity, including airplane-crash and disease-outbreak messages, motivating a combined method and the iFeel API.
2. SENTIMENT ANALYSIS METHODS
This section introduces the investigated sentiment methods, including emoticon-based polarity extraction and its handling of multiple emoticons. It also notes that emoticons occur in fewer than 10% of OSN messages.
- The study investigates eight popular sentiment methods spanning NLP, crowdsourced labels, psychometric scales, and supervised or unsupervised machine learning.
- 2.1 Emoticons: Emoticons represent facial and non-facial expressions such as happiness, sadness, hearts, love, or affection.
- 2.1 Emoticons: Emoticon polarity is assigned from a set of common positive, negative, and neutral emoticons, using the first emoticon when messages contain multiple emoticons.
- 2.1 Emoticons: Fewer than 10% of OSN messages contain at least one emoticon, limiting the coverage of emoticon-based sentiment detection.
2.2 LIWC
LIWC analyzes text through a categorized dictionary and can identify positive and negative affects alongside emotional, cognitive, and structural components. The paper uses LIWC as a lexical tool while contrasting it with adaptive machine-learning approaches.
- LIWC stands for Linguistic Inquiry and Word Count and analyzes text using a dictionary of classified word categories.
- LIWC evaluates emotional, cognitive, and structural components and assigns words to multiple categories, such as affective, positive emotion, and cognitive process.
- The study uses LIWC2007 with an English dictionary containing labels for more than 4,500 words and 100 word categories.
- Machine-learning methods are suited to content-driven or adaptive polarity models for OSN data.
- Prior work expanded LIWC-based classification with OSN features including sentiment words, booster words, emoticons, and repeated punctuation.
- SentiStrength implements a combined learning approach reported as producing the best empirical results and training model in that prior work.
2.4 SentiWordNet
SentiWordNet is a lexical sentiment tool built on WordNet synsets, assigning positive, negative, and objective scores. In this study, polarity is determined by comparing average positive and negative scores.
- SentiWordNet associates positive, negative, and objective scores with WordNet synonym sets called synsets.
- The scores lie in [0, 1] and sum to 1 for each synset.
- The study averages scores across associated synsets and assigns positive polarity when the average positive score exceeds the average negative score.
- Objective sentiment scores are excluded when determining polarity.
2.5 SenticNet
SenticNet performs sentiment analysis at the semantic level by assigning polarity to common-sense concepts rather than analyzing syntax alone.
- SenticNet uses artificial intelligence and semantic Web techniques to infer polarity from common-sense concepts at a semantic level.
- The method applies Natural Language Processing to create polarity values for nearly 14,000 concepts.
- SenticNet was evaluated on patient opinions about England’s National Health Service and on LiveJournal posts labeled with more than 130 moods.
2.6 SASA
SASA is an open-source machine-learning sentiment analyzer included to enable an apple-to-apple comparison with other sentiment methods.
- SASA is a machine-learning-based sentiment analyzer evaluated using 17,000 labeled tweets from the 2012 U.S. Elections.
- The open-source tool used Amazon Mechanical Turk workers to label tweets as positive, negative, neutral, or undefined.
- The study includes SASA because it is open source and had not been compared directly with other sentiment methods.
2.8 PANAS-t
PANAS-t adapts a psychological mood scale to Twitter, using baseline-relative scores across eleven moods and grouping them into positive, negative, or neutral affect.
- PANAS-t adapts the Positive Affect Negative Affect Scale to detect Twitter users’ mood fluctuations.
- The method uses words associated with eleven moods, including joviality, fear, sadness, hostility, fatigue, and attentiveness.
- PANAS-t computes each sentiment’s P(s) score relative to baseline normative values, producing values from −1.0 to 1.0.
- The study classifies joviality, assurance, serenity, and surprise as positive affect; fear, sadness, guilt, hostility, shyness, and fatigue as negative; attentiveness as neutral.
- A related POMS adaptation was excluded because it was not publicly available upon request.
3. METHODOLOGY
The study compares eight sentiment methods using large-scale Twitter event data and human-labeled Web texts, evaluating polarity predictions with coverage and standard classification metrics.
- Datasets: The methodology uses two datasets: a near-complete Twitter log and six human-labeled social-web datasets.
- Datasets: The Twitter log covers 54 million users, 1.9 billion follow links, and 1.7 billion tweets from March 2006 to August 2009.
- Datasets: Six events spanning tragedies, products, movies, politics, health, and sports were identified by searching topic keywords in the Twitter dataset.
- Datasets: Event durations were limited because popular keywords are typically hijacked by spammers, and event tables report keywords and tweet totals.
- Datasets: The labeled datasets contain positive and negative human annotations from MySpace, Twitter, Digg, BBC Forum, Runners World Forum, and YouTube comments.
- Evaluation: SentiStrength was excluded from labeled-data accuracy measurement because it was trained on the same dataset.
- Evaluation: Evaluation reports recall, true negative rate, accuracy, precision, and F-measure using true- and false-classification counts.
- Evaluation: Coverage is the fraction of messages classified as positive or negative, and high coverage is sought alongside high F-measure to reduce bias from unidentified messages.
4. COMPARISON RESULTS
The comparison finds substantial variation across sentiment methods in coverage, agreement, and polarity prediction, with performance depending on the method and text source. Combining methods improves coverage, while polarity analyses reveal systematic positive bias and dataset-specific weaknesses.
- 4.1 Coverage: 90% and 91% coverage were achieved by SentiWordNet and SenticNet, respectively, for the AirFrance event.Emoticons and PANAS-t interpreted less than 10% of relevant tweets.
- 4.1 Coverage: 95% and 93% coverage were achieved by SentiWordNet and SenticNet, respectively, for the H1N1 event.SentiStrength followed with 61% coverage, and the coverage ranking resembled AirFrance.
- 4.1 Coverage: More than 92.75% coverage was achieved for each event by combining two methods.The corresponding uncovered fraction was smaller than 7.24% for every event.
- 4.2 Agreement: LIWC and PANAS-t agreed on 80% of overlapping messages, whereas SenticNet and PANAS-t agreed on 30.77%.PANAS-t and Happiness Index had no intersection; LIWC showed the strongest average agreement with other methods.
- 4.2 Agreement: Agreement across methods ranged from 33% to 80%, and below 50% agreement could correspond to a polarity change.Different sentiment tools could therefore interpret the same social media text differently.
- 4.3 Prediction Performance: The best F-measure was 0.846 for Emoticons, which had the lowest coverage, while results varied widely across labeled datasets.Performance was better on expressive social-network messages and lower on formal datasets; SentiStrength’s evaluation may be biased because it was trained on the dataset.
- 4.4 Polarity Analysis: Most methods produced more positive than negative values, and SenticNet identified the wrong polarity for predominantly negative datasets despite very high coverage.This positive bias also appeared in event data, including the airplane-crash event, which four methods considered positive.
5. COMBINED METHOD
The Combined-method combines existing sentiment approaches to maximize coverage while retaining competitive agreement, and iFeel exposes comparative method results through a public Web service.
- 5.1 Combined-Method: The Combined-method combines seven existing approaches, omitting LIWC because of copyright restrictions, and weights them to prioritize coverage followed by agreement.It uses PANAS-t, Emoticons, SentiStrength, SentiWordNet, SenticNet, SASA, and Happiness Index.
- 5.1 Combined-Method: 95% coverage and a 0.730 F-measure make the Combined-method the highest-coverage option while ranking below Emoticons and above the other methods in F-measure.The evaluation averages results across labeled Web datasets.
- 5.1 Combined-Method: Adding methods increases coverage with diminishing returns: the first four already reach 95%, leaving little room for further improvement.The F-measure decreases as more methods are combined but remains above 0.7.
- 5.1 Combined-Method: Combining more methods slightly lowers accuracy and precision, so the best combination may depend on the data and may reduce resource requirements.The paper notes that using fewer methods can require fewer resources in real-system deployment.
- 5.2 The iFeel Web System: iFeel lets users test and compare the sentiment methods and the Combined-method on chosen texts through a public Web API.The example shows that Emoticons and PANAS-t detect stronger negative affect than SenticNet and SASA for the same input.
6. CONCLUDING REMARKS
The paper compares eight sentiment methods for positive and negative polarity in OSN content and combines them to improve coverage. It also provides iFeel for comparative access, while acknowledging limits in method coverage and sentiment scope.
- 6. CONCLUDING REMARKS: The study presents an apple-to-apple comparison of eight representative sentiment methods for polarity analysis in OSNs.The methods include SentiWordNet, SASA, PANAS-t, Emoticons, SentiStrength, LIWC, SenticNet, and Happiness Index.
- 6. CONCLUDING REMARKS: The comparison measures coverage and agreement, finds that no method is always best across text sources, and motivates the Combined-method.Coverage is the fraction of messages whose sentiment is detected, while agreement measures alignment with ground truth.
- 6. CONCLUDING REMARKS: The Combined-method typically provides the highest coverage and competitive accuracy, and iFeel lets researchers compare sentiment tools through a Web API.The system also provides access to the Combined-method.
- 6. CONCLUDING REMARKS: The evaluation is limited to eight methods because obtaining access to prior tools was difficult, and it focuses on positive and negative polarity rather than broader sentiment categories.Future work proposes adding methods such as POMS and OpinionFinder and expanding sentiment categories.